Sentiment among big non-manufacturers rose to +19 from +16 in the previous quarter, above expectations of +17. That marked the second straight quarter of improvement.
Read MoreFormer BOJ economist: Economy on the mend
Weak economic recovery
Japan is still recovering after a three percentage point consumption tax hike to 8 percent last April tipped the economy into a technical recession. While the economy escaped the recession in the final quarter of last year many remain skeptical on the governments ongoing efforts to spur the economy.
Meanwhile, the Bank of Japan's 2 percent inflation target is increasingly further away, raising expectations that it will have to undertake further stimulus measure to achieve the goals it set out nearly two years ago.
"We already have a lot of hard data for Q1, which tend to provide more reliable indications about the pace of gross domestic product growth than the Tankan, and suggest that economic activity slowed last quarter," Marcel Thieliant, Japan economist at Capital Economics said in a note.
"What [the weak result] does is the yen could weaken even further... some people say it could go back to 130 as the Bank of Japan sees this number and says 'let's just print money,'" Boardman added.
Big firms forecasts capital expenditures for fiscal 2015/16 at -1.2 percent, below expectations of +0.5 percent in a Reuters poll. Among small firms the forecast was -21.2 percent, well below the Reuters of -16.5 percent.
"The capex number is certainly quite negative, but we've seen in the past, companies come out with very negative forecast then they revise higher. Japanese companies will be rising wages this year so that could be part of the expectations.. we won't be raising our investments because we are giving that to our employees," Boardman said.
Marcel Thieliant agreed: "[We wouldn't] place too much weight on firms' predictions for capital spending. For a start, these tend to be distorted by seasonal patterns: companies typically revise down their forecasts between the December and April survey."
"Indeed, firms are predicting a 1.2 percent year-on-year drop in the fiscal year that started today, well below their December forecast of a 8.7 percent rise. What's more, these predictions tend to lag actual capital expenditure," he said.
Japan's benchmark Nikkei index was down 0.8 percent in early trading on Wednesday.