Former Microsoft CEO and major Twitter shareholder Steve Ballmer told CNBC on Thursday he's been on the phone with Twitter CEO Jack Dorsey, and he thinks it's time to focus full-time on the ailing social media brand.

"Every time I talk to Jack, the message is, 'We need more. More innovation. Move, move, move,'" Ballmer said on "Squawk Box." "Get the cost base right and move forward."