In online ads, it's Google, Facebook and then who?

Facebook CEO Mark Zuckerberg, left, and Google CEO Larry Page
Getty Images / David Paul Morris | Bloomberg | Getty Images
Facebook CEO Mark Zuckerberg, left, and Google CEO Larry Page

In the battle for digital ad dollars, it's Google, Facebook and then who?

That's the question marketers are asking as they try to spend effectively on the Web, and even more so on mobile, where traffic is shifting.

Heading into quarterly earnings reports next week, Google and Facebook utterly dominate digital advertising, according to eMarketer.

The two alone control 49 percent of the U.S. digital advertising market and 52 percent of mobile ads. (Tweet This) Way below them, the No. 3, 4 and 6 players—Microsoft, Yahoo and AOL—are losing market share, and nobody below them claims even 2 percent. Twitter ranks fifth at 2.3 percent.

Read MoreHow Google lost control of its anti-spam algorithm

With U.S. online advertising forecast to grow 16 percent this year to $58.6 billion and reach almost $75 billion in 2017, and with Google and Facebook constantly adding services on the Web and smartphones, ad departments are looking for ways to diversify their spending.

They're testing ads on Twitter, LinkedIn, Yelp and Pandora as well as services such as Pinterest and Snapchat. But none in the U.S. has emerged as a clear challenger to the Google-Facebook duopoly, a concern for brands that worry about losing visibility when the top two companies tweak an algorithm or raise prices.

Brands "don't want all their eggs in one basket," said Cathy Boyle, an analyst at eMarketer in New York. "They really do want a third player, but at this point we don't have one with the combined data and reach."

Google and Facebook are ubiquitous. Their users are scouring the web while logged in, enabling the tracking of every click and purchase and producing the kind of targeting capabilities that advertisers so covet.

Facebook is a distant second to Google, but growing much faster. The Menlo Park, California-based company is scheduled to report first-quarter results on Wednesday, and analysts expect sales growth of 42 percent to $3.56 billion and earnings per share of 40 cents up from 34 cents a year earlier, according to a Thomson Reuters survey.

Google, based in Mountain View, California, follows on Thursday. Expectations are for gross sales of $17.5 billion, an increase of 14 percent from a year earlier, with EPS rising to $6.60 from $6.27.

EMarketer sees Google's U.S. market share dropping from 36.6 percent this year to 32.5 percent in 2017, with Facebook gaining almost 2 percentage points to 13.4 percent. So the combined market share, while still dominant, is slowly slipping.

And with the European Union this week filing antitrust charges against Google for allegedly favoring its own properties over rivals in shopping searches, the distraction may open doors for others to enter.

Who's best positioned to take advantage?

Twitter, Amazon.com, LinkedIn and Yelp are all in the top 10 in the U.S., with market shares of between 0.8 percent and 2.3 percent and rising.

Read MoreEU files antitrust charges against Google

Patrick Llewellyn, chief executive officer of 99designs, is struggling in his search to find marketing channels that can actually attract new users. The company, which connects web designers with relevant businesses, spends about 70 percent of its marketing budget on Google, 10 percent on Facebook and spreads the rest across an assortment of sites and services.

Though Google is the most effective source for 99designs, it's getting expensive. Keywords such as "logo design" and "website design" have doubled in price in the past 12 to 18 months on Google's search engine, Llewellyn said, making what was once a relatively cheap marketing channel quite pricey. Facebook remains the best option for retargeting, or sending ads to people who have already expressed some level of interest in the brand, he said.

"Facebook is improving all the time," Llewellyn said. "We are increasing spend with them each month as we get to understand it better and better."

Elsewhere, 99designs is experimenting with audio ads on Pandora and in podcasts. It's also exploring Pinterest, which publicly launched promoted pins in January to let brands pay to get in front of relevant users. The company has run tests with Twitter ads, but "they haven't been as successful for us," Llewellyn said.

Google denies anti-competitive behavior

Only Google and Facebook are capturing a user's activity across mobile and web, giving marketers a comprehensive picture of consumers. With Google, it's a combination of search, YouTube and Android devices, while Facebook's advantage is the social network's 1.39 billion monthly active users and the many apps consumers access using their Facebook credentials.

"What's unique about Facebook and Google is that people are highly incented to be logged into their Google and Facebook accounts on multiple devices," said Eduardo Pretell, vice president of global consumer marketing at Ancestry.com, the provider of online genealogical information.

Like 99designs, Ancestry has found limitations with Twitter's targeting capabilities. Rather, outside of Google and Facebook, Ancestry's biggest digital ad expense is in the growing arena of content marketing. The company's writers are publishing original articles and expert commentary and using services such as Taboola and Outbrain to promote the content, hopefully driving traffic back to its site.

Snapchat jumped into the mobile advertising game late last year, letting brands send disappearing clips to some of the messaging app's 100 million-plus users. The start-up has to begin justifying that $15 billion valuation, but it's trying to do so by staying away from targeting and instead focusing on brand promotion.

Read MoreCould Snapchat see its valuation disappear?

"We want to see if we can deliver an experience that's fun and informative, the way ads used to be, before they got creepy and targeted," the company said in an October blog post.

Among companies intrigued by the potential for message-based advertising is SGN, maker of mobile puzzle games like "Cookie Jam." SGN President Josh Yguado said the company has been in talks with Snapchat as well as video-chat service Tango, but hasn't made the investment yet.

The problem, Yguado says, is that SGN wants to target users, and Google and Facebook "can target better than anyone else."

"Don't buy leads. Who wouldn't rather pay for sales?" -Scott Painter, TrueCar CEO

Others, like TrueCar CEO Scott Painter, are trying to change the advertising game altogether.

Painter's site, which helps consumers buy new and used cars, is soon rolling out a mobile service that lets more than a dozen auto manufacturers offer discounts to shoppers by analyzing their profile data. For example, based on what a consumer tells TrueCar, a promotion for $5,000 off a Mercedes-Benz or $2,000 off a Volkswagen Tiguan may appear.

TrueCar can offer prime placement to manufacturers with the most alluring deals, but the company only gets paid when a transaction is completed. So while Google and Facebook provide targeting, and TV ads enable brand promotion, TrueCar promises something entirely different: accountability.

Read MoreTrueCar zooms higher

"Manufacturers won't pay unless it results in a sale," said Painter.

Of course, TrueCar is limited to the auto industry, so the company is no threat to capture spending elsewhere. But, in a world where nobody can challenge Google and Facebook in terms of reach, web properties have to compete in creative ways.

Painter's pitch to carmakers is one that would surely resonate across many other industries. It goes like this:

"Don't buy leads. Who wouldn't rather pay for sales?"