Google's search engine is displaying Frankenstein-esque characteristics. Like the fictional scientist, Victor Frankenstein, who created a monster more powerful than its master, Google's algorithm designed to rid the Internet of spammy links is proving difficult, if not impossible, to control.
In June, CNBC.com reported on a Google algorithm called Panda that crawls the Web to periodically push what the search provider considers lower-quality sites down in the rankings while elevating better pages. The result is that some small Web businesses that rely on Google for traffic can be decimated overnight.
Read MoreGoogle secretive updates leave small sites scrambling
In keeping with the theme of exotic zoo animals, Google has another algorithm called Penguin, first rolled out in 2012, that aims to punish websites deemed to be abusive in splashing links across the Internet to artificially bolster rankings and lift traffic.
If you own an online music store focused on classic jazz, for example, and Google finds a bunch of unrelated Russian literature websites linking back to your site, the Penguin algorithm can ensure that fans looking for Louis Armstrong records no longer reach you. Late last year, lyrics website Rap Genius was slapped down by Google for its over aggressive use of links. In the industry, such abusive behavior is known as "black hat webspam."
After being hit by Penguin, webmasters seeking to get back in Google's good graces have to locate the spam, get rid of it and then wait for Google to hit the refresh button so they can—hopefully—climb back up the rankings. But here's the problem.
Penguin hasn't refreshed since October, so businesses hurt by the latest update almost a year ago are still reeling even if they've spent massive amounts of time and money to regain compliance. Some of them, as far as they knew, did nothing wrong, says Josh Bachynski, a consultant who works with companies on search engine optimization (SEO) and is a regular critic of Google's practices.
"Not only do they have a lot of collateral damage, with sites that were inadvertently hit that didn't break rules in any way, shape or form," Bachynski said. "But they've been held down for 10 months."
Since introducing Penguin in April 2012, Google has released four updates. Before the current 10-month stretch of inactivity, the longest break between updates had been seven months. The Mountain View, California-based company doesn't pre-announce its refresh plans so as not to give spammers any additional ammunition. And since Google assumes that it's only hurting the bad guys, it's in no hurry to let them recover, Bachynski said.
The last Penguin update was announced in a tweet on Oct. 4, from Matt Cutts, head of Google's webspam team (Cutts said last month that he was taking an indefinite leave of absence to spend more time with his wife).The tweet said that the change would noticeably affect about 1 percent of searches. Google handles more than 3 billion searches a day.
Golden Services was among Penguin's losers. Started in 2012, the housecleaning service near Dallas had been signing up one to two new customers a day until October, said Robert Buehler, whose wife runs the company full time.
Traffic plunged overnight after Google's algorithm change. New sign-ups fell to about one to two a week and the company cut its staff to four from nine, Buehler said. A search on "house cleaning service" and "Frisco, Texas," where Buehler's company is based, buries the site around the third page of listings, even though it lands high in results on Google Maps.
Buehler says he made some mistakes, including buying a company and pointing the domain back to the Golden Services site. But the bigger issue, which he discovered after hours spent on a Google forum and eventually a phone call from one of the forum's participants, was that 450,000 links to his site had been spread to some of the darkest corners of the Web.
In the comments sections of random Russian blogs, links would direct readers to goldenservices.com. One site that Buehler had never heard of had more than 1,000 links to his company, he said.
Buehler claims he's a victim of what's called negative SEO, which goes like this: Company A wants to lift its search ranking over Company B, so it uses black hat techniques to promote Company B, expecting that Penguin will soon punish Company B and, by extension, elevate Company A.
The website negativeseocompany.com spells it out explicitly: "We will build thousands of low quality links to your competitors website. They go down, you take their place. We will use SEO techniques made possible by the Google algorithm to get your site above your competitors."
Google disputes the notion that negative SEO is undermining its policing efforts.
"We frequently hear claims of 'negative SEO,' but when we've investigated we haven't found evidence that it's actually a significant problem," company spokesman Jason Freidenfelds said in an email. "In fact, our algorithms are designed to prevent this, and we use multiple signals to determine quality and relevance."
In a separate move to improve its algorithm, Google said in a blog post Wednesday that its search results are favoring Websites that are encrypted.
By the end of 2013, two months after the latest Penguin update, Buehler had done everything he could to remove the bad links, including contacting site owners to ask for removal and formally disavowing, as Google suggests, more than 2,000 domains where he'd found links. But he can't see any improvement until Google decides to send a refresh.
"It's as clean as I could make it," Buehler said. "I've done everything I'm aware of that I can do, so take me out of the penalty box."
As for the irregularity of the updates and the 10-month gap since the last one, Google's Freidenfelds said, "we do listen to feedback from webmasters, and we're going to keep improving Penguin so, for example, updates might be continuous."
Eric Enge, head of SEO consultancy Stone Temple Consulting, isn't convinced that Google has its arms around the negative SEO problem. The company may actually be struggling to distinguish between the good guys and the troublemakers, thus raising the Frankenstein question—has Google lost control of the algorithm?
"I suspect that there really is a problem they're having trouble dealing with," said Enge, who's worked in the SEO industry for 12 years and recently wrote a blog post describing Penguin as "punitive" and calling for Google to provide a more regular refresh. "Historically, Google has been a company that's had a culture of letting people recover. With Penguin, that's not really realistic right now."
Behind Google's leadership shuffle
Buehler, who works in technology sales, knows that diversifying the source of his traffic and reducing reliance on Google would make life easier. But he says that advertising on social networks hasn't produced results. Search is what makes the business work.
And in the search market, Google is the only real game in town. According to comScore, Google controls 68 percent of U.S. search, with Microsoft and Yahoo capturing most of the rest. Enge says that when he looks at how his clients spend, Google's market share is closer to 85 percent. More than $50 billion of Google's $60 billion in revenue last year was from online advertising, largely from sponsored links that run alongside search results.
Penguin does have its supporters. Steve Wiideman, an independent SEO consultant, works with big brands like Skechers and Meineke. He says that if companies concentrate on building a clean site, producing useful content and doing outreach to business-listing services, rather than spreading links, they've got a much better shot of moving up the search rankings.
"Penguin has cleaned up so much garbage and spam created by black hat SEO companies," Wiideman said, adding that Google manages negative SEO just fine. "It's pretty easy for Google to see what is natural behavior and what is manipulated behavior."
Most of the negative SEO complaints Wiideman has heard are from small businesses that at some point hired the wrong person to try and boost traffic and ended up with a bunch of low-quality links.
That story resonates with Antonio del Drago. The fantasy writer started the site mythicscribes.com in 2010 for fans of science fiction and the like. He gained an early following and the next year said he was encouraged to hire an SEO consultant to grow even faster.
In early 2012, del Drago received a warning from Google for placing "unnatural" links around the Web, so he fired the consultant. When the first version of Penguin was released, his traffic plummeted from about 2,000 visitors a day to just a few hundred.
Read MoreGoogle duplicating US success globally
Del Drago, whose primary income comes from teaching, didn't put much effort into figuring out the problem until late 2013. "I didn't really know anything about SEO. I was overwhelmed by the situation and didn't think I could recover," he said. "Last year, I sat down to read about it to figure it out. For someone on the outside looking in, SEO is a pretty complicated thing."
What del Drago found was hundreds of pornographic sites linking to mythicscribes.com. He's fairly certain they weren't created by his former consultant, but regardless he disavowed all of them along wiith more than 1,000 other domains that looked like they might be related to the porn sites. None of that work will be rewarded until the next Penguin refresh.
There's no guarantee that even with the improvements, an update will lift the rankings for del Drago or anyone else, because links are just one aspect of the equation. But there's a rising chorus of voices from small businesses and industry experts like Glenn Gabe who at least want Google to acknowledge that its crackdown on spam is punishing more than just the black hats.
"It's not fair to let those webmasters sit in Penguin limbo when they have worked hard to remove and disavow links," said Gabe, head of marketing firm G-Squared Interactive. "But that's exactly what's going on."
—By CNBC's Ari Levy