Google Search Results Rigged?
Is Google manipulating its search results to keep competitors down? And does its market dominance mean the government should step in?
The New York Times has questioned “The Google Algorithm,” worrying that the goliath might be skewing search results and hinting that the government should weigh in.
Google handles nearly two-thirds of Internet search queries worldwide. Analysts reckon that most Web sites rely on the search engine for half of their traffic. When Google engineers tweak its supersecret algorithm — as they do hundreds of times a year — they can break the business of a Web site that is pushed down the rankings.
I can personally attest to this. My Gone Hollywood site went from getting a million or so unique visits a month from Google to something like 200,000 three years ago, having only recently partly recovered — mostly owing to high rankings on Bing. OTB itself gets much more organic traffic, but still relies on search for a substantial share of traffic. About three weeks ago, we inexplicably fell into some sort of Google hole, cutting our traffic by nearly two-thirds.
It’s totally inexplicable and there’s no warning, appeal, or remedy. My strong guess is that it’s just some random decision made with no malice or thought about the impact it would have on my traffic. But it’s certainly frustrating.
When Google was a pure search engine, it was easy to appear agnostic about search results, with no reason to play favorites with one Web site or another. But as Google has branched out into online services from maps and videos to comparison shopping, it has acquired pecuniary incentives to favor its own over rivals.
Do a search for almost anything Google runs — blogs, videos, news, email, maps — and their products come up first. Worse yet, because Google’s primary business model is delivering search results based on ad bidding, you’ll also get plenty of paid results that look real.
Google argues that its behavior is kept in check by competitors like Yahoo or Bing. But Google has become the default search engine for many Internet users. Competitors are a click away, but a case is building for some sort of oversight of the gatekeeper of the Internet.
Plenty of bloggers agree:
Danny Sullivan of Search Engine Land jokes that it’s the NYT that needs regulating:
The New York Times is the number one newspaper web site. Analysts reckon it ranks first in reach among US opinion leaders. When the New York Times editorial staff tweaks its supersecret algorithm behind what to cover and exactly how to cover a story — as it does hundreds of times a day — it can break a business that is pushed down in coverage or not covered at all.
Reason‘s Peter Suderman:
The assumption here is that Google effectively belongs to the public now. And because it belongs to the public, the company has an obligation to the public to provide “fair” search results, while the government has the duty to define what constitutes fairness—and enforce it. In other words, Google, whose ingenuity has helped millions chart the web, should be forced to play by the rules of a handful of Washington regulators in service of a nebulous public interest that those same few regulators get to define.
Hot Air‘s Ed Morrissey:
I have my issues with Google, but it’s absurd to argue that Google has some sort of public-utility requirement to treat competitors equally with itself. If people don’t like the search results from Google, they can use Bing, Yahoo, Altavista, Jeeves, or any number of other competitors. Having Google produce search results that favor Google’s other businesses is exactly as shocking as having a Sears clerk recommend Craftsman tools, and as innocuous as well.
There’s a wee problem with all this, though: It’s not merely that Google is so much the default search engine for most people that the brand has become a verb — we “Google” things rather than “searching” — but the average user would simply have no way to know whether the results were less than adequate. The fact that a search for “spreadsheet” doesn’t return a link to Microsoft’s Excel — by leaps and bounds the most popular spreadsheet on the planet for years — on the first page even though its own knockoff product, Google Docs, comes up second, won’t even dawn on people not looking for inconsistencies. (Amusingly, doing a search for “spreadsheet” on Microsoft’s Bing returns essentially the same results. A sponsored link to “Check Out Excel 2010” is atop the page but then it’s Wikipedia, Google Docs, Webopedia, and then Welcome to Google Spreadsheets.)
So, I’m open to the idea that Google is moving dangerously close to the “public utility” line of a monopoly that needs to be scrutinized. The argument that holds me back from saying they’ve crossed the line comes from Gigaom‘s Mathew Ingram:
[I]f there were significant signs that Google was favoring its own properties, wouldn’t large competitors such as Microsoft or Amazon or Yahoo have raised this issue before now, rather than a tiny handful of little-known European competitors? Maybe there is a case for government oversight of Google, but the New York Times has failed to make that case, and so have most of the others who have tried to do so to date.
I do, however, think this is an issue that bears watching.
I scoff. Google is not dumb, and it would be imbecilic to corrupt their core value, which is good search.
That said, good search is hard, and ever evolving. There is no particular reason that in an evolving environment past winners should continue to win. Note also that much of the evolution is a war of measures/counter-measures against various evolutions of “spam.”
(It is also a mistake to think that it was necessarily a google change. A competing web site could get an SEO overhaul and suddenly kick your butt.)
BTW, if this goes through, I could give you head’s up that a couple attempts to comment on steve’s stimulus post failed. No error reported at post, but my comments do not appear.
I do think it’s mostly random. In the case of GHW, our previous rankings were likely unearned, anyway, although the revenue from the massive traffic was nice.
OTB is essentially unlisted in Google except for the homepage. We’ve gone from new posts being indexed within minutes to posts not being indexed at all. And we have a very good, Google-targeted, sitemap.
No clue on the rejected comments: Nothing in the spam filter.
Basically you’re right Jim, Google is heading into public utility territory. Personally I never use any other search engine like most other people. Comparisons with Sears are a bit silly. I’m inclined to agree that much of it is random but there’s clearly a patten of them favoring their own products so it’s not really much of a leap to believe they might do the same for favored clients. The problem is we don’t really know. It is intriguing why Yahoo and Microsoft aren’t making a fuss, but there could be a fairly straightforward explanation. They are doing the same thing so the pot is not calling the kettle black. Given that the NYT is now writing about it I suspect we haven’t heard the last of this.
So was it three weeks ago that you restructured your OTB home page?
Is that the coincidence?
It may just be that you are giving them less to play with.
The old laundry list OTB gave Google many, many things to key on.
The current format is closer to Glenn’s “Heh” as far as searchable content.
Although you have the linkage in your sitemap, the contents of the linked pages are pretty spare and that leads to a lower value for each of your pages.
What you had before traded on the values provided by the other articles on your front page.
Say you already had several articles about Biden on the page (each with several references to his name), adding another article about the Vice President would make the page more valuable in the eyes of the search engine.
When I look at a Texas Instruments (ti.com) web page, each page contains multiple instances of the product name for the item described on the page. These multiple references raise the value of that page for that multiply mentioned key word.
In short, I think that the re-layout of OTB is the reason behind what you are seeing.
So it is borderline dangerous when Google doesn’t include something, but amusing when Bing doesn’t?
I wonder if there is a precedent to turn to here, a free service ostensibly buttressed by a company’s other, pay services — which were built on the free service — and how they impact me as either a seeker, a provider or both. It is an interesting question.
@thomas carr
The timing occurred to me as well. But, no, the move happened two months ago now. And, while your theory on front page keyword density is intriguing, most major sites use a headline/excerpt format — including most of the big blogs. It would be odd, indeed, to penalize the site for going to a more professionalized layout.
@kenny
Not at all: I was pointing out that Google has the subtle power to knock down Microsoft’s rankings but noting that, in this particular case, Microsoft seemed to agree with the rankings.
Are there parallels we can draw here with Microsoft bundling their browser in their operating systems?