Category: Google

Google

Posted on

Revisiting Search Bias at Google

Last week Joshua Wright posted a critique of my January 2011 Measuring Bias in ‘Organic’ Web Search (with Ben Lockwood). Some quick thoughts –

First, there’s some important common ground here: Wright and I both find that Google shows many of its own results, and does so in prominent positions.

Now, Wright says Bing presents its own results slightly often more than Google does so. In contrast, important portions of my analysis indicated that own-service links are particularly prominent at Google. Why the gap between my analysis and Wright’s? One key challenge is the lack of a natural basis of comparison. Suppose Google controlled 50% market share while five competitors held 10% each. Then we could compare Google’s results to a possible consensus among the others — better revealing whether and when Google favors its own services. But in fact the runners-up are much smaller: after Bing+Yahoo, we reach smaller firms like Blekko, each with market share far below 1%. Without a competitive marketplace providing a baseline, comparisons between search engines are necessarily difficult. So it’s no surprise that the numbers come out differently depending on the approach.

Wright criticizes my decision to examine brief, popular searches rather than a selection of actual user searches. But searches are messy and idiosyncratic: Each day at Google, 20% to 25% of searches are completely new, never before seen. Wright favor analysis of searches seen in AOL logs, but this method tends to emphasize unusual searches like “dog who urinate on everything” and “you’re pregnant he doesn’t want the baby” (the first two examples in Paul Boutin’s 2006 examination of AOL logs). In contrast, I chose to focus on short, simple searches where biased results have a particularly broad effect.

Wright argues that if Bing presents its own results as often as Google does, then own-service links must be pro-competitive, raising no antitrust concern. I disagree. The same behavior can have very different consequences when performed by a dominant firm versus a smaller competitor. Indeed, section 2 of the Sherman Act only applies to companies with market power. Meanwhile, companies without market power may engage in the exact conduct that Section 2 prohibits. For example, Microsoft faced antitrust litigation when it included Internet Explorer in Windows, even as Apple permissibly included its Safari browser in MacOS.

Wright suggests that antitrust investigation of Google is stillborn for lack of consumer harm. I see two problems with this argument. First, competitive foreclosure is a sufficient cause for concern. Certainly Google’s own-service links can stymie competition: It’s a tall order to start the next Yelp if Google may adjust its algorithm to always put Google Places first. From 2006 through June 2011, ads from Google Checkout merchants featured a special logo — a benefit unavailable to merchants using competing checkout systems (like PayPal). Even the perception of such favoritism can disrupt competition: In June 2011 a Google Offers salesperson told a merchant that signing up with Offers would provide “SEO benefits” to make the merchant “number one in Google.” Google quickly disavowed that statement, but on Wright’s theory, this tactic would be entirely permissible. Imagine the harm to Groupon, LivingSocial, and fellow travelers if Google gave Offers advertisers the favored map placement that AdWords advertisers already enjoy. In my view, that’s the wrong result. New providers necessarily rely on Google to reach users, but their business plans won’t work if Google can systematically favor its own services at competitors’ expense.

Second, when Wright says there’s no consumer harm, he has the wrong consumers in mind. The folks paying the bills for Google are advertisers — and advertisers pay a high price that has only grown as Google gains market share.

Ultimately, Google’s antitrust problems go far beyond algorithmic search preference. Google’s harsh treatment of advertisers smacks of market power; an advertiser with a real choice of ad networks would not accept Google’s high prices and one-sided terms. Google’s dealings with mobile handset makers similarly draw on Google’s dominance: If there were numerous popular third-party operating systems for mobile handsets, Google wouldn’t be able to compel manufacturers into dropping third-party software like Skyhook; and if there were numerous strong vendors for search, maps, videos, and other core mobile apps, Google wouldn’t be able to bundle its mobile apps to compel handset manufacturers to take all of these as a condition of preinstalling any of them. At every turn, we see Google leveraging its dominance in certain sectors to shore up its position in others – and that’s a n approach that rightly raises significant antitrust concern.

Posted on

Advertisers’ Missing Perspective in the Google Antitrust Hearing

This week Google ex-CEO Eric Schmidt will testify at a Senate Antitrust Subcommittee hearing that investigates persistent allegations of Google abusing its market power. Other witnesses include Jeff Katz, CEO of Nextag, and Jeremy Stoppelman, CEO of Yelp — ably representing the publishers whose sites are pushed lower in search listings as Google gives its own services preferred placement. But who will speak for advertisers’ interests?

Each year Google bills advertisers some $30+ billion; advertisers quite literally pay the bill for Google’s market dominance. Yet advertisers seeking search traffic have little alternative to the prices and terms Google demands. Consider some of Google’s particularly onerous terms:

  • All-or-nothing placements. An advertiser wishing to appear in the Google Search Network must accept placement on the entirety of Search Network, in whatever proportion Google elects to provide. Some Google Search Network properties are excellent, like AOL and New York Times. Others are dubious, like typosquatting sites, adware, and pop-up ads. A competitive marketplace would push Google to offer advertisers a meaningful choice of advertising venues, and advertisers could choose which placements they want. Instead, Google bundles placements in a way that compels advertisers to buy worthless traffic they don’t want yet can’t avoid.
  • Low-quality search partners. Far from a good-faith effort to rid its network of low-quality partners, Google has retained placements through InfoSpace, a traffic syndicator whose undesirable traffic sources are well-known, amply documented (1, 2, 3), and ongoing. In a competitive marketplace, Google would have to offer advertisers high-quality, trustworthy traffic. But in current conditions, Google knows advertisers will accept Google’s traffic even if Google mixes in low-quality traffic advertisers do not want.
  • Opaque ranking and pricing. Google selects, orders, and prices advertisements using algorithms that only Google knows. As a result, advertisers struggle to understand why their ads appear in unfavorable positions or not at all: Is a competitor bidding more? Has Google assessed a competitor’s ads more favorably? (If so, is such assessment accurate or a system malfunction?) Or has Google quietly penalized an advertiser for taking actions adverse to Google, perhaps speaking to a journalist or complaining to a regulator?

    Google tells advertisers nothing about others’ bids, and Google provides only ambiguous information about its assessments of advertisers’ ads. So advertisers are left to wonder “have I been penalized?” without rigorous methods to answer that question. Advertisers would flock to a viable alternative search engine that treated them fairly and predictably while offering high-volume search traffic. But Google’s market power makes any such switch unrealistic.

  • Harsh contract terms. Google’s US Advertising Program Terms purport to let Google place ads “on any content or property provided by Google … or … provided by a third party upon which Google places ads” (clause 2.(y)-(z)) — a circular “definition” that sounds more like a Dr. Seuss tale than a formal contract. If Google does provide information about the sites where it places ads, Google disavows the accuracy of that information (no warranty or guarantee as to “reach, size of audience, demographics , or other purported characteristics of audience” (clause 5.(vi))). Google also “disclaims all warranties [and] guarantees regarding positioning, levels [or] quality … of costs per click, click through rates, … conversions or other results for any ads” (clause 5.(i)-(v)). Furthermore, even if an advertiser proves a violation, Google claims that “any refunds for suspected invalid impressions or clicks are within Google’s sole discretion” (clause 5).

    Even Google’s notification provisions are one-sided: An advertiser with a complaint to Google must sent it by “first class mail or air mail or overnight courier” with a copy by “confirmed facsimile.” (Despite my best efforts, I still don’t know how a “confirmed” facsimile differs from a regular fax.) Meanwhile, Google may send messages to an advertiser merely by “sending an email to the email address specified in [the advertiser’s] account” (clause 9).

    These terms smack of market power: Rare is the advertiser who would accept such terms if reasonable choices were available.

  • Banning tools to help advertisers move elsewhere. Savvy advertisers seek to buy placements through Google as well as competing search engines such as Yahoo and Bing. But Google builds roadblocks to hinder advertisers’ efforts. Certainly any advertiser wanting to run a large campaign on multiple search engines needs tools to help — to make the first copy from Google to competitors, and to perform ongoing sync’s and updates. But Google’s AdWords API brazenly prohibits tool-makers from offering these services — leaving advertisers either to do the work manually (unreasonably slow and costly) or to write their own tools by hand (infeasible for all but the largest advertisers).

    Google has never offered any pro-competitive or competitively-neutral explanation for restricting how advertisers copy their own ad campaigns. In a rare moment of frankness, one Google executive once told me “we don’t have to make it easy” for advertisers to use competitors’ services. That argument might have passed muster a decade ago, but Google’s dominance puts such tactics in a new light.

Google likes to argue that “competition is one click away.” First, I question whether users can actually leave as easily as Google suggests: Popular web browsers Firefox and Chrome strongly favor Google, as Google CFO Patrick Pichette recently admitted (“everybody that uses Chrome is a guaranteed locked-in user for us”). In the mobile context, Android offers Google similar lock-in. And even on non-Google mobile platforms, Google serves fully 95% of searches thanks to defaults that systematically direct users to Google. Meanwhile, syndication contracts assure Google exclusive long-term placement on most top web sites. Against this backdrop, users are bound to flow to Google. Then advertisers must go where the users are. Whatever choice users have, advertisers end up with much less.

In the last ten years, Google grew from 12% to well over 80% worldwide. In that time, Google moved from zero ads to a dozen or more per page; from placing ads only on its own site to requiring advertisers to purchase ads with thousands of partners of dubious or unknown quality; from hustling to convince advertisers to buy its novel offering, to compelling advertisers to accept the industry’s most opaque pricing and most onerous terms. At the start of a new decade, Google is stronger than ever, enjoying unrivaled ability to make advertisers do as Google’s specifies. It’s time for advertisers — and the regulators who protect them — to put a check on Google’s exploitation of its market power.

Posted on

Implications of Google’s Pharmacy Debacle

This week the Department of Justice announced the conclusion of its investigation of Google permitting online Canadian pharmacies to place advertisements through AdWords, facilitating the unlawful importation of controlled pharmaceuticals into the United States. Google’s large forfeiture — fully $500 million — reveals the gravity of the offense, and as part of the settlement, Google affirmatively admits liability. These admissions and the associated documents confirm what I had long suspected: Not only does Google often ignore its stated “policies,” but in fact Google staff affirmatively assist supposed “rule-breakers” when Google finds it profitable to do so.

Google’s Role in Unlawful and Deceptive Advertisements

The DOJ’s non-prosecution agreement has not been widely circulated but is well worth reading because it reveals the depth of Google’s misbehavior. As a condition of the non-prosecution agreement, Google specifically admits its knowledge of, and participation in, unlawful advertising.

  • Google admits that it knew as early as 2003 that Canadian pharmacies were unlawfully advertising through AdWords. Yet Google provided customer support to these pharmacies, including assisting them in placing and optimizing their advertisements and web sites.
  • Google’s policies required pharmacies to obtain certification to show ads to US consumers, but pharmacies found they could easily adjust their geo-targeting to reach US consumers without obtaining certification. Google admits that it knew about this tactic, yet failed to modify its systems to prevent uncertified advertisers from reaching US consumers.
  • Google admits that it knew pharmacies were circumventing certification by intentionally avoiding use of certain terms in the text of their advertisements, yet nonetheless using those same terms as advertising keywords to trigger displays. Google admits that it did not stop advertisers from using this technique until Google learned of the DOJ’s investigation.

Tension with Google’s Prior Statements Denying Knowledge of and Responsibility for Unlawful Advertisements

Previously, Google has always styled itself as an innocent victim of fraudulent online advertising, but a diligent foe of harmful ads. For example, when I presented dozens of deceptive AdWords advertisements in 2006, Google told Information Week “When we become aware of deceptive ads, we take them down.” In a 2010 blog post, Google claimed to “work very hard” to block deceptive ads, calling the process “a cat-and-mouse game” in that advertisers purportedly hide from Google’s efforts.

I have long doubted Google’s claims of innocence. For one, Google has an obvious incentive to allow deceptive and unlawful ads: each extra ad means extra revenue — an ad in lieu of white space, or an extra competitor encouraging other advertisers to bid that much higher. Furthermore, unlawful and deceptive ads have been widespread; I found dozens in just a few hours of work. Meanwhile, it’s hard to reconcile Google’s engineering strength — capably indexing billions of pages and tabulating billions of links — with the company’s supposed inability to identify new advertisements mentioning or targeting a few dozen terms known to deceive consumers. From these facts, I could only suspect what the DOJ investigation now confirms: Unlawful ads persist at Google not just because advertisers seek to be listed, but also because Google intentionally lets them stay and even offers them special assistance.

Problems Reach Beyond Deceptive Advertisements

Unlawful and deceptive ads are just one of many areas where Google has claimed to oppose bad behavior, but where there’s growing reason to doubt Google’s diligence.

Consider advertisements promoting services that infringe copyright. Google’s AdWords Policy Center indicates that Google prohibits ads promoting the copying or distribution of copyrighted content without permission from the rights-holder. How diligent is Google in blocking such ads? A 2007 Wall Street Journal article revealed Google’s affirmative support for sites engaged in copyright infringement: Seeing high traffic to web sites EasyDownloadCenter.com and TheDownloadCenter.com, Google offered those sites account representatives who suggested advertising keywords to optimize their AdWords campaigns. Google also offered those sites a line of credit , whereas ordinary AdWords advertisers must pay in advance. Anyone browsing the sites would have immediately recognized that they distributed copyrighted material without permission from the corresponding rights-holders, which should have caused Google to keep its distance. But Google staff looked the other way in order to retain and expand their business with a profitable advertising customer.

In revising its policy for use of trademarks in advertising, Google also put revenue considerations before users’ interests. Google promises that ads will be shown in a manner that is “clearly identified” to avoid user confusion, even if such placements reduce Google’s revenue. Indeed, through 2004, Google had required a trademark holder’s approval for a trademark to appear in search advertisements. But in 2009, Google identified an opportunity for at least $100 million of additional annual revenue, and potentially more than a billion dollars of additional annual revenue, by reversing that policy. Crucially, Google made that reversal even though Google’s own tests found that the change would cause an “overall very high rate of trademark confusion” in that “94% of users were confused at least once” during Google’s testing of the change.

Google has also ignored unlawful conduct in order to retain and expand its “domain parking” business which includes ads on tens of thousands of typosquatting domains (unlawful under the Anti-Cybersquatting Consumer Protection Act). Google claims to be diligent in preventing placement of ads on unlawful sites. Yet Google acts only in response to trademark owners’ complaints; Google could easily run its own searches, thanks to its superior information-processing systems and instant knowledge of which domains are showing Google ads. By allowing typosquatting to continue, I estimate that Google reaps additional revenue of approximately $497 million per year.

So too in the realm of copyright infringement at YouTube. By 2007, Google had installed a filter to identify YouTube videos which included copyrighted content. Google could have processed all YouTube videos through the filters in order to identify and remove all copyright-infringing content. Instead, Google offered the filter only to rights-holders who signed license agreements to let Google use their content. A copyright holder who simply wanted to keep its content off of YouTube had no means to do so: the company could not use Google’s filter because Google conditioned use of the filter on receipt of a license to the underlying content; and the company could not run its own filter because YouTube’s Terms of Service disallow the automated access and bulk downloads necessary for efficient searches. The DOJ’s investigation gives that much more reason to conclude what content owners long argued: Google’s approach was motivated not by genuine technical necessity, but rather by Google’s desire to impose its will on copyright holders.

Users’ privacy is also vulnerable to Google “errors” for the company’s benefit. When Google engineers deployed hundreds of cars with custom hardware and software that recorded WiFi users’ data, Google claimed the collection was “a mistake.” When Google Toolbar continued tracking users’ browsing even after users “disable[d]” the toolbar and even after the toolbar disappeared from view, Google called that behavior a “bug.” Though Google says its overbroad data collection was unintentional, both examples are suspicious. With thousands of programmers and an engineering culture, how could Google deploy software to hundreds of cars without a thorough code review? And Google Toolbar runs on hundreds of millions of computers, so one might expect at least basic testing of all disable features. Various critics speculated that Google’s WiFi data collection was actually intentional, and ArsTecnica inferred that Google had already known about the Toolbar’s overbroad data collection. Previously, others might have given Google the benefit of the doubt. But seeing Google’s obfuscation and duplicity in pharmaceutical advertising, it gets easier to believe that these “mistakes” were intentional too.

What Comes Next

The DOJ’s pharmacy investigation undermines Google’s credibility on questions of compliance with the law and good faith in enforcing its supposed policies. Previously, when Google argued that it was difficult to find bad ads, trademark-infringing domains, or copyrighted content, the world could only wonder what made these tasks so difficult for Google. Now we know: at least sometimes, Google’s difficulties were a farce; behind the scenes, Google employees were encouraging and supporting the very unlawful conduct they claimed to oppose.

As I noted in June, Google’s bad ads span myriad categories beyond pharmaceuticals– charging for services that are actually free, promising free service when there’s actually a charge, promoting copyright infringement, promoting spyware/adware, bogus mortgage modification offers, work-at-home scams, investment rip-offs, identify theft, and more. Each of these categories of scam advertisements is in fact unlawful, and most are prohibited under Google’s existing advertising policies. But policies alone are not enough. Will Google step forward with a serious effort to block these dubious offers? Or does Google prefer to retain the ads and enjoy the resulting revenue, but leave users vulnerable? The world is watching!

Posted on

Revisiting Unlawful Advertisements at Google

Last week, Google’s 10-Q disclosed a $500 million charge “in connection with a potential resolution of an investigation by the US Department of Justice into the use of Google advertising by certain advertisers.” Google initially declined to say more, but a Wall Street Journal report revealed that the charge resulted from Google’s sale of advertising to online pharmacies that break US laws.

While Google has certainly profited from selling advertisements to rogue pharmacies, that’s just one of many areas where Google sells unlawful advertisements. Here are six other areas where I’ve also seen widespread unlawful AdWords advertisements:

  • Advertisements charging for something that’s actually free. I’ve documented scores of AdWords advertisements that attempt to trick users into paying for software that’s widely available for free — charging for RealPlayer, Skype, WinZip, and more.
  • Advertisements promising “free” service but actually imposing a charge. I have also flagged dozens of advertisements promising “100% complimentary” “free” “no obligation” service that actually comes with a monthly charge, typically $9.99/month or more. Promising “free” ringtones, these services rarely ask users for their credit card numbers. Instead, they post charges straight onto users’ mobile phone bills — combining carrier-direct billing with deceptive advertising claims in order to strengthen the illusion of “free” service.
  • Copyright infringement – advertisements touting tools for infringing audio and video downloads. For example, in 2007 media companies uncovered Google selling advertisements to various download sites, typically folks charging for Bittorrent clients. These programs helped users download movies without permission from the corresponding rights-holders, which is a double-whammy to copyright holders: Not only did labels, studios, artists, and filmmakers get no share of users’ payments, but users’ payments flowed to those making tools to facilitate infringement.
  • Copyright infringement – advertisements touting counterfeit software. For example, Rosetta Stone in six months notified Google of more than 200 instances in which AdWords advertisers offered counterfeit Rosetta Stone software.
  • Advertisements for programs that bundle spyware/adware. At the peak of the spyware and adware mess a few years ago, distributors of unsavory software used AdWords to distribute their wares. For example, a user searching for “screensavers” would receive a mix of advertisements — some promoting software that worked as advertised; others bundling screensavers with advertising and/or tracking software, with or without disclosure.
  • Mortgage modification offers . Consumers seeking mortgage modifications often receive AdWords advertisements making deceptive claims. A recent Consumer Watchdog study found AdWords advertisers falsely claiming to be affiliated with the US government, requiring consumers to buy credit reports before receiving advice or help (yielding immediate referral fees to the corresponding sites), and even presenting fake certification logos. One prominent AdWords advertiser had previously faced FTC litigation for telemarketing fraud, while another faced FTC litigation for falsely presenting itself as affiliated with the US government. Other advertisers suffer unsatisfactory BBB ratings, and some advertisers falsely claim to have 501(c)(3) non-profit status.

Google’s Revenue from Deceptive Advertisements

Google does not report its revenues for specific sectors, so it is generally difficult to know how much money Google receives from particular categories of unlawful advertisements or from particular unlawful practices. That said, in some instances such information nonetheless becomes available. For example, the Wall Street Journal reported that Google charged $809,000 to one company advertising tools for unauthorized audio and video downloads. In 2006, I estimated that Google charged more than $2 million per year for advertisements distributing spyware and adware shown when users search for the single keyword “screensavers.” Scaling up to other keywords pushing spyware and adware, I suggested Google collects $25+ million per year for advertisements distributing spyware and adware.

Importantly, when AdWords advertisements deliver users into unlawful sites, the majority of the profits flow to Google. Consider a keyword for which several advertisers present similar unlawful offers. The advertisers bid against each other in Google’s auction-style advertising sales process — quickly bidding the price to a level where none of them can justify higher payments. If the advertisers are similar, they end up bidding away most of their profits. Indeed, most of these advertisers have low marginal costs, so their profit approaches their revenue, and Google even collects the majority of their revenue. In 2006 I ran an auction simulation to consider bidder behavior with 10 bidders and 20% standard deviation in per-click valuations; I found that in this situation, advertisers on average paid 71% of their revenue to Google. Drawing on litigation documents, the WSJ reports similar values: EasyDownloadCenter and TheDownloadCenter collected $1.1 million from users, but paid $809,000 (74%) to Google for AdWords advertising. Consumer Watchdog’s revenue estimates, drawn from Google’s own traffic estimation tools, reveal that an advertiser would need to pay more than $6 million per year to capture all the clicks from searches for “credit repair” and “bad credit.”

The Scope of Google’s Involvement — and Resulting Liability

Multiple sources have revealed Google’s far-reaching involvement in facilitating and supporting deceptive advertisements. For example, Google staff supplied EasyDownloadCenter and TheDownloadCenter with keywords to reach users, including “bootleg movie download,” “pirated,” and “download harry potter movie.” Similarly, plaintiffs in Goddard v. Google alleged that not only did Google show deceptive advertisements for “free ringtones” and similar searches, but Google’s own systems affirmatively suggested that advertisers target the phrase “free ringtones” (deceptive, since the advertisers’ service weren’t actually free) when advertisers requested only the word “ringtones.” Google’s involvement also extends to financing. For example, the WSJ reports that Google extended credit to EasyDownloadCenter and TheDownloadCenter — letting them expand their advertising effort without needing to pay Google in advance (as most advertisers must). In short, Google knew about these deceptive advertisements, profited from them, and provided assistance to the corresponding advertisers in the selection of keywords, in the provision of credit, and otherwise.

One might naturally expect that Google is liable when its actions cause harm to consumers — especially when Google knows what is occurring and profits from it. But the Communications Decency Act potentially offers Google a remarkable protection: CDA § 230 instructs that a provider of an interactive computer service may not be treated as the publisher of content others provide through that service. Even if a printed publication would face liability for printing the same advertisements Google shows, CDA § 230 is often interpreted to provide that Google may distribute such advertisements online with impunity. Indeed, that’s exactly the conclusion reached in Goddard v. Google, finding that even if Google’s keyword tools suggests “free ringtone” to advertisers and even if Google is aware of fraudulent mobile subscription services, Google is not liable to affected consumers.

The broad application of CDA § 230 immunity has attracted ample criticism. For example, a 2000 DOJ study concluded that “substantive regulation … should, as a rule, apply in the same way to conduct in the cyberworld as it does to conduct in the physical world.” Yet CDA § 230 unapologetically invites Google to show all manner of unlawful advertisements that would create liability if distributed by traditional publishers. And if Google can turn a blind eye to advertisers using its ad platform to defraud or otherwise harm users, advertisers will do so with impunity. For Google to escape liability is all the more puzzling when Google reaps most of the profits from advertisers’ schemes.

CDA § 230 includes several important exceptions. For example, the CDA does not immunize violations of criminal law — and rogue pharmacies implicate various criminal laws, giving rise to the liability for which Google now expects to pay $500 million. But this exception may be broader than critics yet realize. For example, large-scale copyright infringement and distribution of counterfeit goods may also create criminal liability for the underlying advertisers, hence excluding Google from the CDA safe-harbor for the corresponding advertisements.

Ultimately, I stand by my 2006 conclusion: “Google ought to do more to make ads safe.” Since then, Google’s revenue and profit have more than doubled, giving Google that much greater resources to evaluate advertisers. But I wouldn’t say Google’s users are twice as safe — quite the contrary, deceptive and unlawful advertisements remain all too widespread. Kudos to the Department of Justice for holding Google accountable for unlawful pharmacy advertisements — but there’s ample more work to be done in light of Google’s other unlawful advertisements.

Posted on

Remedies for Search Bias

Disclosure: I serve as a consultant to various companies that compete with Google. But I write on my own — not at the suggestion or request of any client, without approval or payment from any client.

In a forthcoming paper (update, November 2011: paper is available), I’ll survey the problem of search bias — search engines granting preferred placement and/or terms to their own links or to others’ links chosen for improper purposes. What purposes are improper? Given others’ work in that area, I’ll defer my thoughts on that subject to the paper. Today I’d like to focus on remedies — what tactics a dominant search engine ought not employ due to their detrimental effects on competition, and how prohibiting those tactics would help assure fair competition in search and related businesses.

The prospect of legal or regulatory oversight of search results has attracted skepticism. A search industry news site recently questioned the wisdom of investigating search bias by arguing that, even if bias were uncovered, “it’s not clear what any remedy would be.” James Grimmelmann last month critiqued the suggestion that search engines can be biased, and he argued that even if such bias exists, the legal system cannot usefully prevent it. Discomfort with the prospect of legal intervention extends even to those who ultimately see a need for oversight: For example, Pasquale and Bracha title a recent paper Federal Search Commission?, ending the title with a question mark to credit the immediate shortfalls of an overly bureaucratic approach. Meanwhile, Google’s caricature of regulation warns of government-mandated homogeneous results and unblockable web spam, offering a particularly pronounced view of search regulation as intrusive and undesirable.

I envision an alternative approach for policy intervention in this area — addressing the improprieties that various sites have alleged and stopping specific practices that ought not continue, while avoiding unnecessary restrictions on search engines’ activities.

Experience from Airline Reservation Systems: Avoiding Improper Ranking Factors

A first insight comes from recognizing that regulators have already — successfully! — addressed the problem of bias in information services. One key area of intervention was customer reservation systems (CRS’s), the computer networks that let travel agents see flight availability and pricing for various major airlines. Three decades ago, when CRS’s were largely owned by the various airlines, some airlines favored their own flights. For example, when a travel agent searched for flights through Apollo, a CRS then owned by United Airlines, United flights would come up first — even if other carriers offered lower prices or nonstop service. The Department of Justice intervened, culminating in rules prohibiting any CRS owned by an airline from ordering listings “us[ing] any factors directly or indirectly relating to carrier identity” (14 CFR 255.4). Certainly one could argue that these rules were an undue intrusion: A travel agent was always free to find a different CRS, and further additional searches could have uncovered alternative flights. Yet most travel agents hesitated to switch CRS’s, and extra searches would be both time-consuming and error-prone. Prohibiting biased listings was the better approach.

The same principle applies in the context of web search. On this theory, Google ought not rank results by any metric that distinctively favors Google. I credit that web search considers myriad web sites — far more than the number of airlines, flights, or fares. And I credit that web search considers more attributes of each web page — not just airfare price, transit time, and number of stops. But these differences only grant a search engine more room to innovate. These differences don’t change the underlying reasoning, so compelling in the CRS context, that a system provider must not design its rules to systematically put itself first.

I credit that some metrics might incidentally favor Google even as they are, on their face, neutral. But periodic oversight by a special master (or similar arbiter) could accept allegations of such metrics; both in the US and in Europe, a similar approach oversaw disputes as to what documentation Microsoft made available to those wishing to interoperate with Microsoft software.

Evaluating Manual Ranking Adjustments through Compulsory Disclosures

An alternative approach to avoiding improper ranking factors would require disclosure of all manual adjustments to search results. Whenever Google adjusts individual results, rather than selecting results through algorithmic rules of general applicability, the fact of that adjustment would be reported to a special master or similar authority, along with the affected site, duration, reason, and specific person authorizing the change. The special master would review these notifications and, where warranted, seek further information from relevant staff as well as from affected sites.

Why the concern at ad hoc ranking adjustments? Manual modifications are a particularly clear area for abuse — a natural way for Google to penalize a competitor or critic. Discourage such penalties by increasing their complexity and difficulty for Google, and Google’s use of such penalties would decrease.

I credit that Google would respond to the proposed disclosure requirement by reducing the frequency of manual adjustments. But that’s exactly the point: Results that do not flow from an algorithmic rule of general applicability are, by hypothesis, ad hoc. Where Google elects to use such methods, its market power demands outside review.

Grimmelmann argues that these ad hoc result adjustments are a “distraction.” But if Google’s manual adjustments ultimately prove to be nothing more than penalties to spammers, then regulators will naturally turn their attention elsewhere. Meanwhile, by forcing Google to impose penalties through general algorithms rather than quick manual adjustments, Google will face increased burdens in establishing such penalties — more code required and, crucially, greater likelihood of an email or meeting agenda revealing Google’s genuine intent.

Experience from Browser Choice: Swapping “Integrated” Components

Many complaints about search bias arise when longstanding innovative services are, or appear to be at risk of becoming, subsumed into Google’s own offerings. No ordinary algorithmic link to Mapquest can compete with an oversized multicolor miniature Google Maps display appearing inline within search results. (And, as Consumer Watchdog documented, Mapquest’s traffic dropped sharply when Google deployed inline maps.)

On one hand it is troubling to see established firms disappear in the face of a seemingly-insurmountable Google advantage. The concern is all the greater when Google’s advantage comes not from intrinsic product quality but from bundling and defaults. After all, if Google can use search to push users to its Maps product, Maps will gain market share even if competitors’ services are, on their merits, superior.

Yet it would be untenable to ask Google to disavow new businesses. It is hard to imagine a modern search engine without maps, news, or local search (among other functions largely absent from core search a decade ago). If legal intervention prevented Google from entering these fields, users might lose the useful functions that stem from integration between seemingly-disparate services.

What remedy could offer a fair chance of multiple surviving vendors (with attendant benefits to consumers), while still letting Google offer new vertical search services when it so chooses? E.C. antitrust litigation against Microsoft is squarely on point, requiring Microsoft to display a large choice screen that prompts users to pick a web browser. An initial listing presents the five market-leading options, while seven more are available if a user scrolls. But there is no default; a user must affirmatively choose one of the various options.

Taking the “browser choice” concept to search results, each vertical search service could, in principle, come from a different vendor. If a user prefers that her Google algorithmic search present embedded maps from Mapquest along with local search from Yelp and video search from Hulu, the user could configure browser preferences accordingly. Furthermore, a user could make such choices on a just-in-time basis. (A possible prompt: “We noticed you’re looking for a map, and there are five vendors to choose from. Please choose a logo below.”) Later, an unobtrusive drop-down could allow adjustments. The technical barriers are reasonable: External objects could be integrated through client-side JavaScript — just as so many sites already embed AdSense ads, YouTube player, and other widgets. Or Google and contributors might prefer server-to-server communications of the sort Google uses in its partnerships with AOL and with Yahoo Japan. Either way, technology need not stand in the way.

I credit that many users may be content with most Google services. For example, Google Maps enjoyed instant success through its early offering of draggable maps. But in some areas, Google’s offerings have little traction. Google’s Places service aspires to assess quality of restaurants and local businesses — but Yelp and Angie’s List draw on specialized algorithms, deeper data, and longstanding expertise. So too for TripAdvisor as to hotel reviews, and myriad other sites in their respective sectors. A user might well prefer to get information in these areas from the respective specialized services, not from Google, were the user able to make that choice.

Google often argues that competition is one click away. But here too, the E.C.’s Microsoft litigation is on point. Users had ample ability to install other browsers if they so chose, but that general capability was not enough when the standard operating system made one choice a default. Furthermore, at least Windows let other browsers truly immerse themselves in the operating system — as the default viewer for .HTML files, the default application for hyperlinks in email messages, and so forth. But there is currently no analogue on Google — no way for a user, even one who seeks this function, to combine Google algorithmic search with a competitor’s maps, local results, or other specialized search services.

Banning Other Bad Behaviors: Tying

Using its market power over search, Google sometimes pushes sites to adopt technologies or services Google chooses. Sometimes, Google’s favored implementations may be competitively neutral — simply technical standards Google wants sites to adopt (for example, presenting an index of pages to Google’s crawlers in a particular format). But in other instances, Google uses its power in search to promote adoption of Google’s own services.

I first flagged this tactic as to Google Affiliate Network (GAN), Google’s affiliate marketing service. GAN competes in one of the few areas of Internet advertising where Google is not dominant, and to date Google has struggled to gain traction in this area. However, Google offers remarkable benefits to advertisers who agree to use GAN: GAN advertisers alone enjoy images in their AdWords advertisements on Google.com; their advertisements always appear in the top-right corner above all other right-side advertisements (never further down the page); they receive preferred payment terms (paying only if a user makes a purchase, not merely if a user clicks; paying nothing if a user returns merchandise, a credit card is declined, or a server malfunctions). Moreover, merchants tend to use only a single affiliate network; coordinating multiple networks entails additional complexity and risks paying duplicate commissions on a single purchase. So if Google can convince advertisers to use GAN, advertisers may well abandon competing affiliate platforms.

Google’s tying strategy portends a future where Google can force advertisers and sites to use almost any service Google envisions. Google could condition a top AdWords position not just on a high bid and a relevant listing, but on an advertiser agreeing to use Google Offers or Google Checkout. (Indeed, Checkout advertisers who also used AdWords initially received dramatic discounts on the bundle, and to this day Checkout advertisers enjoy a dramatic multicolor logo adjacent to their AdWords advertisements, a benefit unavailable to any other class of advertiser.) Google would get a major leg up in mobilizing whatever new services it envisions, but Google’s advantage would come at the expense of genuine innovation and competition.

Posted on

In Accusing Microsoft, Google Doth Protest Too Much

In Accusing Microsoft, Google Doth Protest Too Much. HBR Online. February 3, 2011.

Google this week sparked a media uproar by alleging that Microsoft Bing “copies” Google results. But is that actually the best characterization of what happened? In fact Google’s engineers intentionally clicked bogus listings they had previously inserted into Google’s results, and they did this on computers where they had specifically authorized Microsoft to examine their browsing in order to improve Bing.

Strikingly, Google’s own Matt Cutts previously endorsed the use of Toolbar and similar data to improve search results — calling this approach “a good idea.” And Google’s own Toolbar Privacy Policy allows Google to perform the same analysis Bing used. So I don’t have much sympathy for Google’s allegations of impropriety. Quite the contrary: With Bing’s small market share, this data is important in improving Bing search results and building a viable competitor to Google’s dominant search offering.

Details, including what exactly happened, Google’s prior statements, and Google’s widespread use of others’ intellectual property:

In Accusing Microsoft, Google Doth Protest Too Much

Posted on

Measuring Bias in “Organic” Web Search with Benjamin Lockwood

By comparing results between leading search engines, we identify patterns in their algorithmic search listings. We find that each search engine favors its own services in that each search engine links to its own services more often than other search engines do so. But some search engines promote their own services significantly more than others. We examine patterns in these differences, and we flag keywords where the problem is particularly widespread.

Even excluding “rich results” (whereby search engines feature their own images, videos, maps, etc.), we find that Google’s algorithmic search results link to Google’s own services more than three times as often as other search engines link to Google’s services.

For selected keywords, biased results advance search engines’ interests at users’ expense: We demonstrate that lower-ranked listings for other sites sometimes manage to obtain more clicks than Google and Yahoo’s own-site listings, even when Google and Yahoo put their own links first.

Details, including methodology, analysis, and policy implications:

Measuring Bias in “Organic” Web Search

Posted on

Bias in Search Results?: Diagnosis and Response

Edelman, Benjamin. “Bias in Search Results?: Diagnosis and Response.” Indian Journal of Law and Technology 7 (2011): 16-32.

I explore allegations of search engine bias, including understanding a search engine’s incentives to bias results, identifying possible forms of bias, and evaluating methods of verifying whether bias in fact occurs. I then consider possible legal and policy responses, and I assess search engines’ likely defenses. I conclude that regulatory intervention is justified in light of the importance of search engines in referring users to all manner of other sites, and in light of striking market concentration among search engines.

Posted on

Knowing Certain Trademark Ads Were Confusing, Google Sold Them Anyway — for $100+ Million

Disclosure: I serve as a consultant to various companies that compete with Google. But I write on my own — not at the suggestion or request of any client, without approval or payment from any client.

When a user enters a search term that matches a company’s trademark, Google often shows results for the company’s competitors. To take a specific example: Searches for language software seller "Rosetta Stone" often yield links to competing sites — sometimes, sites that sell counterfeit software. Rosetta Stone think that’s rotten, and, as I’ve previously written, I agree: It’s a pure power-play, effectively compelling advertisers to pay Google if they want to reach users already trying to reach their sites; otherwise, Google will link to competitors instead. Furthermore, Google is reaping where others have sown: After an advertiser builds a brand (often by advertising in other media), Google lets competitors skim off that traffic — reducing the advertiser’s incentive to invest in the first place. So Google’s approach to trademarks definitely harms advertisers and trademark-holders. But it’s also confusing to consumers. How do we know? Because Google’s own documents admit as much.

Today Public Citizen posted an unredacted version of Rosetta Stone’s appellate brief in its ongoing litigation with Google. Google had sought to keep confidential the documents that ground district court and appellate adjudication of the dispute, but now some of the documents are available — giving an inside look at Google’s policies and objectives for trademark-triggered ads. Some highlights:

  • Through early 2004, Google let trademark holders request that ads be disabled if they used a trademark in keyword or ad text. But in early 2004, Google determined that it could achieve a "significant potential revenue impact" from selling trademarks as keywords. (ref)
  • In connection with Google’s 2004 policy change letting advertisers buy trademarks as keywords, Google conducted experiments to assess user confusion from trademarks appearing in search advertisements. Google concluded that showing a trademark anywhere in the text of an advertisement resulted in a "high" degree of consumer confusion. Google’s study concluded: "Overall very high rate of trademark confusion (30-40% on average per user) … 94% of users were confused at least once during the study." (ref)
  • Notwithstanding Google’s 2004 study, Google in 2009 changed its trademark policy to permit the user of trademarks in advertisement text. Google estimated that this policy change would result in at least $100 million of additional annual revenue, and potentially more than a billion dollars of additional annual revenue. Google implemented this change without any further studies or experiments as to consumer confusion. (ref)
  • Google possesses more than 100,000 pages of complaints from trademark holders, including at least 9,862 complaints from at least 5,024 trademark owners from 2004 to 2009. (ref)

Kudos to Public Citizen for obtaining these documents. That said, I believe Google should never have sought to limit distribution of these documents in the first place. In other litigation, I’ve found that Google’s standard practice is to attempt to seal all documents, even where applicable court rules require that documents be provided to the general public. That’s troubling, and that needs to change.

Posted on

Hard-Coding Bias in Google “Algorithmic” Search Results

I present categories of searches for which available evidence indicates Google has “hard-coded” its own links to appear at the top of algorithmic search results, and I offer a methodology for detecting certain kinds of tampering by comparing Google results for similar searches. I compare Google’s hard-coded results with Google’s public statements and promises, including a dozen denials but at least one admission. I conclude by analyzing the impact of Google’s tampering on users and competition, and by proposing principles to block Google’s bias.

Details, including screenshots, methodology, proposed regulatory response, and analogues in other industries:

Hard-Coding Bias in Google “Algorithmic” Search Results