Category: search engines

Disclosure: I serve as a consultant to various companies that compete with Google. That work is ongoing and covers varied subjects, most commonly advertising fraud. I write on my own—not at the suggestion or request of any client, without approval or payment from any client.

Through a FOIA request, the Wall Street Journal recently obtained–and generously provided to the public–never-before-seen documents from the FTC’s 2011-2012 investigation of Google for antitrust violations. The Journal’s initial report (Inside the U.S. Antitrust Probe of Google) examined the divergence between the staff’s recommendation and the FTC commissioners’ ultimate decision, while search engine guru Danny Sullivan later highlighted 64 notable quotes from the documents.

In this piece, I compare the available materials (particularly the staff memorandum’s primary source quotations from internal Google emails) with the company’s public statements on the same subjects. The comparison is revealing: Google’s public statements typically emphasize a lofty focus on others’ interests, such as giving users the most relevant results and paying publishers as much as possible. Yet internal Google documents reveal managers who are primarily focused on advancing the company’s own interests, including through concealed tactics that contradict the company’s public commitments.

About the Document

In a 169-page memorandum dated August 8, 2012, the FTC’s Bureau of Competition staff examined Google’s conduct in search and search advertising. Through a Freedom of Information Act (FOIA) request, the WSJ sought copies of FTC records pertaining to Google. It seems this memorandum was intended to be withheld from FTC’s FOIA request, as it probably could have been pursuant to FOIA exception 5 (deliberative process privilege). Nonetheless, the FTC inadvertently produced the memorandum — or, more precisely, approximately half the pages of the memorandum. In particular, the FTC produced the pages with even numbers.

To ease readers’ analysis of the memorandum, I have improved the PDF file posted by the WSJ. Key enhancements: I used optical character recognition to index the file’s text (facilitating users’ full-text search within the file and allowing search engines to index its contents). I deskewed the file (straightening crooked scans), corrected PDF page numbering (to match the document’s original numbering), created hyperlinks to access footnotes, and added a PDF navigation panel with the document’s table of contents. The resulting document: FTC Bureau of Competition Memorandum about Google — August 8, 2012.

AdWords API restrictions impeding competition

In my June 2008 PPC Platform Competition and Google’s "May Not Copy" Restriction and July 2008 congressional testimony about competition in online search, it seems I was the first to alert policy-makers to brazen restrictions in Google’s AdWords API Terms and Conditions. The AdWords API provided full-featured access to advertisers’ AdWords campaigns. With both read and write capabilities, the AdWords API provided a straightforward facility for toolmakers to copy advertisers’ campaigns from AdWords to competing services, optimize campaigns across multiple services, and consolidate reporting across services. Instead, Google inserted contractual restrictions banning all of these functions. (Among other restrictions: "[T]he AdWords API Client may not offer a functionality that copies data from a non-AdWords account into an AdWords account or from an AdWords account to a non-AdWords account.")

Large advertisers could build their own tools to escape the restrictions. But for small to midsized advertisers, it would be unduly costly to make such tools on their own — requiring more up-front expenditure on tools than the resulting cost-savings would warrant. Crucially, Google prohibited software developers from writing the tools once and providing them to everyone interested — a much more efficient approach that would have saved small advertisers the trouble and expense of making their own tools. It was a brazen restriction with no plausible procompetitive purpose. The restriction caused clear harms: Small to midsized advertisers disproportionately used only Google AdWords, although Microsoft, Yahoo, and others could have provided a portion of the desired traffic at lower cost, reducing advertisers’ overall expense.

Historically, Google staff disputed these effects. For example, when I explained the situation in 2008, AdWords API product manager Doug Raymond told me in a personal email in March 2008 that the restrictions were intended to prevent "inaccurate comparisons of data [that] make it difficult for the end advertiser to understand the performance of AdWords relative to other products."

But internal discussions among Google staff confirm the effects I alleged. For example, in internal email, Google director of product management Richard Holden affirmed that many advertisers "don’t bother running campaigns on [Microsoft] or Yahoo because [of] the additional overhead needed to manage these other networks [in light of] the small amount of additional traffic" (staff memo at p.48, citing GOOGWOJC-000044501-05). Holden indicated that removing AdWords API restrictions would pave the way to more advertisers using more ad platforms, which he called a "significant boost to … competitors" (id.). He further confirmed that the change would bring cost savings to advertisers, noting that Microsoft and Yahoo "have lower average CPAs" (cost per acquisition, a key measure of price) (id.), meaning that advertisers would be receptive to using those platforms if they could easily do so. Indeed, Google had known these effects all along. In a 2006 document not attributed to a specific author, the FTC quotes Google planning to "fight commoditization of search networks by enforcing AdWords API T&Cs" (footnote 546, citing GOOGKAMA-0000015528), indicating that AdWords API restrictions allowed Google to avoid competing on the merits.

The FTC staff report reveals that, even within Google, the AdWords API restrictions were controversial. Holden ultimately sought to "to eliminate this requirement" (key AdWords API restrictions) because the removal would be "better for customers and the industry as a whole" since it would "[r]educe friction" and make processes more "efficient" by avoiding time-consuming and error-prone manual work. Holden’s proposal prompted (in his own words) "debate" and significant opposition. Indeed, Google co-founder Larry Page seems to have disapproved. (See staff report p.50, summarizing the staff’s understanding, as well as footnote 280 as to documents presented to Page for approval in relaxing AdWords API restrictions; footnote 281 reporting that "Larry was OK with" a revised proposal that retained "the status quo" and thus cancelled the proposed loosening of restrictions.) Hal Varian, Google’s chief economist, also sought to retain the restrictions: "We’re the dominant incumbent in this industry; the folks pushing us to develop our PAI will be the underdogs trying to unseat us" (footnote 547, citing GOOGVARI-0000069-60R). Ultimately Holden’s proposal was rejected, and Google kept the restrictions in place until FTC and EC pressure compelled their removal.

From one perspective, the story ends well: In due course, the FTC, EC investigators, and others came to recognize the impropriety of these restrictions. Google removed the offending provisions as part of its 2013 commitments to FTC (section II) and proposed commitments to the EC (section III). Yet advertisers have never received refunds of the amounts they overpaid as a result of Google’s improper impediments to using competing tools. If advertisers incurred extra costs to build their own tools, Google never reimbursed them. And Google’s tactics suppressed the growth of competing search engines (including their recruitment of advertisers to increase revenue and improve advertising relevance), thereby accelerating Google’s dominance. Finally, until the recent release of the FTC staff report, it was always difficult to prove what we now know: That Google’s longstanding statements about the purpose of the restrictions were pretextual, and that Google’s own product managers knew the restrictions were in place not to improve the information available to advertisers (as Raymond suggested), but rather to block competitors and preserve high revenue from advertisers that used only Google.

Specialized search and favoring Google’s own services: benefiting users or Google?

For nearly a decade, competitors and others have questioned Google’s practice of featuring its own services in its search results. The core concern is that Google grants its own services favored and certain placement, preferred format, and other benefits unavailable to competitors — giving Google a significant advantage as it enters new sectors. Indeed, anticipating Google’s entry and advantages, prospective competitors might reasonably seek other opportunities. As a result, users end up with fewer choices of service providers, and advertisers with less ability to find alternatives if Google’s offerings are too costly or otherwise undesirable.

Against this backdrop, Google historically claimed its new search results were "quicker and less hassle" than alternatives, and that the old "ten blue links" format was outdated. "[W]e built Google for users," the company claimed, arguing that the design changes benefit users. In a widely-read 2008 post, Google Fellow Amit Singhal explained Google’s emphasis on "the most relevant results" and the methods used to assure result relevance. Google’s "Ten things we know to be true" principles begin with "focus on the user," claiming that Google’s services "will ultimately serve you [users], rather than our own internal goal or bottom line."

With access to internal Google discussions, FTC staff paint quite a different picture of Google’s motivations. Far from assessing what would most benefit users, Google staff examine the "threat" (footnote 102, citing GOOG-ITA-04-0004120-46) and "challenge" of "aggregators" which would cause "loss of query volumes" to competing sites and which also offer a "better advertiser proposition" through "cheaper, lower-risk" pricing (FTC staff report p.20 and footnote 102, citing GOOG-Texas-1486928-29). The documents continue at length: "the power of these brands [competing services] and risk to our monetizable traffic" (footnote 102, citing GOOG-ITA-05-0012603-16), with "merchants increasing % of spend on" competing services (footnote 102, citing GOOG-ITA-04-0004120-46). Bill Brougher, a Google product manager assessed the risks:

[W]hat is the real threat if we don’t execute on verticals? (a) loss of traffic from Google.com because folks search elsewhere for some queries; (b) related revenue loss for high spend verticals like travel; (c) missing opty if someone else creates the platform to build verticals; (d) if one of our big competitors builds a constellation of high quality verticals, we are hurt badly

(footnote 102, citing GOOG-ITA-06-0021809-13) Notice Brougher’s sole focus on Google’s business interests, with not a word spent on what is best for users.

Moreover, the staff report documents Google’s willingness to worsen search results in order to advance the company’s strategic interests. Google’s John Hanke (then Vice President of Product Management for Geo) explained that "we want to win [in local] and we are willing to take some hits [i.e. trigger incorrectly sometimes]" (footnote 121, citing GOOG-Texas-0909676-77, emphasis added). Google also proved willing to sacrifice user experience in its efforts to demote competing services, particularly in the competitive sector of comparison shopping services. Google used human "raters" to compare product listings, but in 2006 experiments the raters repeatedly criticized Google’s proposed changes because they favored competing comparison shopping services: "We had moderate losses [in raters’ assessments of quality when Google made proposed changes] because the raters thought this was worse than a bizrate or nextag page" (footnote 154, citing GOOGSING-000014116-17). Rather than accept raters’ assessment that competitors had high-quality offerings that should remain in search results, Google changed raters’ criteria twice, finally imposing a set of criteria in which competitors’ services were no longer ranked favorably (footnote 154, citing GOOGEC-0168014-27, GOOGEC-0148152-56, GOOGC-0014649).

Specialized search and favoring Google’s own services: targeting bad sites or solid competitors?

In public statements, Google often claimed that sites were rightly deprioritized in search results, indicating that demotions targeted "low quality," "shallow" sites with "duplicate, overlapping, or redundant" content that is "mass-produced by or outsourced to a large number of creators … so that individual pages or sites don’t get as much attention or care." Google Senior Vice President Jonathan Rosenberg chose the colorful phrase "faceless scribes of drivel" to describe sites Google would demote "to the back of the arena."

But when it came to the competing shopping services Google staff sought to relegate, Google’s internal assessments were quite different. "The bizrate/nextag/epinions pages are decently good results. They are usually well-format[t]ed, rarely broken, load quickly and usually on-topic. Raters tend to like them. …. [R]aters like the variety of choices the meta-shopping site[s] seem… to give" (footnote 154, citing GOOGSING-000014375).

Here too, Google’s senior leaders approved the decision to favor Google’s services. Google co-founder Larry Page personally reviewed the prominence of Google’s services and, indeed, sought to make Google services more prominent. For example: "Larry thought product [Google’s shopping service] should get more exposure" (footnote 120, citing GOOG-Texas-1004148). Product managers agreed, calling it "strategic" to "dial up" Google Shopping (footnote 120, citing GOOG-Texas-0197424). Others noted the competitive importance: Preferred placement of Google’s specialized search services was deemed important to avoid "ced[ing] recent share gains to competitors" (footnote 121, citing GOOG-Texas-0191859) or indeed essential: "most of us on geo [Google Local] think we won’t win unless we can inject a lot more of local directly into google results" (footnote 121, citing GOOGEC-0069974). Assessing "Google’s key strengths" in launching product search, one manager flagged Google’s control over "Google.com real estate for the ~70MM of product queries/day in US/UK/De alone" (footnote 121, citing GOOG-Texas-0199909), a unique advantage that competing services could not match.

Specialized search and favoring Google’s own services: algorithms versus human decisions

A separate divergence from Google’s public statements comes in the use of staff decisions versus algorithms to select results. Amit Singhal’s 2008 post presented the company’s (supposed) insistence on "no manual intervention":

In our view, the web is built by people. You are the ones creating pages and linking to pages. We are using all this human contribution through our algorithms. The final ordering of the results is decided by our algorithms using the contributions of the greater Internet community, not manually by us. We believe that the subjective judgment of any individual is, well … subjective, and information distilled by our algorithms from the vast amount of human knowledge encoded in the web pages and their links is better than individual subjectivity.

2011 testimony from Google Chairman Eric Schmidt (written responses to the Senate Committee on the Judiciary Subcommittee on Antitrust, Competition Policy, and Consumer Rights) made similar claims: "The decision whether to display a onebox is determined based on Google’s assessment of user intent" (p.2). Schmidt further claimed that Google displayed its own services because they "are responsive to what users are looking for," in order to "enhance[e] user satisfaction" (p.2).

The FTC’s memorandum quotes ample internal discussions to the contrary. For one, Google repeatedly changed the instructions for raters until raters assessed Google’s services favorably (the practice discussed above, citing and quoting from footnote 154). Similarly, Page called for "more exposure" for Google services and staff wanted "a lot more of local directly into search results" (cited above). In each instance, Google managers and staff substituted their judgment for algorithms and user preferences as embodied in click-through rate. Furthermore, Google modified search algorithms to show Google’s services whenever a "blessed site" (key competitor) appeared. Google staff explained the process: "Product universal top promotion based on shopping comparison [site] presence" (footnote 136 citing GOOGLR-00161978) and "add[ing] a ‘concurring sites’ signal to bias ourselves toward triggering [display of a Google local service] when a local-oriented aggregator site (i.e. Citysearch) shows up in the web results" (footnote 136 citing GOOGLR-00297666). Whether implemented by hand or through human-directed changes to algorithms, Google sought to put its own services first, contrary to prior commitments to evenhandedness.

At the same time, Google systematically applied lesser standards to its own services. Examining Google’s launch report for a 2008 algorithm change, FTC staff said that Google elected to show its product search OneBox "regardless of the quality" of that result (footnote 119, citing GOOGLR-00330279-80) and despite "pretty terribly embarrassing failures" in returning low-quality results (footnote 170, citing GOOGWRIG-000041022). Indeed, Google’s product search service apparently failed Google’s standard criteria for being indexed by Google search (p.80 and footnote 461), yet Google nonetheless put the service in top positions (p.30 and footnote 170, citing GOOG-Texas-0199877-906).

The FTC’s documents also call into question Eric Schmidt’s 2011 claim (in written responses to a Senate committee) that "universal search results are our search service — they are not some separate ‘Google product or service’ that can be ‘favored.’" The quotes in the preceding paragraph indicate that Google staff knew they could give Google’s own services "more exposure" by "inject[ing] a lot more of [the services] into google results." Whether or not these are "separate" services, they certainly can be made more or less prominent–as Google’s Page and staff recognized, but as Schmidt’s testimony denies. Meanwhile, in oral testimony, Schmidt said "I’m not aware of any unnecessary or strange boosts or biases." But consider Google’s "concurring sites" feature, which caused Google services to appear whenever key competitors’ services were shown (footnote 136 citing GOOGLR-00297666). This was surely not genuinely "necessary" in the sense that search could not function without it, and indeed Google’s own raters seemed to think search would be better without it. And these insertions were surely "strange" in the sense that they were unknown outside Google until the FTC memorandum became available last week. In response to a question from Senator Lee, asking whether Google "cooked it" to make its results always appear in a particular position, Schmidt responded "I can assure you, we’ve not cooked anything"–but in fact the "concurring sites" feature exactly guaranteed that Google’s service would appear, and Google staff deliberated at length over the position in which Google services would appear (footnote 138).

All in all, Google’s internal discussions show a company acutely aware of its special advantage: Google could increase the chance of its new services succeeding by making them prominent. Users might dislike the changes, but Google managers were plainly willing to take actions their own raters considered undesirable in order to increase the uptake of the company’s new services. Schmidt denied that such tampering was possible or even logically coherent, but in fact it was widespread.

Payments to publishers: as much as possible, or just enough to meet waning competition?

In public statements, Google touts its efforts to "help… online publishers … earn the most advertising revenue possible." I’ve always found this a strange claim: Google could easily cut its fees so that publishers retain more of advertisers’ payments. Instead, publishers have long reported — and the FTC’s document now explicitly confirms — that Google has raised its fees and thus cut payments to publishers. The FTC memorandum quotes Google co-founder Sergey Brin: "Our general philosophy with renewals has been to reduce TAC across the board" (footnote 517, citing GOOGBRIN-000025680). Google staff confirm an "overall goal [of] better AFS economics" through "stricter AFS Direct revenue-share tiering guidelines" (footnote 517, citing GOOGBRAD-000012890) — that is, lower payments to publishers. The FTC even released revenue share tiers for a representative publisher, reporting a drop from 80%, 85%, and 87.5% to 73%, 75%, and 77% (footnote 320, citing GOOG-AFS-000000327), increasing Google’s fees to the publisher by as much as 84%. (Methodology: divide Google’s new fee by its old fee, e.g. (1-0.875)/(1-0.77)=1.84.)

The FTC’s investigation revealed the reason why Google was able to impose these payment reductions and fee increases: Google does not face effective competition for small to midsized publishers. The FTC memorandum quotes no documents in which Google managers worry about Microsoft (or others) aggressively recruiting Google’s small to midsized publishers. Indeed, FTC staff report that Microsoft largely ceased attempts in this vein. (Assessing Microsoft’s withdrawal, the FTC staff note Google contract provisions preventing a competing advertising service from bidding only on those searches and pages where it has superior ads. Thus, Microsoft had little ability to bid on certain terms but not others. See memorandum p.106.)

The FTC notes Microsoft continuing to pursue some large Google publishers, but with limited success. A notable example is AOL, which Google staff knew Microsoft "aggressively woo[ed] … with large guarantees" (p.108). An internal Google analysis showed little concern about losing AOL but significant concern about Microsoft growing: "AOL holds marginal search share but represents scale gains for a Microsoft + Yahoo! Partnership… AOL/Microsoft combination has modest impact on market dynamics, but material increase in scale of Microsoft’s search & ads platform" (p.108). Google had historically withheld many features from AOL, whereas AOL CEO Tim Armstrong sought more. (WSJ reported: "Armstrong want[ed] AOL to get access to the search innovation pipeline at Google, rather than just receive a more basic product.") By all indications Google accepted AOL’s request only due to pressure from Microsoft: "[E]ven if we make AOL a bit more competitive relative to Google, that seems preferable to growing Bing" (p.108). As usual, Google’s public statements contradicted their private discussions; despite calling AOL’s size "marginal" in internal discussions (p.108), a joint press release quotes Google’s Eric Schmidt praising "AOL’s strength."

A Critical Perspective

The WSJ also recently flagged Google’s "close ties to White House," noting large campaign contributions, more than 230 meetings at the White House, high lobbying expenditures, and ex-Google staff serving in senior staff positions. In an unusual press release, the FTC denied that improper factors affected the Commission’s decision. Google’s Rachel Whetstone, SVP Communications and Policy, responded by shifting focus to WSJ owner Rupert Murdoch personally, then explaining that some of the meetings were industry associations and other matters unrelated to Google’s competition practices.

Without records confirming discussion topics or how decisions were made, it is difficult to reach firm conclusions about the process that led the FTC not to pursue claims against Google. It is also difficult to rule out the WSJ’s conclusion of political influence. Indeed, Google used exactly this reasoning in critiquing the WSJ’s analysis: "We understand that what was sent to the Wall Street Journal represents 50% of one document written by 50% of the FTC case teams." Senator Mike Lee this week confirmed that the Senate Committee on the Judiciary will investigate the possibility of improper influence, and perhaps that investigation will yield further insight. But even the incomplete FTC memorandum reproduces scores of quotes from Google documents, and these quotes offer an unusual opportunity to compare Google’s internal statements with its public claims. Google’s broadest claims of lofty motivations and Internet-wide benefits were always suspect, and Google’s public statements fall further into question when compared with frank internal discussions.

There’s plenty more to explore in the FTC’s report. I will post the rest of the document if a further FOIA request or other development makes more of it available.