- The Australian government has shown that people want big, bold action against the tech companies.
- With the stroke of a pen, the Biden administration can bring competition to the search engine market.
- Opening up Google’s web index is how the United States can catch up in the international race to regulate Google.
- Zack Maril is the founder of the Knuckleheads’ Club, an organization dedicated to opening up Google’s web index.
- This is an opinion column. The thoughts expressed are those of the author.
- See more stories on Insider’s business page.
It was unthinkable even just a few months ago, but Google and Facebook have decisively lost their fight with the Australian government over its proposed News Media Bargaining Code. The Australian government is getting exactly what it wanted, and this won’t be lost on governments and regulators around the world.
People want their governments to go after tech companies, and this is most clear in America. The backlash against the tech companies over the past five years has washed away the last of the aura of invincibility they once so deftly wore.
Right now, the Biden Administration has an opportunity to give the people what they want. They have the chance to take the lead in the international race to regulate Google and jump leagues ahead of every other government. Just out of sight politically, there’s a hidden heart to Google, something that sits at the center of everything they do, something which classical economics demands be regulated without delay.
A different kind of index fund
The key to understanding and regulating Google is that Google has an immense advantage when it comes to “web crawling.” Web crawling is where Google’s strength flows from and the more you look into it, the more it clarifies the dynamics of the entire tech industry.
Web crawling is the industrial process where search engine operators go out and collect all the information about all the websites on the internet that they can. When you search for something on Google, Google isn’t going out and visiting every website looking for your query, they are looking in their saved cache of copies of all the websites they collected information from while crawling the web.
In order to quickly provide the answers to your search queries, search engine operators like Google and Bing have hundreds of thousands of computers engaging in web crawling day and night. These web crawling computers are browsing the internet and saving a copy of everything they can find. When you search for something on a search engine, the search engine looks in that saved index of all the websites the computers found while they crawled the web.
The better this index, the better the quality of search results that a search engine can provide. A search engine can only provide you links to web pages that it knows about and the more web pages a search engine knows about – and the sooner it knows about them – the more competitive advantage that search engine has over all the other search engines.
Google gets this advantage because it costs websites money when they are crawled by search engines. Naturally, website operators only want to be crawled by search engines that will send them lots of traffic and many website operators will limit who is allowed to crawl their websites to only the major search engines like Google, Bing, and a handful of others.
Website operators have to pay for web servers to serve the extra traffic from the crawlers. It’s an open secret within the industry that some of the major websites spend millions of dollars a year to pay for servers to serve traffic for just Google’s crawlers alone. This makes economic sense because Google will send them enough traffic that they can monetize, thus more than making up for the cost of being crawled.
But other search engines, especially new entrants to the search engine market, are locked out by this dynamic. Without access to many of the most important websites in the world, the quality of the search results that the search engine can provide suffers. Without the ability to provide high quality search results, the search engine has little hope of attracting many users. And without a large number of users, many website operators will limit that search engine’s access to their websites or block them entirely by default. This dynamic is a big reason why it is so hard to compete with Google.
How to fix Google’s monopoly
I’ve been researching this dynamic for over two years now and the evidence keeps piling up that the government needs to step in and open up Google’s index of the web.The New York Times recently published an article detailing how this dynamic plays out, with many search engine operators going on the record that they struggle to compete with Google because of the limits website operators put in place as well as the high costs associated with doing the crawling.
Over the past several decades, the economics of web crawling have become clear enough that we can call it out for what it is: web crawling is a natural monopoly. What this means is that it’s overall more economically efficient to have one firm engaging in web crawling than to have multiple firms all redoing the same amount of work to try and recreate that same index Google has already assembled. All we need to do is apply the well understood principles of regulating natural monopolies that we have been developing for over a century now.
The Biden Administration can jump start a new age in the digital era by ending Google’s unnatural hold on this natural monopoly and opening up Google’s web index to its competitors. With the stroke of a pen, the government can unleash innovations that have been held back for decades just because executives at Google didn’t think it would earn Google enough profits.
What we need now, and what the American people want, is leadership in the fight to take back control of our lives from tech companies. Web crawling is the most under-regulated but major aspect of the technology industry and regulating web crawling is the opportunity just waiting for the Biden administration to seize on.
To start, the administration could direct the Department of Justice and the Federal Trade Commission to look into whether Google ever acted anti-competitively in regards to its or another search engine’s web index. Next up, the Biden administration could study and publish a plan for opening up Google’s web index to its competitors. To tie it all together, the Biden administration could work with Congress to pass whatever laws might be needed to give them the tools they need to open up Google’s web index.
Opening up Google’s web index is still just out of sight politically, but not for long if I have anything to say about it. Through my founding of the Knuckleheads’ Club – an organization dedicated to advancing regulation of Google’s web crawling advantage – I was able to work with Congress to get this issue highlighted in the Big Tech antitrust report published last year. I’ve been in contact with governments across the world since then and have been excited at the progress already being made on this front. Australia has shown that you can win by going big and me and my colleagues are excited to see who will go after Google next.
Opening up Google’s web index is about the closest thing to a silver bullet that I’ve ever come across in politics. Opening up Google’s web index is how the Biden Administration can cement American’s position as a leader in technological innovation. This would be a bold and highly effective move that would have immense positive economic impact for decades to come.
Opening up Google’s web index is how the United States can catch up in the international race to regulate Google.
Zack Maril is the founder of the Knuckleheads’ Club, an organization dedicated to opening up Google’s web index and bringing back the open web. He is an expert in understanding and regulating Google, with his work having been featured in The New York Times and cited by Congress. He lives and works in Washington DC.