Search Engines and Their Continued Importance

Over the past 30 years search engines like Google have become the central gateway to the internet. Even if classic search interfaces—where a user types one or more search terms and receives a list of links to relevant websites—may eventually be fully replaced by AI-based interaction methods, search engines will remain important.

Already users increasingly ask questions to a chatbot that then queries a search engine to generate an answer. That lets the chatbot include up-to-date information not available at its training time. Search engines are also essential for collecting training data for large language models. Their role as crawlers that regularly discover, analyze, evaluate, and maintain a central index of web pages will continue, because the internet’s basic architecture requires it. Only this enables fast, efficient lookup of which websites and which content exist.

At the network-technical level the internet is fundamentally decentralized, but today’s search engines are centralized by nature. One company holds roughly 90% of the market, and for Google as for nearly all alternatives, how results are ranked is entirely at the companies’ discretion. The workings of those ranking algorithms are opaque and driven by commercial interests. Because search results are existentially important for the modern internet and thus for modern society, this concentration of power is worrying. Although the EU and others have regulatory rules intended to ensure fair competition and neutrality, many cases show Google systematically abuses its position (see “Google Shopping antitrust case” and “Google spam policy investigation”). Moreover, the ranking Google applies has literally become a questionable but unavoidable standard for the “quality” or “relevance” of web content.