Putting “Google” and “Reddit” in the same sentence is sure to elicit a sigh from those in the online publishing industry, since Google is currently the only search engine that can scrape Reddit and include new posts in search results. Google is currently the only search engine capable of scraping Reddit and placing new posts in search results.

404 Media took note of this, noting yesterday that non-Google search engines such as Bing and DuckDuckGo are not showing Reddit results from the past week in their search results. com” in another search engine like DuckDuckGo and set it to show only results from the past week, you can try it yourself. As of this writing, DuckDuckGo does not show such search results, but Google does.

This is likely due to a change in Reddit's robots.txt file, which is a file that almost every website has that tells bots like search engines which pages on the site are “not allowed” for scraping. In addition to preventing search engines from scraping pages, this file is useful for websites that are trying to prevent their data from being scraped for AI training by not allowing AI crawlers.

However, it appears that Reddit has recently made changes to prevent any bots from scraping websites. You don't have to take our word for it. You can check for yourself by visitinghttps://www.reddit.com/robots.txt.. The bottom few lines of the page basically tell bots that they are not allowed to scrape any pages on Reddit. And if there is no scraping, it will not appear in the search results. This is how search engines work, simply put: they do the scraping, they rank, and they show up when users search for relevant terms.

However, Google still shows the new Reddit results in its search results. This means that they have access to Reddit information even though their robots.txt does not allow it.

One can begin to wonder if this is related to Google's partnership with Reddit and Google's sole access to Reddit's site content for AI training, which provides the following reassurance.

Reddit spokesperson Tim Russchmidt told The Verge, “This has nothing to do with our recent partnership with Google,” and continued, “We have been in discussions with multiple search engines, including for use in AI, and we have no We have not been able to reach agreement with all search engines, as some are unable or unwilling to make enforceable commitments regarding the use of their content”

.

To my ears, however, it sounds like this problem is only indirectly caused by the Reddit-Google partnership. If the partnership agreement allows Google to exclusively use Reddit's data for AI training, then it makes sense that Reddit would not allow other search engines to scrape websites if they “cannot reach an agreement.”

If anything, this explanation takes things back a notch and makes it seem like there is another downside to the original Google-Reddit partnership. This mark is that Reddit will not allow other search engines to scrape their site unless they make an “enforceable” commitment to use Reddit's content for AI. (What “enforceable” means here is left to your imagination.)

Tim Sweeney, CEO of Epic Games, says this is “part of a disturbing acceleration of monopolies that are expanding to further stifle competition and rob users.”

By the way, this comes a few months after Google began pushing Reddit threads up the search result rankings for various terms. They are actively searching for them. It makes sense to show it to keep the search results relevant and to keep everyone happy.

There has been a lot of talk in the publishing world about how this Reddit-boosting change is or could be devastating for some smaller publications and independent sites. It may be because Google is right and people want Reddit to appear at the top of search results.

Well, that particular issue may have been mostly a problem for publishers, but this issue is certainly a bigger problem for end users. That is, if end users are not resistant to Google's ever-increasing search monopoly.

Think of it this way: if Google is right and end users really care that much about Reddit in search results, then Google now has exclusive search engine access to one of the things end users care about most. Why would they use any other search engine?

None of this even mentions the risks of the partnerships already in place that allow Google to have exclusive access to Reddit for AI training. One wonders what government agencies in the US, UK, and EU think about this, having just agreed to an initiative to prevent monopolies in the AI industry.