Web Bots: What Every Website Owner Needs to Know

By Charlie Minesinger, Director of Sales, Distil Networks

If you or your company operates a website or utilizes online advertising, then web bots and spiders are a cause of serious concern. No one is immune from bot traffic, so the question is: how much bad bot traffic is acceptable? Can you measure your bad bot traffic? Malicious bots can threaten the viability of your ad campaigns, affect your KPI tracking and analytics, and negatively impact your sales, conversions and overall customer experience. To get an idea of their true impact, let’s first discuss the kinds of web bots to which your site could fall victim.

Theft Bots

Theft bots aim to steal content, pricing, images and other information from sites on the web. Sometimes, they do this to sell the content to rival websites or content farms; at other times, they simply use it to post directly to their own sites. In some cases, a competitor may even use your data to get an inside look at your content and pricing, so they can get a leg up in the marketplace. In addition to scraping your content and data, theft bots can also harvest lists of email addresses for spam purposes.

In addition to these negative effects, theft bots can also cause SEO problems for site owners. If content is stolen and duplicated elsewhere on the web, it severely downgrades the original site’s search engine rankings and could even result in getting a site blacklisted.

A great example of a theft bot can be seen in a recent court case. Last year, the Metropolitan Regional Information Systems Inc., or MRIS, filed suit against the American Home Realty Network for copyright infringement. MRIS alleged that AHRN had used web bots to effectively steal content from various real estate MLS listing sites and then pass the information off as its own. While the case is still pending, MRIS was granted an injunction, which likely means that AHRN will be found guilty.

Fraud Bots

Fraud bots create fraudulent traffic by clicking on paid advertisements and display ads throughout the web, and sometimes they even post spam to websites, blogs and various web forms. These bots can create fraudulent accounts on databases or networks to gain access to directories of customer information. Fraud bots mean bad news all around for your site, your customers and your sales.

The biggest issue with fraud bots is the problem of click fraud. In fact, according to AdWeek, web bots steal an average of $6 billion from advertisers every year through fraudulent traffic and clicks. Furthermore, according to Medialink president Wenda Millard, about 25 percent of all traffic attracted by online ads is likely fraudulent. That means one out of every four clicks your site gets on its PPC or display ad campaigns could be a fake. When you consider the costs of those fraudulent clicks over a week, a month or even a year, it can result in substantial financial losses.

More on Bad Bots

In addition to the above repercussions, web bots also lower the quality of the user’s experience as a whole. Bots can cause slow moving, poorly performing websites; infect users’ devices, computers and mobile phones; and diminish the overall web-surfing experience. Businesses and website owners may see their bandwidth decreased, their KPIs and analytics become hard to decipher, and their intellectual property, content and data stolen.

The number of both fraud and theft bots has grown exponentially over the last few years. According to Distil Network’s most recent Bad Bot Landscape Report, the number of damaging web bots doubled in 2013. Bad bots accounted for just 12.25 percent of all web traffic in the first quarter, but climbed to a whopping 23.6 percent by the end of the fourth quarter. Furthermore, though many bad bots come from overseas, the Landscape Report reveals that more bad bots originate in the United States than any other country in the world.

On May 15, 2014, the Executive Director and President of the Online Trust Alliance (OTA), Craig Spiezle, testified to the U.S. Senate about the risks of malicious ads and web bots. OTA research reports that malicious advertising, or “malvertising,” rose by more than 200 percent in 2013. In addition to threatening the effectiveness of your website, malicious ads can be used to infect personal computers, steal personal information, or manipulate devices into becoming part of a DDoS attack. In addition, one botnet called “Pushdo” was detected on more than 4 million IP addresses, including IP addresses of most every major Internet Provider in North America and Europe.

Fortunately, there are technologies and strategies for blocking bots and protecting your website and your ads from fraud and theft without negatively affecting SEO and your (human) customer experience. We’ll be discussing these strategies, as well as more on web bots, in an upcoming post, so stay tuned! In the meantime, take your traffic readings and PPC results with a grain of salt—you never know when web bots may strike.


  1. I think this article fails to truly explain the intricacy of the issue. If I am correct, what Mr. Minesinger is calling “Theft Bots” is actually “Web Scraping” there are many methods of web scraping, and like any technology, can be used for both bad and good. I apologize to Mr. Minesinger if he was speaking about only the negative uses of the technology, however the article does not make that clear.

    I think the things that we should think about before just blanketing web scraping as a nefarious practice are the following:

    1. Does the company performing the web scraping and the company getting scraped have an agreement? If so, would he still call it a “theft bot?

    2. What content should be considered theft? Is pricing information which a company freely provides to the world, copyright?

    3. Are there certain uses of the technology that can benefit both parties?

    4. Where is the line between “Theft Bot” and “Search Engine”? Search Engines have to get their content from somewhere. Is the difference that search engines obey, do not crawl requests?

    A good example of where web scraping can be used appropriately is all the travel websites, that have agreements with the airlines.

    Web scraping is actually a form of meta search.

    The problem that we see here, when we site court cases, is, court cases only happen when one party is scraping another’s website without permission, and presumably refuses to stop.

    Ill agree that some individuals can use the technology for harm, but Im not sure we should be adding to the idea that this practice is all bad.

    Paul Arneson


Please enter your comment!
Please enter your name here