Bots and You: What Are They, and Should You Care?

Last Updated on August 5, 2022 by SERP Kingz

I reference “bots” a lot in my articles. As a site owner, you may be wondering, “What exactly are bots?”

“Are bots good for my site, or bad?”

And most of all, “Do I need to care about any of this, or can I go do literally anything else?”

I’ll answer all these questions—and many more you didn’t ask—in this comprehensive guide to bots and how they affect your site.

What is a bot?

Okay—brass tacks time.

A “bot” is just a software application that runs automated tasks (called “scripts”) over the internet.

Now before you say anything, I know that definition probably didn’t help much, so let’s break it down a bit further:

• Software: a bot is a piece of software, like Microsoft Word or Adobe Photoshop.
• Application: a bot is designed to perform a specific function or task.
• Automated: a bot carries out its task without human intervention.
• Tasks (scripts): a bot’s task is typically repetitive, like visiting websites or checking prices.

So, what does that mean for you as a site owner?

• If you run an e-commerce site, bots can be used to scrape pricing information from your competitors (or from you).
• If you run a news site, bots can be used to gather data and compile stories.
• If you run a blog, you’re probably most familiar with bots as the things that leave asinine comments like “Great post! I have found this to be a great resource.”

The list goes on, but you get the idea. If you run any kind of online store or business site, bots are playing some role in it, whether you’re aware of it or not.

In fact, they’re probably on your site right now.

Bots can be used for good or bad intentions, depending on what they’re designed to do, and what entity/organization is controlling them.

What is bot traffic?

Separating your site’s human traffic from bot traffic is important to having a good idea of how many people are visiting your site, but it can be tricky; in many cases, bot traffic is designed to look human. (Google, to their credit, filters out known bot traffic in Google Analytics 4.)

Basically, any traffic on your site that doesn’t come from a human is considered “bot traffic.” That includes everything from search engine crawlers to malicious scraper bots of various affiliations.

Some bots are benign and helpful, (like Google’s crawlers), others are generally bad (like email harvesters), and some are basically evil (like the ones that try to battering ram your WordPress login page every day—check your server logs).

Bot traffic isn’t all bad by any stretch. There are bots that are beneficial for your site. Bots that you want to be on your site.

But sort of like hearing someone talk about “good bacteria,” or “good insects,” the net’s also teeming with “bad” bots.

And outside of SEO, those harmful bots are usually the ones that people are talking about when they use the term “bot.”

Here are a few examples of the kinds of bots your site may encounter:

The “Good” Bots

These are the ones that you want on your site. They help you in some way, and don’t cause any problems.

• Search engine crawlers: These are the bots that visit your site and index your pages for search engines. They help people find your site when they’re looking for something online.
• Monitoring/Analytics bots: These are the bots that collect data about your site’s traffic and performance. They help you understand how people are using your site, and keep an eye out for things like server downtime and errors.
• Feed/aggregator bots: These can be good or bad, depending on the situation, and where the bot actually comes from. They help more people see your content by checking your RSS feed and displaying your latest content in their own feed, which is great—if that’s what you want.

The “Bad” Bots

These bots don’t really have a good side, they were created with malicious intent, and can cause all sorts of problems.

• Content scrapers: These are the bots that copy your content and republish it on their own site. They’re basically stealing your hard work, and can cause problems with search engines if they republish your content without adding a noindex tag.
• Comment spammers: These are the bots that leave spammy comments on your blog posts in an attempt to get mass backlinks to their own site. That’s basically it.
• Email scrapers: These are the bots that crawl your site specifically to harvest email addresses. They then sell these email addresses to spammers, who will bombard you and everyone else on the list with junk. That junk can be basic mass advertising, or it can contain malicious links to try to steal your information.

The “Ugly” Bots

These bots are the worst of the worst. Unlike scrapers and comment spammers, they don’t even pretend to have constructive uses. They’re all about causing harm, and you protect your site from them.

• Hacker bots: These basically do what it says on the label. They try to gain access to your site for various purposes, by pretending to be a human user. Once they’re in, they can wreak all sorts of havoc, from defacing your company’s site to stealing sensitive information.
• “Zombie” bots: Not a single bot, but a swarm of them, all networked together. Networks of these things are used to carry out denial of service (DDoS) attacks. They work by flooding a server with so much traffic that it overloads and crashes. This can be used to take down a site, or just make it inaccessible for a period of time (sometimes for days at a time). Possible purposes behind a bot attack could be to take a competitor (i.e. you) offline temporarily, or to ransom your site in exchange for Bitcoin.

Why you should care about them:

Bots Affect Your Search Engine Rankings

Good bots help your content get indexed and ranked in search engines. Bad bots scraping your original content and republishing it sloppily, without permission, can lead to duplicate content issues, which in turn can hurt your site’s search engine rankings.

Bots Are a Security Risk

Bad bots can be used to carry out all sorts of malicious activities, from spamming your comment section to trying to brute force their way into your WordPress admin panel.

Bots Affect Your Site’s Performance

A swarm of bots can overload your server and take your site down, or slow it to a crawl for extended periods of time. Not good if you run an e-commerce site.

Bots Can Affect Your Customers’ Experience

Your human users can be bombarded with spam comments, or malicious links that try to steal their information.

Should You Block Bots?

The bad ones, sure. Definitely. The good ones, probably not. (Google’s crawlers, definitely not.) If your site is getting excessive traffic from supposedly “good” bots that don’t benefit you or your target audience/customers in any way, you might want to rate-limit them or ban their IP addresses so they don’t bog down your site.

Usually this isn’t necessary, though, as well-behaved bots will generally respect your site and back off long before they start to overload your server. Flukes do happen, though, so it’s something to keep an eye on.

Should You Help Some Bots?

You definitely want to make things as easy as possible for Google’s bots. That means, at a minimum, having a sitemap, and making sure your site is well-structured and easy to navigate. There’s a lot more to it than that, but those are the basics.

In some cases, you can help search engine bots (and your own site) by blocking them from certain areas of your site where they shouldn’t be anyway (i.e. admin areas, test pages, etc.), or from redundant content like your on-site search result pages.

Another tip: Unless you have tons of discussion going on in your comments section, or a very specific use case, get rid of the RSS feed for your blog comments that WordPress automatically creates. No one wants to look at that feed; not even bots. But they will waste a lot of time and energy (electrical energy) crawling your blog comments RSS over and over again for no reason, if you let them.

But, in general, good bots are not something you have to worry too much about one way or the other.

As for other bots, it’s up to you. If you want your content to be scraped and republished without your permission, that’s your call. In general, though, it’s a good idea to err on the side of caution and make sure your site is as resistant to bad bots as possible.

The Bottom Line

Bots are everywhere. Some of them are good, some of them are bad, and some of them are downright ugly.

The good ones help you with things like SEO and analytics. The bad ones can spam your comment section, cause SEO problems, or even break into your site.

If your site is online, you’re going to have to deal with both.

One of the most straightforward ways to defend a WordPress site from malicious bots is to be proactive and use a security plugin that will help you stop the bad ones in their tracks.

You should also keep an eye on your site’s performance and security logs, so you can spot any issues early on and take action accordingly.

If you get massive traffic spikes without any increase in conversions, leads, or customer interaction… there’s a good chance they’re caused by bots.

Services like CloudFlare can also help, by blocking malicious IP addresses and rate-limiting requests from known botnets.

The downside to services like these is that if a real human (i.e. potential customer) visits your site, and the service thinks they’re suspicious for some reason, they may get delayed for a few seconds while the service pulls them out of the security line and makes them go through the metal detector again.

You know how incredibly impatient and easily distracted humans can be on the web these days.

Potentially slowing some visitors down for a few seconds with a warm “we’re checking you… because we don’t believe you’re human” screen before they can even look at your site is something to keep in mind. (It’s not an uncommon occurrence, either.)

Hopefully I’ve given you a better understanding of what bots are, why they exist, and how to deal with them.

But back to the most important question: Should you care about bots?

If you have an online storefront for your business, an affiliate site, or an e-commerce site that’s generating revenue for you, the answer is “yes.”