How to Block Bots with Your .htaccessfile

Posted 29/03/2018

In PBN

Blocking robots is a major part of ensuring your website’s safety. It is a necessary part of site maintenance, one that you simply cannot do without. This article presents a quick explanation on what makes this process so vital and how it is done.

Understanding How Bots Work

Before anything else, we need to give readers a definition of exactly what we mean by Bots. For those who are unfamiliar with the term, bots are essentially computer programs which go through the different websites in order to execute a myriad of different tasks. For instance, bots are a crucial aspect of search engines. They are in charge of indexing web pages in order to take note of the relevant terms so that the search engine can determine your rankings. Most affected by these are blog websites.

However, it should be noted that not all bots are harmless. These bad bots are there to search for security vulnerabilities. For instance, some of them will try to sift through your website in search of web forms and user email addresses. This will be used to you and users spam.

That being said, it is vital that you install htaccess block bad bots. Just keep in mind that to htaccess block all bots could prove to be a real challenge. After all, there are innumerable bad bots now. Of course, this only proves how urgent this task is.

Identifying the Bot You Want to Block

As earlier mentioned, you do not want block all bots htaccess. You just want to block the bad ones. There are a few tasks that you need to do before you are able to effectively htaccess block spam bots. The first priority task should be to find its origin. This can be done either by finding its IP address or by identifying the specific User Agent string that it is using.

In order to do these things, you will need to look into your site’s web log and search for the bot manually. Simply download the log from your web host and uncompress it. Take the file and open it with your preferred plain text editor. Combing through the logs may seem like an arduous task but it is necessary. It would also help if you knew the time it hit your site or the specific page it tried to access as this will give you something to go on.

Handling the Bot’s IP Address and User Agent String

Once you have done this and located the entries belonging to the bot, you will need to look for the IP address and the user agent string. This is the next step that needs to be done in order to block spam bots htaccess.

Before you proceed though, it is vital that this part of the process be approached with caution. If you do not approach the problem correctly, then you might be doing more harm than good to your website. To help you with this task, you need to keep in mind that the IP address is a series of numbers. It is made distinct by the fact that they are separated by dots instead of spaces. On the other hand, the term User Agent string pertains to the program used to access your site. It is also worth noting that you do not need the entire sequence for the user agent string, you just need to find the specific part of it that is unique to that particular bot.

Readers need to understand that there is no guarantee that you will be rid of a bot permanently after blocking a particular IP address. As you may well know, viruses and malware are specifically designed to infect computers of all types. They can migrate from one system to another with ease. By carelessly blocking an IP address, you may actually be blocking a real user. In worst cases, you may even end up banning multiple users and a whole set of potential customers simply because you blocked the IP address. To make things worse, you are still vulnerable to that particular bot as it can access your site from a new IP.

Similarly enough, bad bots also intentionally make use of common User Agent names. They usually make use of names which are used by ordinary web browsers, utilizing it as a form of camouflage. This makes the task more grueling as it makes the bot harder to identify. Like with IP addresses, blocking them carelessly would be a mistake as you may also be blocking all the users who are using a particular browser.

Downloading your .htaccess File

Upon learning the bot’s IP address or user agent string, you will need to take the necessary measures to block it. First, you will need to connect to your site through an FTP or SFTP client. Once there, access the top web directory and go to where your home page is located. You will need to look for the .htaccess file. If it is there then simply download.

If it is not there, then you need to ensure that it is not simply hidden from your view. While the process of finding it differs depending on the FTP program being used, the gist of it remains the same. Just set a Remote file mask in the program’s options. This may require you to log off beforehand. Once you have enabled the mask, log in to the program to check for the .htaccess file again.

Now, there is no need to worry if you are still unable to find it despite following the steps presented. Most web hosts do not include it in their default setup anyway. However, this means that you will have to create one yourself.

Creating and Uploading the .htaccess File

A .htaccess file is necessary if you want to block bots effectively. Whether you are creating one from scratch or editing an old file, you will need the technical expertise of the professionals. We won’t bore you with the technical stuff and the specific code lingo. This is where we come in. We are more than adept at working it out for you.

Once that is done, it is simply a matter of integrating the new code to the .htaccess file, saving it, and uploading it to the website. In order to upload the new file to the web server, we will be making use of an FTP/SFTP program.

Blocking bots is an important aspect of site security. However, it needs to be done carefully and correctly. Fortunately, we would be more than willing to do it for you.