The BOT Story

[dan]

I did a little experiment to verify how bots can effect traffic.

Some background. I use .htaccess to control how bots interact with my site. There are good and bad bots. For example, google spider is a good bot -- its job is to index your website for google search engine.

What is surprising is how many bad bots there are -- there are tons. These bots are usually used to gather statistics or for spamming purposes. A little research and you can find a long list of bad bots. Most webmasters make an effort to stop these bots because their function has no significant impact on your site except to eat up bandwidth, slow down your site or in some cases cause problems or steal information.

A lot of webmasters try to stop bad bots through meta tags -- which may or may not work. Some webmasters use robots.txt file to do the job. I prefer .htaccess. It seems more effective, but robots.txt may do the job as well.

As I expected, when I commented out my bad bot list, my traffic stats increased significantly. Even though bots are not human beings, but programs accessing your site, they still effect your site statistics. But what good are these stats if it doesn't turn into action (i.e. becoming a member).

The reason for my experiment was a friendly bet with another webmaster and to satisfy another theory. My friend's point was that meta tags could do the job as well as .htaccess file. Nope, it did not. If I'm wrong, than it is because I don't know how to effectively use meta tags. Your feedback would be appreciated.

I'm now back to blocking out the bad bots. However, there are new ones being harvested daily. If only there was an easier way to discover and stop these bots. If you are another webmaster who found an effective way to get rid of these creatures, please drop me a line or post a comment in the feedback section of the forum.

The Database Problem

There was a recent issue that we discovered about site registration. It seems our site was being slowed down, bottle necked, etc during certain hours of the day (mostly at night during my bed time hours). We are 99.9% sure it was a bot programmed to crawl my site at a certain time of the day. This bot would eat up bandwidth to the point that it slowed down the site. What was a little puzzling, is how it effected the registration process while other areas of the site did not seem to be effected.

I went almost a week without realizing it was a bot. However, there is that chance that this bot's intention was not to cause harm. Can you imagine if a search engine bot slowed my site down what a problem that would be. What would I do? I can't afford to stop a search engine bot from indexing my page -- however, this is was not the case. We are almost 99% sure this bot was not a spider for a search engine (100% sure it was not a bot for a major search engine).

Since a bot is nothing more than a script, it could indeed be a good bot but because of the way it was written, it may not interact with the coding on your site in the same way it does on another site. For that reason, I won't label this bot as a bad or good bot but will keep my eye open for further information. I have not found it on any bad list while there are some bots whose destructive nature have been reported by many and deserve to be labeled as a bad bots.

Know your bots!!!!!!!!!!

Cheers
Dan

[dan]

I got an email concerning the bot story. The person did not want to register to post, so let me point out that there is a certain section of the forum that ALL can post (even visitors) without having to register. Everything under the category “Visitors and Members can Post -- No REGISTRATION or LOG-IN needed” is open for ALL to post without having to register.

Good email – I would like to answer or comment on some of his points.

1. You stated that robots.txt was better than .htaccess because .htaccess can slow your site down. Interesting! I never heard this but will investigate further. Believe it or not, as easy as robots.txt may be, I’m use to using .htaccess even though it is probably harder to use. I have not noticed any speed problems using .htaccess but I was never made aware that there may be a speed difference. It is worth checking out so I will.

2. Spyware, virus, etc. Here I disagree with you or misunderstood your point. I do have programs to detect spyware, virus, etc on my computer. But it only scans and detects problems on my computer, not on my server. It is my hosts job to make sure there is no viruses on the server and usually viruses is not an issue. They should have adequate software/security to prevent this from happening, but anything can happen (ask Microsoft, IBM, and the military). Once (over a year ago) I did have a viruses on my server (Ipower). Once detected, Ipower rid the server of the virus (note: I don’t want to debate shared v.s. dedicated servers here). Ipower had to do it, the software I had for my computer is not on the server so how would it detect or rid it of a problem on the server? Best scenario it never happens, but if it does, your host should take immediate action.

3. You asked if there is software that detected if a bot was bad or good. Not that I know of. Again, I got the impression you are mixing up server problems with local computer problems. Bots usually don’t reside on your computer (if they do they are probably not called bots but viruses). The only way I know to determine if a bot is good or bad is to research. Your logs will show you what spiders/bots crawled your site. If they do not have a name (some logs call unknown bots “other”) you can do your research by IP address.

Ok, I could be all wet or wrong with my info. I did reply to your email to let you know I was going to comment/answer your email on the forum. I thought it was good info for all to read or discuss. If you or anyone else has further comments, don’t be afraid to post on the forum. Thanks for your info.

Dan