How to Fix Performance Issues Caused by Bots and Crawlers
First: do you have a problem? Generally, performance for regular shop visitors might be affected if you have more than 10 bot pageviews per minute or bot traffic exceeds 50% of total pageviews. MageReport Premium offers a handy graph that displays the summary of live bot traffic. When your shop has at least 1600 PHP requests in one day and 30% of those are bots, the bot traffic check in MageReport comes out red.
Among the thousands of shops on our platform, excessive bot traffic is the number one reason for bad Magento performance. Often, bots get stuck in the layered navigation feature of Magento. This causes the bot to crawl every combination of every product in combination with other parameters and is now creating millions of URLs to index. Every single page needs to be generated, which will cost a lot of resources.
Layered navigation pages do not require crawling, and in fact, their indexation could produce a penalty for your search engine ranking, as it produces a lot of duplicate content. So you are advised to resolve this, both for performance and SEO reasons.
With these four measures, you will resolve this situation completely.
How to Block Abusive Bots (If Any)
If MageReport says you are high on bot traffic, log in to your Hypernode to determine which bots are hammering your shop. This command will show you the top 10 bots that visited your site yesterday:
$ pnl --yesterday --php --bots --fields ua | sort | uniq -c | sort -n [...snip...] 1902 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 3796 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) 12813 Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
In this example, there were almost 4K Bingbot pageviews, 2K Google pageviews and almost 13K MegaIndex pageviews. So you could eliminate a large chunk of load by blocking MegaIndex (a shady crawler whose benefits to you are disputable). Here are instructions on blocking specific bots on Hypernode.
How to Educate Bots Using Robots.txt
Legitimate bots will honor instructions from your robots.txt. However, it will take some time before crawlers notice the changes from this file. An example robots.txt for Magento on Hypernode van be found here.