Robots txt generator
- Categoría: Web Developer Tools
- Licencia: Freeware
- Tamaño: < 1 KB
- Sistema: w2000 wXP vista
- Idiomas:
Are you fed up of the spiders that visit your website and consume bandwidth?
Spider robots are usually search engines that look in your website and link the content. But this has a negative side, as they consume a lot of your website`s bandwidth. The reason why a program like Robots txt generator exists is to control and restrict these spiders and their access to your website.
First and foremost Robots txt generator is a web utility, totally free, very small and simple to use (it doesn`t even need installing), that lets you limit the number of consultations a spider can do, per minute or hour, within your website. The program database comes equipped by default with a list of 300 spider bots, but you can add new ones if you want, and also specify different restrictions for each bot.
For example, in Robots txt generator you can let a Google Bot have access, but not a Google Image Bot.
DownloadRobots txt generator related downloads
Website Access Manager 5.0
DownloadA web utility to manage access rights for users of a specific website.