Peter,
Don't the 'bad robots' typically skip the robots.txt file? If they don't
read the file, how would your proposal work?
I would think the only way you could do this is compare your robots.txt
readers to bot-like requests in your log files to find the bots that
aren't playing by the rules then automatically blacklist them. Or is that
what you are proposing?
-- Brad Perkins
> David,
>
> that's for the good ones. Check who's reading the robot.txt file and
> put them automatically on your "Is_a_bot-list". Save the IP address
> _and_ the browser client. For my own system I generated so a list of
> possible robots and just send them a "bad-robot-page" from the OWA.
>
> Am 11.04.2009 um 18:15 schrieb David Ringsmuth:
>
>> Is there a better way to solve this problem beside using the HTML and
>> robots.txt?
>>
>> <META NAME="ROBOTS" CONTENT="NOINDEX">
>>
>> <META NAME="ROBOTS" CONTENT="NOFOLLOW">
>
> Mit freundlichen Grüßen
> [4D-Consulting.com]eK, Wiesbaden
> Peter Schumacher
> --------------------------------------------------------
> Web:
http://www.4D-Consulting.com/> FreeCall: 0800-434 636 7
> Tel.: 0611-9406.850 - Fax: 0611-9406.744
> iChat/Skype: PeterInWiesbaden
> 4D-Consulting.com eK - Scharnhorststr. 36 - 65195 Wiesbaden
> HR Wiesbaden: HRA 4867
> Mitglied im Entwicklernetzwerk
http://www.die4dwerkstatt.de>
>
>
>
>
>
>
> _______________________________________________
> Active4D-dev mailing list
>
Active4D-dev@...
>
http://mailman.aparajitaworld.com/mailman/listinfo/active4d-dev> Archives:
http://mailman.aparajitaworld.com/archive/active4d-dev/>
_______________________________________________
Active4D-dev mailing list
Active4D-dev@...
http://mailman.aparajitaworld.com/mailman/listinfo/active4d-devArchives:
http://mailman.aparajitaworld.com/archive/active4d-dev/
