Re: Detecting and refusing crawlers/robots

> Peter,
>
> Don't the 'bad robots' typically skip the robots.txt file? If they  
> don't
> read the file, how would your proposal work?
>
> I would think the only way you could do this is compare your  
> robots.txt
> readers to bot-like requests in your log files to find the bots that
> aren't playing by the rules then automatically blacklist them. Or is  
> that
> what you are proposing?
>
> -- Brad Perkins
>
>> David,
>>
>> that's for the good ones. Check who's reading the robot.txt file and
>> put them automatically on your "Is_a_bot-list". Save the IP address
>> _and_ the browser client. For my own system I generated so a list of
>> possible robots and just send them a "bad-robot-page" from the OWA.
>>
>> Am 11.04.2009 um 18:15 schrieb David Ringsmuth:
>>
>>> Is there a better way to solve this problem beside using the HTML  
>>> and
>>> robots.txt?
>>>
>>> <META NAME="ROBOTS" CONTENT="NOINDEX">
>>>
>>> <META NAME="ROBOTS" CONTENT="NOFOLLOW">
>>
>> Mit freundlichen Grüßen
>> [4D-Consulting.com]eK, Wiesbaden
>> Peter Schumacher
>> --------------------------------------------------------
>> Web: http://www.4D-Consulting.com/
>> FreeCall:  0800-434 636 7
>> Tel.:      0611-9406.850 - Fax: 0611-9406.744
>> iChat/Skype: PeterInWiesbaden
>> 4D-Consulting.com eK - Scharnhorststr. 36 - 65195 Wiesbaden
>> HR Wiesbaden: HRA 4867
>> Mitglied im Entwicklernetzwerk http://www.die4dwerkstatt.de
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> Active4D-dev@...
>> http://mailman.aparajitaworld.com/mailman/listinfo/active4d-dev
>> Archives: http://mailman.aparajitaworld.com/archive/active4d-dev/
>>
>
> _______________________________________________
> Active4D-dev mailing list
> Active4D-dev@...
> http://mailman.aparajitaworld.com/mailman/listinfo/active4d-dev
> Archives: http://mailman.aparajitaworld.com/archive/active4d-dev/