| ||
Wed 2024-11-20
Tue 2024-11-19
Mon 2024-11-18
Sun 2024-11-17
Love, Acceptance, and Screeching Modems Sat 2024-11-16
Fri 2024-11-15
Search
Archives
2023
12 11 10 09 08 07 06 05 04 03 02 01
2022
2021
2020
2019
2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
One good site
MDN
Nelson Minar
Blog licensed under a Creative Commons License
|
Spurred on by
Mark
Pilgrim's
and
Tim Bray's
postings, I looked at the access log for my four week old blog to see
what robots were visiting me.
No fewer than 71 different hosts grabbed my robots.txt in eight days. Two creepy ones: NameProtect (trademark enforcement) and TurnItIn (anti-plagiarism). This doesn't bug me too much. I do wonder why some of these crawlers felt it necessary to fetch robots.txt forty times in eight days. |
|
