[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Message 00417: Re: .gov crawl

ok. i have an idea. i have to get to the office and do a couple phone calls, but let me write up a suggested approach. i think analyzing his dump is a good stand-alone exercise and yields a variety of longer-term benefits. so, this will be phase 1.

i'll write something up mid-day for you to look at and see if you agree ...

On Nov 11, 2008, at 7:53 AM, Aaron Swartz wrote:

how current is that crawl? I really have issues with the way brewster does crawls ... it has never been very timely or complete and he has really
strange restrictions on use.

They just finished it and they're starting QA. I think things have
changed a lot since Gordon took over.