Re: .gov crawl

[Carl Malamud <xxxx@media.org>]

ok.  i have an idea.  i have to get to the office and do a couple  
phone calls, but let me write up a suggested approach.  i think  
analyzing his dump is a good stand-alone exercise and yields a variety  
of longer-term benefits.  so, this will be phase 1.

i'll write something up mid-day for you to look at and see if you  
agree ...

On Nov 11, 2008, at 7:53 AM, Aaron Swartz wrote:

> > how current is that crawl?  I really have issues with the way  
> > brewster does
> > crawls ... it has never been very timely or complete and he has  
> > really
> > strange restrictions on use.
>
> They just finished it and they're starting QA. I think things have
> changed a lot since Gordon took over.