Re: distributed filesystem

[Carl Malamud <xxxx@media.org>]

SHARDS looks really nice ... unfortunately, I'm not the right person to be 
writing that letter. For things like file systems and operating systems, we're 
not leading edge. I use whatever the big boys are using. I spin less than .5 
petabytes in a fairly simple configuration.

The kind of people you want writing the letter are people like Paul Vixie at 
ISC or Brewster Kahle at Internet Archive. Their opinion would carry much more 
weight.

It does sound perfect for some of our massive video archiving things .... 
brewster is doing 24x7 video recording and this sounds like exactly the kind of 
thing that would make his eyes light up.

I would think there are probably some great home/soho implications to this. 
I've got a half-dozen Drobo boxes that I use for staging, and one of the big 
things with them is when there is a disk failure or you need to expand, it 
takes forever to rebuild the file system before it is usable again. Something 
where you can start ripping drives off might be a real breakthrough for that 
market.

Best regards,

Carl

On Jun 3, 2011, at 1:56 PM, Aaron Swartz wrote:

> Hi, Carl --
> 
> My dad (cc-ed) is working on a distributed file system for robust storage of 
> very large collections of data -- which, if it works, he thinks would be 
> great for storing bulk.resource.org-style stuff. (Demo: A filesystem spread 
> across ten drives. He rips out the drives, one-by-one, and replaces them with 
> blanks. A large video file keeps streaming off the filesystem without missing 
> a beat, the data was replicated to the blank replacements so fast.) He's 
> currently applying for a small business grant to help fund the development 
> and was wondering if he could include a letter from you saying you'd be 
> interested in using it.
> 
> I've attached a short overview; if you're interested you two can talk in more 
> detail.
> 
> ---------- Forwarded message ----------
> From: Robert Swartz <xxxxxxx@interaccess.com>
> Date: Fri, Jun 3, 2011 at 4:40 PM
> Subject: For Carl
> To: Aaron Swartz <me@aaronsw.com>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> <blurb3.pdf>