Hi-cap storage question

Fri Jun 18 16:07:41 PDT 2004

Okay, here's one for you high-capacity folks:

We're looking at our storage requirements right now.

We have two needs:  short-term storage and access on live media, and
long-term storage on archival media.

Simple, right?

Well, maybe not.

You see, the amount of data is a bit larger than what most people are
used to dealing with.  It's roughly an exabyte.

We won't be generating it all at once, but that's our
short-to-medium-term need.  Right now, today, we're storing/archiving
a terabyte every 20 days, and getting by with hardware RAID (only
keeping about 100-200GB live, and rotating the available partitions on a
FIFO basis) and LTO1 jukeboxes.

We expect that need to double within a year, and within two to three?
years, jump exponentially.

Right now, the vast bulk of the data is stored in flat files, which does
not present a problem for us at the moment, neither in terms of access
(it's write-only during collection), or inode count (the number of files
is small and finite).  Moving to a database isn't out of the question,
but if we did, "free" is a very good word.

Thoughts on the physical storage issue?

-- 
Mark C. Langston                                    Sr. Unix SysAdmin
mark at bitshift.org                                       mark at seti.org
Systems & Network Admin                                SETI Institute
http://bitshift.org                               http://www.seti.org