On Thu, 2007-09-13 at 08:39 -0500, Bruno Wolff III wrote:
On Wed, Sep 12, 2007 at 20:07:42 -0400,
Matthew Flaschen <matthew.flaschen(a)gatech.edu> wrote:
> >
> > I have directories with several million files in them.
>
> Just curious...what for?
Legacy, they need to be fixed.
I am tracking IP addresses that sent me spam and blocking them and I save
a file for each IP address with the headers in it in case I want to check
on things later. When I started doing that it took a long time to get
hit bit 100K unique addresses. Now it takes a couple of months to get hit
by over a million unique addresses. I don't check as often as I used to because
even doing ls -f | wc takes several minutes to run. Not too mention that each
file takes up 8K of disk space on ext3 file systems.
I have been meaning to rewrite my tracking system to put everything in a
single (or perhaps several) files to make dealing with the data less
burdensome. I just haven't gotten round to it yet.
Bruno,
this is screaming "database me".
Andy