The existing scanner code is pretty solid. I'm happy that it's not
regex based in that we can be pretty accurate.
However, if we instead process the tarballs, it turns out that the
txt.gz file is txt, not HTML, so we run into problems.
I'm really not looking forward to rewriting the scanner to use
complicated regexen again.
I would reason that
-- if we know not to even /look/ at previous months we've only scanned
-- we use DB transactions (so a month of Fedora-devel is only one SQL
statement)
then the daily cost of running the scanner even if it does download
individual message pages will be acceptably low.
The only pain will be running them back to the dawn of mailing list time.
I'm going to try to the above first and then we can also see if another
db is faster, though I bet if we use transactions even sqlite is
sufficiently fast. TBD.
Are we ok with the above?
--Michael