On Fri, Nov 07, 2008 at 11:43:30AM -0500, Michael DeHaan wrote:
The existing scanner code is pretty solid. I'm happy that
it's not
regex based in that we can be pretty accurate.
However, if we instead process the tarballs, it turns out that the
txt.gz file is txt, not HTML, so we run into problems.
I'm really not looking forward to rewriting the scanner to use
complicated regexen again.
I would reason that
-- if we know not to even /look/ at previous months we've only scanned
-- we use DB transactions (so a month of Fedora-devel is only one SQL
statement)
then the daily cost of running the scanner even if it does download
individual message pages will be acceptably low.
The only pain will be running them back to the dawn of mailing list time.
I'm going to try to the above first and then we can also see if another
db is faster, though I bet if we use transactions even sqlite is
sufficiently fast. TBD.
Are we ok with the above?
It sounds sane, given that we'll have a dedicated server working on
it. We can then plan the direction of the code based on it being an
always-running service v. an occasional snap-shotter.
I will note that there are some tools specifically designed for
searching mail archives, such as 'grepmail'. That is, since mail is a
standard format and we are only looking for things found in the
headers, maybe the regex wouldn't need to be present or so evil.
I would only consider doing that if we wanted a dual-purpose tool,
that is good for snapshots-from-scratch as well as ongoing data
mining.
Side topic -- Max, if Infra is concerned about staff power, I am a
junior member of that team who has never tried getting sponsored for
something. I could pitch in to help design, define, provision, and
support our system. That would give us the advantage of having
someone within the team who can e.g. tweak cron settings, roll out new
code to the server, etc. In the case of a publictest server, the
developer does have that control, but once it moves to Infra control,
things get all Puppety. :)
- Karsten
--
Karsten 'quaid' Wade, Community Gardener
http://quaid.fedorapeople.org
AD0E0C41