Michael DeHaan wrote:
The existing scanner code is pretty solid. I'm happy that
it's not
regex based in that we can be pretty accurate.
However, if we instead process the tarballs, it turns out that the
txt.gz file is txt, not HTML, so we run into problems.
I'm really not looking forward to rewriting the scanner to use
complicated regexen again.
I would reason that
-- if we know not to even /look/ at previous months we've only scanned
-- we use DB transactions (so a month of Fedora-devel is only one SQL
statement)
then the daily cost of running the scanner even if it does download
individual message pages will be acceptably low.
The only pain will be running them back to the dawn of mailing list time.
I'm going to try to the above first and then we can also see if another
db is faster, though I bet if we use transactions even sqlite is
sufficiently fast. TBD.
Are we ok with the above?
--Michael
_______________________________________________
ekg mailing list
ekg(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/ekg
Transactions are helping.
Also a bit unexpected -- just cutting down the amount of prints by 20x
seems to speed it up a good bit too.
No benchmarks yet, but it feels about 3x faster.
I'll see about some parallelism and db changes too. I'm sure the mail
server won't mind and it's not using close to 100% CPU yet.
--Michael