Michael DeHaan wrote:
The existing scanner code is pretty solid. I'm happy that it's not regex based in that we can be pretty accurate.
However, if we instead process the tarballs, it turns out that the txt.gz file is txt, not HTML, so we run into problems.
I'm really not looking forward to rewriting the scanner to use complicated regexen again.
I would reason that -- if we know not to even /look/ at previous months we've only scanned -- we use DB transactions (so a month of Fedora-devel is only one SQL statement)
then the daily cost of running the scanner even if it does download individual message pages will be acceptably low.
The only pain will be running them back to the dawn of mailing list time.
I'm going to try to the above first and then we can also see if another db is faster, though I bet if we use transactions even sqlite is sufficiently fast. TBD.
Are we ok with the above?
--Michael _______________________________________________ ekg mailing list ekg@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/ekg
Transactions are helping.
Also a bit unexpected -- just cutting down the amount of prints by 20x seems to speed it up a good bit too.
No benchmarks yet, but it feels about 3x faster.
I'll see about some parallelism and db changes too. I'm sure the mail server won't mind and it's not using close to 100% CPU yet.
--Michael