Karsten 'quaid' Wade wrote:
> On Fri, Nov 07, 2008 at 11:43:30AM -0500, Michael DeHaan wrote:
>
>
>> The existing scanner code is pretty solid. I'm happy that it's not
>> regex based in that we can be pretty accurate.
>>
>> However, if we instead process the tarballs, it turns out that the
>> txt.gz file is txt, not HTML, so we run into problems.
>>
>> I'm really not looking forward to rewriting the scanner to use
>> complicated regexen again.
>>
>> I would reason that
>> -- if we know not to even /look/ at previous months we've only scanned
>> -- we use DB transactions (so a month of Fedora-devel is only one SQL
>> statement)
>>
>> then the daily cost of running the scanner even if it does download
>> individual message pages will be acceptably low.
>>
>> The only pain will be running them back to the dawn of mailing list time.
>>
>> I'm going to try to the above first and then we can also see if another
>> db is faster, though I bet if we use transactions even sqlite is
>> sufficiently fast. TBD.
>>
>> Are we ok with the above?
>>
>>
> It sounds sane, given that we'll have a dedicated server working on
> it. We can then plan the direction of the code based on it being an
> always-running service v. an occasional snap-shotter.
>
> I will note that there are some tools specifically designed for
> searching mail archives, such as 'grepmail'. That is, since mail is a
> standard format and we are only looking for things found in the
> headers, maybe the regex wouldn't need to be present or so evil
>
Nice... wasn't aware of that one.
> .
>
> I would only consider doing that if we wanted a dual-purpose tool,
> that is good for snapshots-from-scratch as well as ongoing data
> mining.
>
> Side topic -- Max, if Infra is concerned about staff power, I am a
> junior member of that team who has never tried getting sponsored for
> something. I could pitch in to help design, define, provision, and
> support our system. That would give us the advantage of having
> someone within the team who can e.g. tweak cron settings, roll out new
> code to the server, etc. In the case of a publictest server, the
> developer does have that control, but once it moves to Infra control,
> things get all Puppety. :)
>
>
Would we need RPM packaging and inclusion in Fedora repos for that? I
don't believe all of the infrastructure tools
are packaged.
Mainly I'm pondering how we update the EKG instance. If it's something
we have SSH to, it's just as simple as "git pull", which I like, for
something in active development.
If anyone has /any/ spare server around for this purpose, that might be
somewhat easier than having to contend with Puppet for something this
simple.
Thoughts?
> - Karsten
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> ekg mailing list
> ekg(a)lists.fedorahosted.org
>
https://fedorahosted.org/mailman/listinfo/ekg
>
>
_______________________________________________
ekg mailing list
ekg(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/ekg
Also, I've just added code to avoid fetching the month pages and walking
through them if a month has already been completely scanned.
The current month will always be rescanned.
This should be perfectly happy for cron usage now.
Assuming we get our test machine, I think it's safe to remove the check
code that limits the amount of months we go back in time and then let it
do a long run (in "screen", of course) to process everything. I first
want to check to see if changing databases to Postgres makes a
difference though.
I see we've removed fedora-commit-list from the scanning indexes, do we
want to reinstate the code that scans every mailing list we host or
leave them opt-in? Opt-in may actually be ok, as it stands those lists
also tell the graph pages what to include so we can optionally generate
smaller graphs just by temporarily replacing the config file.
Longer term, we should probably have an option to be able to specify
which config file to use, at least while graphing.
An example would be if we were showing the results for a project and
only wanted to show graphs for associated projects.
--Michael