EKG Status Report 2009.02.27
by Yaakov Nemoy
Hey List,
I'm up to 17.9 hours, still working on scanner, but in thinking about
how there were comments for the general case data sources, i did a bit
of refactoring. There are alot of wrinkles to be ironed out, but i'm
working on making it possible for more behavior to be defined in the
model, rather than through some iterative process. The end result is
that most of the bits that were in scanner are in the model now.
This became important when i realized that pipermail on fp.o does not
save all the mbox archives as gzip, but some are just plain text. It
got me thinking of a simple way to handle this. I'm not sure this is
the 'simplest' way of handling it, but it should be general purpose
enough for us to handle lots of other things to, like:
* Downloading emails from a web page rather than an mbox
* Some benefactor that provides us SQL, Imap, or Maildir dumps, rather than mbox
* Using a wiki page as a source, and the history is a set of facts
* Git and CVS
Not that i actually coded in support for this, it just means the code
you have to write to handle these cases does not need to be in a
million and one places at once.
--Yaakov
15 years, 3 months
EKG status report 2009.02.25
by Yaakov Nemoy
Hey List,
Here's where i stand with EKG so far. According to my handy bean counter, i've spent a total of 14 hours on EKG so far. This puts me behind schedule just a bit. I think i predicted about 16 hours to redo the scanner, and another 4-8 for the grapher. (Max, that's an official 2 weeks, like we discussed.)
So here's what i have going on right now.
I've broken down what we're looking at into 'sources' which are places where things happen, and 'facts' which are actions someone undertook. Each fact has one source. Since the details of the data model aren't hammered out, i'm going to add getting sqlalchemy-migrate setup to my todo list.
So far there's an abstract base class called mailman, that handles getting gzipped mbox files of logs from various sources, and then there are some simple functions that break it down into facts in the database. The reason for this is that in order to write a new mailman for a new source, you simply need to feed it some URLs, and a regex or two it can use to parse the web pages that link to the mbox files. This idea can be extended to other ways of getting facts from other sources. Right now, since we're focused on mailing lists, it uses a few idioms related to that, which will have to be factored out at some point.
As for as the code base goes, it needs all the usual refinements, like using the config file i nicely set up, setting up a python egg so we can package and install it, a bit of refactoring some nasty bits, and probably performance help.
I'll have a stab at making sure we can parse pretty much every mailing list on the v2 settings list tomorrow morning.
-Yaakov
15 years, 3 months
EKG status 2009.02.25 evening edition
by Yaakov Nemoy
Hey List,
Like i said in my previous email, i put in the bits for mailing lists in a config file. This brings grapher up to a reasonable feature parity with the ruby version. At 15.4 hours, i've come in just under schedule, if you overlook the fact that we can't read JBoss lists yet.
Here's how it works.
in the config file, there's a section that looks like this:
[lists]
fedora-wiki = fedoraproject
On the left hand side is the name of the list, and the right hand side is the source. The name from the source can be found in each Mailman subclass as the attribute 'source'. Obviously, we're going to have to document better what sources are available. ;).
This tells the program to use the fedoraproject mailman handler for reading the list fedora-wiki. The rest is taken care of from there. Feel free to port over the old lists diagram to the new one if you want, otherwise i'll probably take care of it tomorrow or friday.
For now, more homework.
-Yaakov
15 years, 3 months
ekg v3 settings.ini
by Max Spevack
[mspevack@localhost v3]$ pwd
/home/mspevack/ekg/v3
[mspevack@localhost v3]$ head settings.ini
scan_mailmen = 0
report_months = 2008-March, 2008-April, 2008-May, 2008-June, 2008-July,
2008-August, 2008-September, 2008-October, 2008-November, 2008-December
mailmen = ,
track_limit = 0.01
limit_months = 12
all_your_base_owner = us
new_fin_year = 2008-March
===
Before we get too far down the path, can we have a discussion about what
each of these settings means, why we need them, whether or not some of
them are deprecated and will be removed, and what/if any alternatives
there are?
Thanks,
Max
15 years, 3 months
EKG design
by Max Spevack
Let's get away from the language debate, and discuss the design and
architecture of the code. Maybe the language falls out of that
discussion.
=======
Ignoring what currently is implemented in v1 and v2, here is what I care
about:
(1) Pulling data down from wherever *exactly once* and storing it in a
database forever after that.
(2) The primary feature of the config file is a "list of projects",
which represents the projects that EKG will analyze. For each project
in the list, the pieces of data that are collected will probably expand
over time, so it's important to take this into account when we
architect. Maybe we're only starting with mailing lists, but we'll
eventually expand to commits, maybe to bugs, etc. Basically I want the
program to think of something like Spacewalk not just as the spacewalk
mailing list(s), but the various other sources of data that exist.
(3) Having an "update my data" feature that basically looks at the list
of projects that EKG cares about, and can say "well, my data on
Spacewalk hasn't been updated since January 12th, so let me go get all
the new data" or "oVirt has just been added to the list, so let me go
and get everything that exists".
(4) Being able to analyze the data in time slices. Monthly, quarterly,
and yearly are what interests me most. I'd like to be able to get that
either from a calendar point of view (year starts on January 1) or a Red
Hat fiscal year point of view (year starts on March 1).
(5) Some use cases:
* There are 100 projects in the database. Show me the report of the
last quarter for all of them.
* Here is a list of 10 projects that are most important to Brian
Stevens. Show me the breakdown for the last two years, with quarterly
deltas.
* Show me all projects where the external mailing list percentage is
greater than 50%.
* Show me all the projects where @redhat.com is committing more than 75%
of the code.
* Show me the projects where the total volume of mailing list traffic
has increased/decreased by more than 25% since the last quarter.
15 years, 4 months
summary of FOSDEM ekg discussion
by Max Spevack
Yaakov and I spoke about EKG at FOSDEM.
I'll let him lay out some of the roadmap/feature stuff we talked about,
but I want to propose a larger question to the list, and especially to
Michael:
The first item on the agenda was "is Ruby really the language that is
going to make this project accessible to a wider number of contributors,
and most maintainable for the future?" followed by "if we have a clear
list of enhancements and/or new features, we should decide if Ruby or
Python is this project's future before we write them."
So, in short:
Michael, how much of your own personal time do you see yourself devoting
to EKG in the next 3 months? I say 3 months, because that is the
timeline for development that I want to work on in order to have EKG
meet the CommArch team's goals. Do you still see yourself as the lead
developer of this project, or is it up to CommArch to find the
engineering resources (which for the moment will be Yaakov) to take this
project further?
And somewhat separate from that question, is it worth spending (at
Yaakov's estimation) 10 days or so to rewrite the core of the code into
Python, setting it up for a better future?
--Max
15 years, 4 months
Scanner
by Michael DeHaan
Yaakov and others,
Before I said let me handle the scanner, and it's clear I'm way too
busy, so if you are interested in working on the scanner, you are
welcome to do so.
I would suggest we make it do the archive tarballs and see how it speeds up.
External lists also is on my wish list. (What's the LKML mix?
Gnome? Nagios? Etc). This may take some doing where formats differ.
--Michael
15 years, 4 months
Wish list for features?
by Yaakov Nemoy
Hey List,
I've been going through EKG's grapher finding ways to optimize some of
the SQL queries, but it seems that no matter how many bench marks i
run, i can't make it inherently faster. So now it's time to work on
phase two (my phase two.)
What features do you guys want. Namely, what reports do you want to
see? What do you want quantifiable? Give me features to implement.
-Yaakov
15 years, 4 months
Trac admin plz
by Karsten Wade
Can someone bump my access to the EKG Trac instance up to 'admin'?
I want to tweak the components, milestones, versions, and so forth.
k?thx!bai!! - Karsten
--
Karsten 'quaid' Wade, Community Gardener
http://quaid.fedorapeople.org
AD0E0C41
15 years, 4 months