This is a bit further reaching than what we discussed at FOSDEM. My comments as follows.
2009/2/11 Max Spevack mspevack@redhat.com:
Let's get away from the language debate, and discuss the design and architecture of the code. Maybe the language falls out of that discussion.
=======
Ignoring what currently is implemented in v1 and v2, here is what I care about:
I can airlift the design from v2 into this, without any difficulty.
(1) Pulling data down from wherever *exactly once* and storing it in a database forever after that.
That's goal #1 for today.
(2) The primary feature of the config file is a "list of projects", which represents the projects that EKG will analyze. For each project in the list, the pieces of data that are collected will probably expand over time, so it's important to take this into account when we architect. Maybe we're only starting with mailing lists, but we'll eventually expand to commits, maybe to bugs, etc. Basically I want the program to think of something like Spacewalk not just as the spacewalk mailing list(s), but the various other sources of data that exist.
I have a few ideas on how to do this. I'm gonna use a few ideas i figured out in Devshell. (When all you have is a hammer, everything looks like a nail.)
I can create classes that handle certain kinds of data streams, namely, mailing lists, bugzilla, trac, wikis, etc... Then each project is a composite object of these classes. Finally, we can find a way to serialize an object to a config file like file, that can be used to combine pieces like this, without requiring anything complicated.
(3) Having an "update my data" feature that basically looks at the list of projects that EKG cares about, and can say "well, my data on Spacewalk hasn't been updated since January 12th, so let me go get all the new data" or "oVirt has just been added to the list, so let me go and get everything that exists".
I'm not 100% sure about the best way to go about this, but it doesn't seem difficult.
(4) Being able to analyze the data in time slices. Monthly, quarterly, and yearly are what interests me most. I'd like to be able to get that either from a calendar point of view (year starts on January 1) or a Red Hat fiscal year point of view (year starts on March 1).
We talked about this. This is going to be the #2 feature, after the #1 bit.
(5) Some use cases:
- There are 100 projects in the database. Show me the report of the
last quarter for all of them.
- Here is a list of 10 projects that are most important to Brian
Stevens. Show me the breakdown for the last two years, with quarterly deltas.
- Show me all projects where the external mailing list percentage is
greater than 50%.
- Show me all the projects where @redhat.com is committing more than 75%
of the code.
- Show me the projects where the total volume of mailing list traffic
has increased/decreased by more than 25% since the last quarter.
This sounds like business reporting. It's that wishlist bit that goes beyond a simple prototype and simple working solution. Some of these use cases should be pretty simple to do. Some of them may require more work.
-Yaakov