Let's get away from the language debate, and discuss the design and architecture of the code. Maybe the language falls out of that discussion.
=======
Ignoring what currently is implemented in v1 and v2, here is what I care about:
(1) Pulling data down from wherever *exactly once* and storing it in a database forever after that.
(2) The primary feature of the config file is a "list of projects", which represents the projects that EKG will analyze. For each project in the list, the pieces of data that are collected will probably expand over time, so it's important to take this into account when we architect. Maybe we're only starting with mailing lists, but we'll eventually expand to commits, maybe to bugs, etc. Basically I want the program to think of something like Spacewalk not just as the spacewalk mailing list(s), but the various other sources of data that exist.
(3) Having an "update my data" feature that basically looks at the list of projects that EKG cares about, and can say "well, my data on Spacewalk hasn't been updated since January 12th, so let me go get all the new data" or "oVirt has just been added to the list, so let me go and get everything that exists".
(4) Being able to analyze the data in time slices. Monthly, quarterly, and yearly are what interests me most. I'd like to be able to get that either from a calendar point of view (year starts on January 1) or a Red Hat fiscal year point of view (year starts on March 1).
(5) Some use cases:
* There are 100 projects in the database. Show me the report of the last quarter for all of them.
* Here is a list of 10 projects that are most important to Brian Stevens. Show me the breakdown for the last two years, with quarterly deltas.
* Show me all projects where the external mailing list percentage is greater than 50%.
* Show me all the projects where @redhat.com is committing more than 75% of the code.
* Show me the projects where the total volume of mailing list traffic has increased/decreased by more than 25% since the last quarter.
This is a bit further reaching than what we discussed at FOSDEM. My comments as follows.
2009/2/11 Max Spevack mspevack@redhat.com:
Let's get away from the language debate, and discuss the design and architecture of the code. Maybe the language falls out of that discussion.
=======
Ignoring what currently is implemented in v1 and v2, here is what I care about:
I can airlift the design from v2 into this, without any difficulty.
(1) Pulling data down from wherever *exactly once* and storing it in a database forever after that.
That's goal #1 for today.
(2) The primary feature of the config file is a "list of projects", which represents the projects that EKG will analyze. For each project in the list, the pieces of data that are collected will probably expand over time, so it's important to take this into account when we architect. Maybe we're only starting with mailing lists, but we'll eventually expand to commits, maybe to bugs, etc. Basically I want the program to think of something like Spacewalk not just as the spacewalk mailing list(s), but the various other sources of data that exist.
I have a few ideas on how to do this. I'm gonna use a few ideas i figured out in Devshell. (When all you have is a hammer, everything looks like a nail.)
I can create classes that handle certain kinds of data streams, namely, mailing lists, bugzilla, trac, wikis, etc... Then each project is a composite object of these classes. Finally, we can find a way to serialize an object to a config file like file, that can be used to combine pieces like this, without requiring anything complicated.
(3) Having an "update my data" feature that basically looks at the list of projects that EKG cares about, and can say "well, my data on Spacewalk hasn't been updated since January 12th, so let me go get all the new data" or "oVirt has just been added to the list, so let me go and get everything that exists".
I'm not 100% sure about the best way to go about this, but it doesn't seem difficult.
(4) Being able to analyze the data in time slices. Monthly, quarterly, and yearly are what interests me most. I'd like to be able to get that either from a calendar point of view (year starts on January 1) or a Red Hat fiscal year point of view (year starts on March 1).
We talked about this. This is going to be the #2 feature, after the #1 bit.
(5) Some use cases:
- There are 100 projects in the database. Show me the report of the
last quarter for all of them.
- Here is a list of 10 projects that are most important to Brian
Stevens. Show me the breakdown for the last two years, with quarterly deltas.
- Show me all projects where the external mailing list percentage is
greater than 50%.
- Show me all the projects where @redhat.com is committing more than 75%
of the code.
- Show me the projects where the total volume of mailing list traffic
has increased/decreased by more than 25% since the last quarter.
This sounds like business reporting. It's that wishlist bit that goes beyond a simple prototype and simple working solution. Some of these use cases should be pretty simple to do. Some of them may require more work.
-Yaakov
On Wed, Feb 11, 2009 at 02:26:08PM +0100, Yaakov Nemoy wrote:
This sounds like business reporting. It's that wishlist bit that goes beyond a simple prototype and simple working solution. Some of these use cases should be pretty simple to do. Some of them may require more work.
I wonder if we should be looking at existing FLOSS business analysis tools? E.g. http://www.jaspersoft.com/JasperSoft_BI_Suite.html
- Karsten, who knows there are Jasper bits coming for F11
Karsten Wade wrote:
On Wed, Feb 11, 2009 at 02:26:08PM +0100, Yaakov Nemoy wrote:
This sounds like business reporting. It's that wishlist bit that goes beyond a simple prototype and simple working solution. Some of these use cases should be pretty simple to do. Some of them may require more work.
I wonder if we should be looking at existing FLOSS business analysis tools? E.g. http://www.jaspersoft.com/JasperSoft_BI_Suite.html
- Karsten, who knows there are Jasper bits coming for F11
ekg mailing list ekg@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/ekg
I haven't ever used the tool to say if it's good or not, but if we're going to take this to the logical end, and we are eventually moving off Google Charts because we can't host an instance of EKG in Fedora because it's not in Fedora itself, we would also need Jaspersoft packaged in Fedora.
I think EKG should be treated as a Fedora project.
--Michael
On Fri, 13 Feb 2009, Michael DeHaan wrote:
I haven't ever used the tool to say if it's good or not, but if we're going to take this to the logical end, and we are eventually moving off Google Charts because we can't host an instance of EKG in Fedora because it's not in Fedora itself, we would also need Jaspersoft packaged in Fedora.
I think EKG should be treated as a Fedora project.
I agree. If Jasper is going to land in F11 though, that is a possibility for the relatively near future.
--Max
Max Spevack wrote:
On Fri, 13 Feb 2009, Michael DeHaan wrote:
I haven't ever used the tool to say if it's good or not, but if we're going to take this to the logical end, and we are eventually moving off Google Charts because we can't host an instance of EKG in Fedora because it's not in Fedora itself, we would also need Jaspersoft packaged in Fedora.
I think EKG should be treated as a Fedora project.
I agree. If Jasper is going to land in F11 though, that is a possibility for the relatively near future.
--Max _______________________________________________ ekg mailing list ekg@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/ekg
Is that something Jaspersoft was pushing for? If so, goodness. I hadn't heard.
--Michael
On Fri, Feb 13, 2009 at 04:32:37PM -0500, Michael DeHaan wrote:
Is that something Jaspersoft was pushing for? If so, goodness. I hadn't heard.
I've been gently pushing on all the ISVs to use Fedora 11 as a good target, particularly as code may feed in to RHEL 6. I haven't seen anything official from Jasper in a while, but I believe they are on track to get some reporting packages in F11. I'm hoping it's bits that are useful to us. :)
- Karsten
This sounds like business reporting. It's that wishlist bit that goes beyond a simple prototype and simple working solution. Some of these use cases should be pretty simple to do. Some of them may require more work.
-Yaakov
FWIW, I don't see our goal as being to make a prototype. We have a prototype, what we want now /is/ a usable and expandable solution.
On Wed, Feb 11, 2009 at 01:22:24PM +0100, Max Spevack wrote:
(4) Being able to analyze the data in time slices. Monthly, quarterly, and yearly are what interests me most.
Although it won't normally be as relevant, seeing down to the day or even hourly level might be interesting.
For example, we usually have an uptick of interest in joining Docs following a release. I'd like to be able to look for that trend in other sub-projects and show that trend to other team leaders so they can be prepared for such an uptick. (This one is perhaps tracked by emails where the subject contains .*[I,i]ntro.*
So, there are a few points in time each year where this matters. As another example, I'd be interested to see what happened to mailing list traffic following last Fall's security incident. Maybe nothing, maybe something that lends weight to another argument.
- Karsten
Karsten Wade wrote:
On Wed, Feb 11, 2009 at 01:22:24PM +0100, Max Spevack wrote:
(4) Being able to analyze the data in time slices. Monthly, quarterly, and yearly are what interests me most.
Although it won't normally be as relevant, seeing down to the day or even hourly level might be interesting.
For example, we usually have an uptick of interest in joining Docs following a release. I'd like to be able to look for that trend in other sub-projects and show that trend to other team leaders so they can be prepared for such an uptick. (This one is perhaps tracked by emails where the subject contains .*[I,i]ntro.*
So, there are a few points in time each year where this matters. As another example, I'd be interested to see what happened to mailing list traffic following last Fall's security incident. Maybe nothing, maybe something that lends weight to another argument.
- Karsten
I had a conversation recently where I experessed, about EKG, the desire to choke out impulses to collect data that we can't use to some actionable end.
That being said, I don't think trying to learn about things that we can't /use/ is a good use of time.
I'd rather find data we can use that will help us make decisions and influence projects.
Another problem is that correlation != causation, so it's impossible to draw a line between list traffic and any one particular event.
Perhaps people wash more clothes on Tuesday in February than Tuesdays in January. Is that because of the stock market or the super bowl? Or just because it's cold outside.
EKG's mission, I think, should be two things: -- to learn what projects people are interested in and have strong communities around them -- to learn which projects could use more help on the community side, and to point out other projects they could look to as examples
--Michael
ekg mailing list ekg@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/ekg
Max Spevack wrote:
Let's get away from the language debate, and discuss the design and architecture of the code. Maybe the language falls out of that discussion.
=======
Ignoring what currently is implemented in v1 and v2, here is what I care about:
(1) Pulling data down from wherever *exactly once* and storing it in a database forever after that.
Mostly it does, it just fetches the indexes more than once.
However, it needs to work on tarballs, not web content.
(2) The primary feature of the config file is a "list of projects", which represents the projects that EKG will analyze. For each project in the list, the pieces of data that are collected will probably expand over time, so it's important to take this into account when we architect. Maybe we're only starting with mailing lists, but we'll eventually expand to commits, maybe to bugs, etc. Basically I want the program to think of something like Spacewalk not just as the spacewalk mailing list(s), but the various other sources of data that exist.
Agreed, a project definition may contain more than just a list URL.
(3) Having an "update my data" feature that basically looks at the list of projects that EKG cares about, and can say "well, my data on Spacewalk hasn't been updated since January 12th, so let me go get all the new data" or "oVirt has just been added to the list, so let me go and get everything that exists".
Agreed, particularly if you only want to graph two projects you might not care about the others.
(4) Being able to analyze the data in time slices. Monthly, quarterly, and yearly are what interests me most. I'd like to be able to get that either from a calendar point of view (year starts on January 1) or a Red Hat fiscal year point of view (year starts on March 1).
I'd ideally like to see multiple pie graphs, each representing different internals, so see how that pie changes over time.
(5) Some use cases:
- There are 100 projects in the database. Show me the report of the
last quarter for all of them.
- Here is a list of 10 projects that are most important to Brian
Stevens. Show me the breakdown for the last two years, with quarterly deltas.
- Show me all projects where the external mailing list percentage is
greater than 50%.
- Show me all the projects where @redhat.com is committing more than 75%
of the code.
- Show me the projects where the total volume of mailing list traffic
has increased/decreased by more than 25% since the last quarter.
Yes...
ekg mailing list ekg@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/ekg
On Wed, 11 Feb 2009, Max Spevack wrote:
Ignoring what currently is implemented in v1 and v2, here is what I care about:
(1) Pulling data down from wherever *exactly once* and storing it in a database forever after that.
(2) The primary feature of the config file is a "list of projects", which represents the projects that EKG will analyze. For each project in the list, the pieces of data that are collected will probably expand over time, so it's important to take this into account when we architect. Maybe we're only starting with mailing lists, but we'll eventually expand to commits, maybe to bugs, etc. Basically I want the program to think of something like Spacewalk not just as the spacewalk mailing list(s), but the various other sources of data that exist.
(3) Having an "update my data" feature that basically looks at the list of projects that EKG cares about, and can say "well, my data on Spacewalk hasn't been updated since January 12th, so let me go get all the new data" or "oVirt has just been added to the list, so let me go and get everything that exists".
(4) Being able to analyze the data in time slices. Monthly, quarterly, and yearly are what interests me most. I'd like to be able to get that either from a calendar point of view (year starts on January 1) or a Red Hat fiscal year point of view (year starts on March 1).
Well, the goals and deadlines that the CommArch team is operating under demand that progress be made, in particular on the enhancements and new features around the time slicing, and it is those features that have triggered the larger architectural discussion.
Yaakov, I'd like to see from you, preferably before Wednesday night EST (because I need it for a meeting that I have on Thursday), your development roadmap for the rest of February and as far into March as you can give.
Be relatively detailed about the technical stuff -- even if some of it goes over my head, some of the real programmers on the list will understand it, and I know more than I let on. :)
Thanks, Max
2009/2/17 Max Spevack mspevack@redhat.com:
Yaakov, I'd like to see from you, preferably before Wednesday night EST (because I need it for a meeting that I have on Thursday), your development roadmap for the rest of February and as far into March as you can give.
Be relatively detailed about the technical stuff -- even if some of it goes over my head, some of the real programmers on the list will understand it, and I know more than I let on. :)
This isn't much of a roadmap, but it will suffice:
So far, i've spent a total of 4 hours on recreating EKG in Python. At this point, i have it downloading monthly mailboxes, skipping over ones that haven't changed, and tracking this in the DB. I also have some very barebones code for importing the actual mbox into the DB.
This weekend, i plan on spending 8-12 hours finishing the scanner in Python. I can't imagine using that time up fully.
After that, i'm looking at 12-16 hours to recreate the reporting feature. Also, there is plenty of good graphing software for python and/or javascript in the Fedora repository that we can use, so we will move away from the Google Charts
After that, i really don't have a roadmap, because i'm not sure how long it's going to take. I'm estimating that we'll have the monthly deltas feature coded in two hours though.
I'm also planning to organize things similar to what i have with Fedora Devshell. Each scanner and reporter is a simple python class that handles a particular type of source or report (mailing list, bug tracker, chart or graph) and a specific implementation for particular sources, (Fedora Hosted, trac, red hat bugzilla, red hat's mailman), and finally a project class which aggregates these classes together. EKG will use information in the configuration file to autoload a class with particular worker classes, per project, so you can get a good per project overview.
Let me know if you need more information.
Also, i'll be busy with schoolwork till at least friday, so there won't be much coming from me in the next couple of days.
-Yaakov