2009/2/25 Max Spevack <mspevack(a)redhat.com>:
On Wed, 25 Feb 2009, loupgaroublond(a)gmail.com wrote:
> I've broken down what we're looking at into 'sources' which are
places
> where things happen, and 'facts' which are actions someone undertook.
> Each fact has one source.
This sounds sane, based on your description and our discussion at FOSDEM
and on-list. Can you give us a few examples just for further clarity?
Wikipedia is a wiki source of name wikipedia. Fedora wiki is also a
wiki source of name fedora wiki. Linus's git tree is a source of git
with the name linux.
Each wiki change is a fact. Each git patch is a fact.
(Technically, my 'source' is actually a month on each mailing list. It
has to do with that i'm pulling monthly mbox archives. I may
denormalize the data slightly to handle a source as a single entity
rather than broken apart by month.)
> So far there's an abstract base class called mailman, that handles
> getting gzipped mbox files of logs from various sources, and then
> there are some simple functions that break it down into facts in the
> database. The reason for this is that in order to write a new mailman
> for a new source, you simply need to feed it some URLs, and a regex or
> two it can use to parse the web pages that link to the mbox files.
> This idea can be extended to other ways of getting facts from other
> sources. Right now, since we're focused on mailing lists, it uses a
> few idioms related to that, which will have to be factored out at some
> point.
Also sounds sane. I'd recommend that you test the flexibility of your
class by figuring out how to slurp down the KVM mailing lists, and
adding those as sources and facts.
http://kvm.qumranet.com/kvmwiki/Lists%2C_IRC
This looks like it could be a tad more difficult. There aren't any
compressed archives, which runs into the same problem we had with v2
downloading emails directly from the web sites. We may want to ask the
maintainer of those mailing lists for some special dispensation, but
barring that, i'm sure i could figure out a way to parse web pages. It
just won't be as nice. I'm also not sure what the limits or gmane are
for that sort of thing.
-Yaakov