Concrete idea for my GSoC project
by Wojtek Walczak
Hi all,
I have prepared a description of my possible ekg-GSoC project.
Here it goes:
I. Network of connections between members based on reply pairs.
AIM: Create a network of connections based on the structure of replies
and number of mails sent by every user in each reply-pair and answer
such questions as:
- who is the core member of the community (by e-mail domain)?
Add pie chart.
- who is a regular member of the community (by e-mail domain)?
Add pie chart.
- who is a temporary (passive) member of the community
(by e-mail domain)? Add pie chart.
- how many members are in each group? Add pie chart.
- what is the level of exclusiveness and responsiveness for the mailing
list?
- are the temporary (passive) members active, i.e. do they speak with
each other? (the assumption is: temporary members are there to ask
some specific questions, receive the answer and go to their work;
they are not participating in discussions other than the discussions
they started; and if they do - if there are pairs of replies:
temp_member <-> temp_member - then it means that they are also
participating in general discussion, which should be a good prognosis
for particular community)
In case of weekly/monthly/quarterly slices show the dynamics:
- are the core/regular/temporary groups changing?
Are the numbers changing? Are the e-mail domains changing?
Is the density in groups changing? Present these results
on line charts.
II. Basic data structure
Connecting users on the basis of the structure of replies has two
dimensions:
- number of connections to other users
- number of mails sent by each user to any other user
In terms of data structures it could look like this:
X = {
emailX: {
'emailY': number_of_replies,
'emailZ': number_of_replies,
...
},
emailY: {
'emailX': number_of_replies,
'emailZ': number_of_replies,
...
},
...
}
A number of connections for user 'emailY' is:
number_of_connections = len(X['emailY'])
A number of all the messages sent by user 'emailY' is:
number_of_all_emails_sent = sum(d['emailY'].values())
A number of e-mails sent by user 'emailY' to user 'emailX' is:
number_of_emails_sent_to_emailX = X['emailY']['emailX']
I consider filling such a data structure with actual data as one
of the mile stones of my project. But, let's assume that it is filled,
so we can continue our thought experiment.
III. Core members, regular members and temporary members
Beased on these two indicators - number of connections and number
of mails sent by each user - we are able to create a network of
connections between members and thus distinguish core
members group, which should be defined as a group of people for
which the numbers of connections and mails sent are highest.
Moreover, we are able to distinguish regular users and temporary
users. Temporary users come and go, their networks of connections
are weak and they are not exchanging too many mails. Regular
users are somewhere between.
When it comes to deciding who is in the core group and who is not,
a number of algorithms may be used.
A possible algorithm could look like this:
1. Count the average number of connections for all users.
New variable: avg_number_of_connections.
In terms of proposed data structure:
connections_sum = 0
for user in X:
connections += len(user[X])
avg_number_of_connections = connections / float(len(X))
2. Count the average number of mails sent for all users:
New variable: avg_number_of_mails_sent.
In terms of proposed data structure:
mails_sent = 0
for user in X:
mails_sent += sum(X[user].values())
avg_number_of_mails_sent = mails_sent / float(len(X))
3. Create three levels of participation for both parameters
(number of connections and number of mails sent):
a) high (core members):
- 'number_of_connections' is rated as high if it is
75% or more of the average;
New variable: num_of_connections_high:
num_of_connections_high = avg_number_of_connections * 0.75
- 'number_of_mails_sent' is rated as high if it is
75% or more of the average;
New variable: num_of_mails_sent_high:
num_of_mails_sent_high = avg_numer_of_mails_sent * 0.75
b) low (temporary members):
- 'number_of_connections' is rated as low if it is
25% or less than the average;
New variable: num_of_connections_low:
num_of_connections_low = avg_number_of_connections * 0.25
- 'number_of_mails_sent' is rated as low if it is
25% or less than the average;
New variable: num_of_mails_sent_low:
num_of_mails_sent_low = avg_numer_of_mails_sent * 0.25
c) medium (regular members):
- 'number_of_connections' is rated as medium if it is
found in range between 25% and 75% of the average;
- 'number_of_mails_sent' is rated as medium if it is
found in range between 25% and 75% of the average;
4. Go through the dictionary of users and assign them to one of three
groups: core_members, regular_members, temp_members.
If both variables - number_of_connections and number_of_mails_sent -
for particular user are higher than 75% of the average (vide:
num_of_connections_high and num_of_mails_sent_high), then the user
is a member of the core group.
If both variables - number_of_connections and number_of_mails_sent -
for particular user are lower than 25% of the average (vide:
num_of_connections_low and num_of_mails_sent_low), then the user
is a member of the temporary members group.
Else: assign the user to regular members group.
Now that we have our data structure filled and we have identified
the groups of members, we are able to answer a few of the questions
stated at the beginning, but some of them still remain unanswered.
It is because the idea of counting the level of exclusiveness,
responsiveness and temporary users participation requires better
explanation (and more code).
IV. Exclusiveness, responsiveness, and temporary users participation levels
Possible reply pairs (reverses included):
a) core member <-> core member (CC)
b) regular member <-> regular member (RR)
c) temporary member <-> temporary member (TT)
d) code member <-> temporary member (CT)
e) code member <-> regular member (CR)
f) regular member <-> temporary member (RT)
Given our lists of core/regular/temporary members and our basic
data structure we are able to count what follows (reverses included):
CCS = sum of all CC replies
TTS = sum of all TT replies
RRS = sum of all RR replies
CTS = sum of all CT replies
CRS = sum of all CR replies
RTS = sum of all RT replies
* Level of exclusiveness: CCS / (CCS+TTS+RRS+CTS+CRS+RTS)
Level of exclusiveness indicates how often core developers speak
only with each other. Possible values: min. - 0.0 and max - 1.0.
Minimum value means that core developers don't speak with each other
at all, and maximum value means that they speak only with each other.
* Level of responsiveness: CTS / (CCS+TTS+RRS+CTS+CRS+RTS)
Level of responsiveness indicates how often core developers speak
with temporary members. Possible values: min. - 0.0 and max - 1.0.
Minimum value means that core developers don't speak with temporary
members at all, and maximum value means that core developers don't
loose any chance to speak with temporary members.
* Level of temporary users participation: TTS /
(CCS+TTS+RRS+CTS+CRS+RTS)
Level of temporary users participation indicates how often temporary
members speak with each other. Possible values: min. - 0.0 and max
- 1.0.
Minimum value means that temporary users don't speak with other
temporary
users which might mean that they came only to get the answer they need
and then go. Maximum value means that we probably have some new strong
contributors.
V. Network of connections: a graph
A bit of social networks analysis. Show a map of connections between
users including the information on how strong the ties between them
are (based on the number of mails exchanged).
OK, that's all for now, thanks for reading :-)
All your comments are greately appreciated!
--
Regards,
Wojtek Walczak,
http://tosh.pl/gminick/
15 years, 2 months
dateutil.parser.parse() also known as dup.parse() problem
by Wojtek Walczak
Folks,
when I try to run the scanner it breaks after a few seconds.
Python complains about this line (175) from model.py:
match_dict['month'] = dup.parse('%s %s' % (match_dict['month'],
match_dict['year']))
I am able to repeat this problem in python's interactive interpreter:
>>> import dateutil.parser as dup
>>> dup.parse('January 2009')
datetime.datetime(2009, 1, 30, 0, 0)
>>> dup.parse('February 2009')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/usr/local/lib/python2.5/site-packages/python_dateutil-1.4.1-py2.5.egg/dateutil/parser.py",
line 697, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File
"/usr/local/lib/python2.5/site-packages/python_dateutil-1.4.1-py2.5.egg/dateutil/parser.py",
line 310, in parse
ret = default.replace(**repl)
ValueError: day is out of range for month
>>> dup.parse('March 2009')
datetime.datetime(2009, 3, 30, 0, 0)
It's not that hard to guess that dateutil.parser
breaks down when it's being asked for a date that
doesn't exist in the calendar.
This bug is not related to ekg's code itself, but
I am curious if you see it too.
Cheers,
--
Wojtek Walczak,
http://tosh.pl/gminick/
15 years, 2 months
[GSoC] EKG question
by Wojtek Walczak
Hi all,
I am a prospective GSoC participant interested in
contributing to EKG.
I've browsed the archives and it seems that Kasia
Hayden is also interested in this project.
Basically I would like to ask if she (or: you, Kasia,
if you are reading it :-)) is applying,
and if she is, what exactly does she plan to propose.
I mean, I wouldn't like to repeat her ideas
in my application, and thus I am asking.
To introduce myself shortly: my name is Wojtek Walczak,
I am 24 and I study sociology. I am a hobbyst Python
programmer for about 8 years now. My main sociological
concern is to apply my programming knowledge to it :-)
I've created a social networks analyzing tool (available
on my homepage), two Django-based web apps (with Google
Charts API used for visualizations) used to conduct
media research on my University, and now I am working
on my masters thesis tool which does Internet news portals
analysis.
What you guys are doing is exactly what I am interested
in, and it's a pity that EKG is hidden so deep on the
mentoring orgs list.
Best regards,
--
Wojtek Walczak,
http://tosh.pl/gminick/
15 years, 2 months
current roadmap
by Max Spevack
So Yaakov,
With interest coming from various other contributors, and your own time
soon to be in very limited quantity, I'm curious to know if we have a
solid feature roadmap. There's still a few things left from "my list"
that need to be finished, and then I think it's time to have another
discussion of where EKG needs to go.
I am glad for the progress that we've made, but I also feel like there
are some bugs to still work out before we can consider that we have a
successful Alpha "release" rather than a snapshot of code that sort of
just works.
Do we have more specific descriptions of the rhmailman bugs, so that I
can help to poke RH IT for solutions, if they are needed?
--Max
15 years, 2 months
RE: ekg Digest, Vol 4, Issue 17
by Hayden, Katherine M.
Michael DeHaan <mdehaan(a)redhat.com> wrote:
I think data is obviously one great way to expand -- another way to
expand is in terms of new visualizations that can be produced. Namely,
making it possible to compare projects on the same graphs (or, well,
actually having graphs again now that we've moved away from Google
Chart), and providing new ways to access all of the data at a glance. If
you're interested in doing this, I don't see a problem with you either
focusing on data collection, visualizations ("the grapher"), or a little
of both.
So, if you want to think about visualization ideas, those too, would be
useful.
I think you'll find there are a lot of different places we can take this.
--Michael
I've been thinking about visualization ideas as well, and thinking that it might be most valuable to concentrate on getting some solid visualization tools for the mailing list data we already have.
Besides the original pie charts, it could be useful to plot the projects based on their communities (which has the largest population, which has the most/least percentage of Fedora people). Being able to see these visualizations change over a specified time would be very valuable, I think, especially if you could see them change fluidly. I was inspired by this TED talk on stats and data (http://www.youtube.com/watch?v=hVimVzgtD6w), starting from 4:00 and going to 5:15, or starting from 5:00 and going to 5:15 if you just want the Instant Replay version.
So I propose that I'll make my focus data visualization so that you get some payoff for the work you've already done on collecting the data and show off what's possible with ekg. I agree with what Yaakov said, that "What is most relevant at the moment is the kinds of analysis we can do."
Kasia
Katherine "Kasia" Hayden
Bennington College, Class of 2011
312.576.8175
khayden(a)bennington.edu
15 years, 2 months
GSoC EKG project
by Hayden, Katherine M.
Hi Michael, Max and Karsten,
I've read through this mailing list and spoken with Yaakov on IRC already about contributing to this project over the summer.
We went through some basic interview questions and left off with this request from Yaakov:
"The application itself, as you've read on the wiki, is going to look
for original analysis ideas. Last summer, i tried to get a student for
another data analysis project to contribute, but that failed. I had
asked him to put together a few mockups, and the only code i received
in the end was a CSS stylesheet that matched the mockups. Instead, i
would like you to focus on the the following.
What kind of data is important for us to be looking at. You're going
to collect data from mailing lists, wikis, version control systems,
message boards, irc channels, and other possible sources of input.
What kind of data is important and what is chaff?"
These are my thoughts on the above.
>From reading through the EKG mailing list archives, it seems that everyone agrees that EKG's focus is, as Michael put it:
1) To learn about projects people are interested in and have strong communities around
2) To learn which projects could use more help on the community side and to point to other projects they could look to as examples
To look for projects that have strong communities we have to decide how we evaluate a community and decide whether it's strong or not. Is that defined by having a lot of members on the mailing list? What if 75% of the mailing list members never contribute? Or maybe it's the frequency of a project's discussion on IRC. Or the amount of quality documentation?
Here are some data I think are work collecting:
Breakdown of members
Mailing lists:
Number of posts (on it's own and/or discounted by number of members)
Average time between an original post and a follow-up post
Average number of times a user posts. Also the ranges: most and least posts by a user. Could do box plots
Number of joins/leaves in a month (or whatever greater time slice you select)
And what's already incorporated:
Total number of members
Members by email
IRC:
A graph of the number of users on over a 24 hour period
Average number of users on at any given time
Length of time a user spends on channel for a given session
If we didn't mind keeping track of users:
Length of time a user spends on channel in a month (or whatever greater time slice you select)
The frequency of new users
Number of posts (posts might be the wrong terminology here, would ims be better?)
Average time between an original post and a follow-up post
Average number of times a user posts. Also the ranges: most and least posts by a user. Could do box plots
Version control systems:
Number of commits
Time since last commit
Wikis:
Number of page visits
Length of average visit
Number of wiki edits
Message boards:
Number of threads, replies, views
What are your critiques/feedback? Is that what you were looking for Yaakov?
Kasia
Katherine "Kasia" Hayden
Bennington College, Class of 2011
312.576.8175
khayden(a)bennington.edu
15 years, 2 months
IRC tracking
by Karsten Wade
Found out about this recently, the IRC Support SIG has been using it
to track statistics on #fedora:
PISG, the Perl IRC Stats Generator:
http://pisg.sourceforge.net/
Here is an example of it's use from #fedora; Scott has been tracking
stats for a long time
http://theglaserfamily.org/ircstats/fedora-weekly.html
We could use some hostmasks to tell, e.g., @redhat.com NAT pools,
etc. When people have a Fedora hostmask, we could know more of who
they are.
Rather than rewrite this, could we look at ways to incorporate other
projects in as part of an 'EKG toolki'? Or should we be looking at a
larger toolkit that EKG is just a part of?
- Karsten
--
Karsten 'quaid' Wade, Community Gardener
http://quaid.fedorapeople.org
AD0E0C41
15 years, 2 months
rhmailman failure
by Yaakov Nemoy
Hey List,
for some odd reason, when i went to run scanner.py last night on max's
settings file, i got about 8000 facts for a time period spanning
several years. This seemed odd, but since i started it at around bed
time expecting it to take all night, i didn't get a chance to debug
it. My apologies max.
I'm gonna have a look at it in a bit.
-Yaakov
15 years, 2 months
Success with RHMailman
by Yaakov Nemoy
Hey all,
I've put in a temporary fix for RHMailman so we can extract *most* of the data. Some parts still aren't coming through, like when an email has a bad date attached to it. This is something that is not a serious problem, however.
Max, feel free to run your settings.ini on it. It will take a while though.
-Yaakov
15 years, 2 months
rhmailman
by Yaakov Nemoy
Hey List,
I did a bit of tweaking to RHmailman and then had it run on
fedora-devel-list for a while. It ran for about an hour, and then
crashed because of some other trivial formatting error of the
time/date that it couldn't parse properly. I have to get started on
school work though, so i don't think i'll have much time for it more
today.
Most notably though, i mentioned that we can use multiple sources and
facts, based on other sources of information. The patch i've committed
to git demonstrates how we can use Source and Fact polymorphically as
well. (Read object oriented). This is by no means a final API to
anything, it's just to show you that it can be done concisely.
-Yaakov
15 years, 2 months