Hi all,
I have prepared a description of my possible ekg-GSoC project.
Here it goes:
I. Network of connections between members based on reply pairs.
AIM: Create a network of connections based on the structure of replies
and number of mails sent by every user in each reply-pair and answer
such questions as:
- who is the core member of the community (by e-mail domain)?
Add pie chart.
- who is a regular member of the community (by e-mail domain)?
Add pie chart.
- who is a temporary (passive) member of the community
(by e-mail domain)? Add pie chart.
- how many members are in each group? Add pie chart.
- what is the level of exclusiveness and responsiveness for the mailing
list?
- are the temporary (passive) members active, i.e. do they speak with
each other? (the assumption is: temporary members are there to ask
some specific questions, receive the answer and go to their work;
they are not participating in discussions other than the discussions
they started; and if they do - if there are pairs of replies:
temp_member <-> temp_member - then it means that they are also
participating in general discussion, which should be a good prognosis
for particular community)
In case of weekly/monthly/quarterly slices show the dynamics:
- are the core/regular/temporary groups changing?
Are the numbers changing? Are the e-mail domains changing?
Is the density in groups changing? Present these results
on line charts.
II. Basic data structure
Connecting users on the basis of the structure of replies has two
dimensions:
- number of connections to other users
- number of mails sent by each user to any other user
In terms of data structures it could look like this:
X = {
emailX: {
'emailY': number_of_replies,
'emailZ': number_of_replies,
...
},
emailY: {
'emailX': number_of_replies,
'emailZ': number_of_replies,
...
},
...
}
A number of connections for user 'emailY' is:
number_of_connections = len(X['emailY'])
A number of all the messages sent by user 'emailY' is:
number_of_all_emails_sent = sum(d['emailY'].values())
A number of e-mails sent by user 'emailY' to user 'emailX' is:
number_of_emails_sent_to_emailX = X['emailY']['emailX']
I consider filling such a data structure with actual data as one
of the mile stones of my project. But, let's assume that it is filled,
so we can continue our thought experiment.
III. Core members, regular members and temporary members
Beased on these two indicators - number of connections and number
of mails sent by each user - we are able to create a network of
connections between members and thus distinguish core
members group, which should be defined as a group of people for
which the numbers of connections and mails sent are highest.
Moreover, we are able to distinguish regular users and temporary
users. Temporary users come and go, their networks of connections
are weak and they are not exchanging too many mails. Regular
users are somewhere between.
When it comes to deciding who is in the core group and who is not,
a number of algorithms may be used.
A possible algorithm could look like this:
1. Count the average number of connections for all users.
New variable: avg_number_of_connections.
In terms of proposed data structure:
connections_sum = 0
for user in X:
connections += len(user[X])
avg_number_of_connections = connections / float(len(X))
2. Count the average number of mails sent for all users:
New variable: avg_number_of_mails_sent.
In terms of proposed data structure:
mails_sent = 0
for user in X:
mails_sent += sum(X[user].values())
avg_number_of_mails_sent = mails_sent / float(len(X))
3. Create three levels of participation for both parameters
(number of connections and number of mails sent):
a) high (core members):
- 'number_of_connections' is rated as high if it is
75% or more of the average;
New variable: num_of_connections_high:
num_of_connections_high = avg_number_of_connections * 0.75
- 'number_of_mails_sent' is rated as high if it is
75% or more of the average;
New variable: num_of_mails_sent_high:
num_of_mails_sent_high = avg_numer_of_mails_sent * 0.75
b) low (temporary members):
- 'number_of_connections' is rated as low if it is
25% or less than the average;
New variable: num_of_connections_low:
num_of_connections_low = avg_number_of_connections * 0.25
- 'number_of_mails_sent' is rated as low if it is
25% or less than the average;
New variable: num_of_mails_sent_low:
num_of_mails_sent_low = avg_numer_of_mails_sent * 0.25
c) medium (regular members):
- 'number_of_connections' is rated as medium if it is
found in range between 25% and 75% of the average;
- 'number_of_mails_sent' is rated as medium if it is
found in range between 25% and 75% of the average;
4. Go through the dictionary of users and assign them to one of three
groups: core_members, regular_members, temp_members.
If both variables - number_of_connections and number_of_mails_sent -
for particular user are higher than 75% of the average (vide:
num_of_connections_high and num_of_mails_sent_high), then the user
is a member of the core group.
If both variables - number_of_connections and number_of_mails_sent -
for particular user are lower than 25% of the average (vide:
num_of_connections_low and num_of_mails_sent_low), then the user
is a member of the temporary members group.
Else: assign the user to regular members group.
Now that we have our data structure filled and we have identified
the groups of members, we are able to answer a few of the questions
stated at the beginning, but some of them still remain unanswered.
It is because the idea of counting the level of exclusiveness,
responsiveness and temporary users participation requires better
explanation (and more code).
IV. Exclusiveness, responsiveness, and temporary users participation levels
Possible reply pairs (reverses included):
a) core member <-> core member (CC)
b) regular member <-> regular member (RR)
c) temporary member <-> temporary member (TT)
d) code member <-> temporary member (CT)
e) code member <-> regular member (CR)
f) regular member <-> temporary member (RT)
Given our lists of core/regular/temporary members and our basic
data structure we are able to count what follows (reverses included):
CCS = sum of all CC replies
TTS = sum of all TT replies
RRS = sum of all RR replies
CTS = sum of all CT replies
CRS = sum of all CR replies
RTS = sum of all RT replies
* Level of exclusiveness: CCS / (CCS+TTS+RRS+CTS+CRS+RTS)
Level of exclusiveness indicates how often core developers speak
only with each other. Possible values: min. - 0.0 and max - 1.0.
Minimum value means that core developers don't speak with each other
at all, and maximum value means that they speak only with each other.
* Level of responsiveness: CTS / (CCS+TTS+RRS+CTS+CRS+RTS)
Level of responsiveness indicates how often core developers speak
with temporary members. Possible values: min. - 0.0 and max - 1.0.
Minimum value means that core developers don't speak with temporary
members at all, and maximum value means that core developers don't
loose any chance to speak with temporary members.
* Level of temporary users participation: TTS /
(CCS+TTS+RRS+CTS+CRS+RTS)
Level of temporary users participation indicates how often temporary
members speak with each other. Possible values: min. - 0.0 and max
- 1.0.
Minimum value means that temporary users don't speak with other
temporary
users which might mean that they came only to get the answer they need
and then go. Maximum value means that we probably have some new strong
contributors.
V. Network of connections: a graph
A bit of social networks analysis. Show a map of connections between
users including the information on how strong the ties between them
are (based on the number of mails exchanged).
OK, that's all for now, thanks for reading :-)
All your comments are greately appreciated!
--
Regards,
Wojtek Walczak,
http://tosh.pl/gminick/