Hello,
I have two students interested in diploma thesis called Yum plugin for suggesting packages based on usage:
TL;DR - from anonymized access log, create a database of suggested packages using data mining techniques and provide a Yum plugin that would suggest "Users of vim also installed: ctags, git, ..."
I am gonna create a Fedora Feature wiki page shortly describing this in more detail. Our goal is to offer this project for integration into Fedora later on, at least provide Fedora packages for it.
To do that, we need good source of data. It would be best to collect access logs from one or two main Fedora mirrors. We would provide short script in Python that would parse access logs and anonymize the data (IP address hash-salted) and filtered only relevant data (RPM files from latest Fedora release or updates repositories). That would be phase one which should give us a sample data.
Phase two would be to integrate this script with logrotate and for one Fedora release cycle (Fedora 19) the script would collect relevant anonymized data into a file. Final suggested package database would be created from this file (or maybe files to allow us to move them on the fly out of the stat directory).
The big (legal) question is if we are able to provide this anonymized data to public, or if we want to sign NDA with all people involved. I am CCing Tom for this question.
I need your help with connecting to relevant people. Any comments are appreciated.
Many thanks and I hope this effort will lead to improving user experience with Fedora packaging.
This sounds akin to apt (debian), emerge (gentoo), 'recommends'. Added functionality but not a functional dependency. Am I on mark with intention? On May 3, 2013 8:35 AM, "Lukas Zapletal" lzap@redhat.com wrote:
Hello,
I have two students interested in diploma thesis called Yum plugin for suggesting packages based on usage:
TL;DR - from anonymized access log, create a database of suggested packages using data mining techniques and provide a Yum plugin that would suggest "Users of vim also installed: ctags, git, ..."
I am gonna create a Fedora Feature wiki page shortly describing this in more detail. Our goal is to offer this project for integration into Fedora later on, at least provide Fedora packages for it.
To do that, we need good source of data. It would be best to collect access logs from one or two main Fedora mirrors. We would provide short script in Python that would parse access logs and anonymize the data (IP address hash-salted) and filtered only relevant data (RPM files from latest Fedora release or updates repositories). That would be phase one which should give us a sample data.
Phase two would be to integrate this script with logrotate and for one Fedora release cycle (Fedora 19) the script would collect relevant anonymized data into a file. Final suggested package database would be created from this file (or maybe files to allow us to move them on the fly out of the stat directory).
The big (legal) question is if we are able to provide this anonymized data to public, or if we want to sign NDA with all people involved. I am CCing Tom for this question.
I need your help with connecting to relevant people. Any comments are appreciated.
Many thanks and I hope this effort will lead to improving user experience with Fedora packaging.
-- Later,
Lukas "lzap" Zapletal irc: lzap #theforeman _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Fri, May 03, 2013 at 08:42:02AM -0400, Steven Acres wrote:
This sounds akin to apt (debian), emerge (gentoo), 'recommends'. Added functionality but not a functional dependency. Am I on mark with intention?
Hi,
exactly. But this is more Google-like approach - we have the information in the house (logs from Apache httpd), why not to use it? No need of extra functionality on the client side or collecting any data.
Both apt and emerge has these soft dependencies in the packages themselves, this approach is different. We will try to find knowledge in the usage statistics.
Big advantage of this approach is there is no action needed to maintain those soft dependencies once the algorithm is tuned. On the other hand, it will generate some false recommendations.
That being said, it is worth trying.
LZ
Hi,
----- Mensaje original -----
De: "Lukas Zapletal" lzap@redhat.com Para: infrastructure@lists.fedoraproject.org Enviados: Viernes, 3 de Mayo 2013 14:35:45 Asunto: Anonymized access log from a fedora mirror
Hello,
I have two students interested in diploma thesis called Yum plugin for suggesting packages based on usage:
TL;DR - from anonymized access log, create a database of suggested packages using data mining techniques and provide a Yum plugin that would suggest "Users of vim also installed: ctags, git, ..."
I am gonna create a Fedora Feature wiki page shortly describing this in more detail. Our goal is to offer this project for integration into Fedora later on, at least provide Fedora packages for it.
To do that, we need good source of data. It would be best to collect access logs from one or two main Fedora mirrors. We would provide short script in Python that would parse access logs and anonymize the data (IP address hash-salted) and filtered only relevant data (RPM files from latest Fedora release or updates repositories). That would be phase one which should give us a sample data.
Phase two would be to integrate this script with logrotate and for one Fedora release cycle (Fedora 19) the script would collect relevant anonymized data into a file. Final suggested package database would be created from this file (or maybe files to allow us to move them on the fly out of the stat directory).
The big (legal) question is if we are able to provide this anonymized data to public, or if we want to sign NDA with all people involved. I am CCing Tom for this question.
I need your help with connecting to relevant people. Any comments are appreciated.
Many thanks and I hope this effort will lead to improving user experience with Fedora packaging.
Not sure from our side, but Debian has always a package "popularity-contest", which automatically submitted packages to make those list of recommended packages.
Maybe it will require some information from legal team, but from the initiative part, it sounds good :)
Regards, Pablo
On Fri, May 03, 2013 at 08:43:15AM -0400, Pablo Iranzo Gómez wrote:
Hi, Not sure from our side, but Debian has always a package "popularity-contest", which automatically submitted packages to make those list of recommended packages.
Hello, yes this is other approach to the problem. The main drawback is we would need to maintain another server-side system to gather those data.
Maybe I could extend the scope and do both, or one student could pick up popularity-contest one. We will discuss this if there is any interest.
Maybe it will require some information from legal team, but from the initiative part, it sounds good :)
Definitely.
Thanks for comments!
On 3 May 2013 06:35, Lukas Zapletal lzap@redhat.com wrote:
Hello,
I have two students interested in diploma thesis called Yum plugin for suggesting packages based on usage:
TL;DR - from anonymized access log, create a database of suggested packages using data mining techniques and provide a Yum plugin that would suggest "Users of vim also installed: ctags, git, ..."
I would suggest that you will need to find a lower level mirror than any of the top level ones. The reason being is that there is a lot of noise in the top levels from other mirrors and various developers using mock/reposync to get packages. [I tried using the dl.fedoraproject.org mirrors for this a while ago and a large % of the yum traffic is from mock builds.] If you go with smaller mirrors you may need more than one.
On Fri, 3 May 2013 14:35:45 +0200 Lukas Zapletal lzap@redhat.com wrote:
Hello,
I have two students interested in diploma thesis called Yum plugin for suggesting packages based on usage:
TL;DR - from anonymized access log, create a database of suggested packages using data mining techniques and provide a Yum plugin that would suggest "Users of vim also installed: ctags, git, ..."
So can you explain how this would work?
How do we know that any particular person who installed yum installed anything else? Are you using IP address to try and see what each IP user installed? I can think of... a lot of ways that won't work. ;)
Another approach might be to work on https://fedorahosted.org/census/ This is the replacement for smolt, but never seems to have gotten very far. It would be an application end users install.
I am gonna create a Fedora Feature wiki page shortly describing this in more detail. Our goal is to offer this project for integration into Fedora later on, at least provide Fedora packages for it.
To do that, we need good source of data. It would be best to collect access logs from one or two main Fedora mirrors. We would provide short script in Python that would parse access logs and anonymize the data (IP address hash-salted) and filtered only relevant data (RPM files from latest Fedora release or updates repositories). That would be phase one which should give us a sample data.
We had a discussion about making our logs public a while back, and I think that discussion ended with us saying the IP addresses wouldn't be safe to publish, even hashed.
http://lists.fedoraproject.org/pipermail/infrastructure/2012-April/011658.ht...
Phase two would be to integrate this script with logrotate and for one Fedora release cycle (Fedora 19) the script would collect relevant anonymized data into a file. Final suggested package database would be created from this file (or maybe files to allow us to move them on the fly out of the stat directory).
The big (legal) question is if we are able to provide this anonymized data to public, or if we want to sign NDA with all people involved. I am CCing Tom for this question.
it's been asked before.
I want to be cautious about this. ;)
I need your help with connecting to relevant people. Any comments are appreciated.
Many thanks and I hope this effort will lead to improving user experience with Fedora packaging.
kevin
On Fri, May 03, 2013 at 09:21:07AM -0600, Kevin Fenzi wrote:
How do we know that any particular person who installed yum installed anything else? Are you using IP address to try and see what each IP user installed? I can think of... a lot of ways that won't work. ;)
The main issue I see is NAT of course, there are couple of scenarios which will generate incorrect data. But the main goal is to find clusters in data and I hope majority of the log records will be one IP = one user.
Another approach might be to work on https://fedorahosted.org/census/ This is the replacement for smolt, but never seems to have gotten very far. It would be an application end users install.
Thanks for the link.
LZ
Just an idea; abrt .it's already in place. On May 3, 2013 3:01 PM, "Lukas Zapletal" lzap@redhat.com wrote:
On Fri, May 03, 2013 at 09:21:07AM -0600, Kevin Fenzi wrote:
How do we know that any particular person who installed yum installed anything else? Are you using IP address to try and see what each IP user installed? I can think of... a lot of ways that won't work. ;)
The main issue I see is NAT of course, there are couple of scenarios which will generate incorrect data. But the main goal is to find clusters in data and I hope majority of the log records will be one IP = one user.
Another approach might be to work on https://fedorahosted.org/census/ This is the replacement for smolt, but never seems to have gotten very far. It would be an application end users install.
Thanks for the link.
LZ
-- Later,
Lukas "lzap" Zapletal irc: lzap #theforeman _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
The framework is already in place, though abrt is bug related ... in your use case it's simply an extension, module I would posit. On May 3, 2013 3:13 PM, "Lukas Zapletal" lzap@redhat.com wrote:
On Fri, May 03, 2013 at 03:04:41PM -0400, Steven Acres wrote:
Just an idea; abrt .it's already in place.
What you mean exactly?
LZ
-- Later,
Lukas "lzap" Zapletal irc: lzap #theforeman _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Fri, 3 May 2013 21:01:06 +0200 Lukas Zapletal lzap@redhat.com wrote:
On Fri, May 03, 2013 at 09:21:07AM -0600, Kevin Fenzi wrote:
How do we know that any particular person who installed yum installed anything else? Are you using IP address to try and see what each IP user installed? I can think of... a lot of ways that won't work. ;)
The main issue I see is NAT of course, there are couple of scenarios which will generate incorrect data. But the main goal is to find clusters in data and I hope majority of the log records will be one IP = one user.
Right, but then this information is security sensitive...
User installed httpd-x.y-Z on YYYY-MM-DD, but on looking you don't see them installing the security update that was released after that -> target.
Or even, user installs foo, foo is insecure and is dropped from fedora, you might know that they have it still installed and can leverage that.
Or you see that user does security updates every friday, so you know they might be vulnerable thursdays.
Also, you may see users install something, but we have no way of knowing if they try it and hate it and remove it right after.
Also, the way our mirroring works, they can get the package from any mirror at all, so we may not see patterns that are there if we could see logs of all mirrors instead of just one.
Another approach might be to work on https://fedorahosted.org/census/ This is the replacement for smolt, but never seems to have gotten very far. It would be an application end users install.
Thanks for the link.
No problem...
kevin
On Fri, May 03, 2013 at 03:30:39PM -0600, Kevin Fenzi wrote:
Right, but then this information is security sensitive...
User installed httpd-x.y-Z on YYYY-MM-DD, but on looking you don't see them installing the security update that was released after that -> target.
Or even, user installs foo, foo is insecure and is dropped from fedora, you might know that they have it still installed and can leverage that.
Or you see that user does security updates every friday, so you know they might be vulnerable thursdays.
Also, you may see users install something, but we have no way of knowing if they try it and hate it and remove it right after.
All true, that's the reason why IP address will never be available from the data.
Also, the way our mirroring works, they can get the package from any mirror at all, so we may not see patterns that are there if we could see logs of all mirrors instead of just one.
Yeah, I did not realized that - this is quite limiting. Taking NAT issue into account, I don't think anymore it is good source of data :-(
On Mon, 6 May 2013 08:32:40 +0200 Lukas Zapletal lzap@redhat.com wrote:
On Fri, May 03, 2013 at 03:30:39PM -0600, Kevin Fenzi wrote:
Right, but then this information is security sensitive...
User installed httpd-x.y-Z on YYYY-MM-DD, but on looking you don't see them installing the security update that was released after that -> target.
Or even, user installs foo, foo is insecure and is dropped from fedora, you might know that they have it still installed and can leverage that.
Or you see that user does security updates every friday, so you know they might be vulnerable thursdays.
Also, you may see users install something, but we have no way of knowing if they try it and hate it and remove it right after.
All true, that's the reason why IP address will never be available from the data.
Sure, if you can see the anonized logs you can still figure out your IP address hash easily, so that could allow you to see for example what other people behind your same NAT/company are installing.
There's lots of weird corner cases here, which is why we decided it wouldn't work last time we visited it. ;(
Also, the way our mirroring works, they can get the package from any mirror at all, so we may not see patterns that are there if we could see logs of all mirrors instead of just one.
Yeah, I did not realized that - this is quite limiting. Taking NAT issue into account, I don't think anymore it is good source of data :-(
Yeah. ;(
kevin
On Thu, May 09, 2013 at 10:43:34AM -0500, Kevin Fenzi wrote:
On Mon, 6 May 2013 08:32:40 +0200 Lukas Zapletal lzap@redhat.com wrote:
On Fri, May 03, 2013 at 03:30:39PM -0600, Kevin Fenzi wrote:
Right, but then this information is security sensitive...
User installed httpd-x.y-Z on YYYY-MM-DD, but on looking you don't see them installing the security update that was released after that -> target.
Or even, user installs foo, foo is insecure and is dropped from fedora, you might know that they have it still installed and can leverage that.
Or you see that user does security updates every friday, so you know they might be vulnerable thursdays.
Also, you may see users install something, but we have no way of knowing if they try it and hate it and remove it right after.
All true, that's the reason why IP address will never be available from the data.
Sure, if you can see the anonized logs you can still figure out your IP address hash easily, so that could allow you to see for example what other people behind your same NAT/company are installing.
There's lots of weird corner cases here, which is why we decided it wouldn't work last time we visited it. ;(
We could create our own mapping of {IP address : complete random value}, and then hash those random values, and give out the info that way. This solves the problem of reversing the the simple {IP address : hash(IP address)} scheme.
It would require us to generate such a mapping, and keep it private, though.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On Fri, May 03, 2013 at 09:01:06PM +0200, Lukas Zapletal wrote:
On Fri, May 03, 2013 at 09:21:07AM -0600, Kevin Fenzi wrote:
How do we know that any particular person who installed yum installed anything else? Are you using IP address to try and see what each IP user installed? I can think of... a lot of ways that won't work. ;)
The main issue I see is NAT of course, there are couple of scenarios which will generate incorrect data. But the main goal is to find clusters in data and I hope majority of the log records will be one IP = one user.
Another issue I could see is laptops: as they are often online on other places then home, and Fedora doesn't consider your location before warning you for updates, they might install updates or new packages from multiple IP addresses, and there would be no way to tell that's still the same user.
Patrick
infrastructure@lists.fedoraproject.org
-
Kevin Fenzi -
Lukas Zapletal -
Matt Domsch -
Pablo Iranzo Gómez -
Patrick マルタインアンドレアス Uiterwijk -
Stephen John Smoogen -
Steven Acres