For a while now I've been working on a webapp/database called mirrormanager to track the status of all the various mirror systems that you graciously administer. Right now, all the tracking (IP addresses for the ACLs, yum mirror lists, static web pages listing the mirrors by country, etc) are all manual - someone (lately, me), has to edit a bunch of files for each change (add, remove, move a mirror). So, I got tired and wrote a program to do that for me.
At this point, I'm looking for volunteers to try out the site, enter their data, and let me know if something causes a python backtrace and what you did to get there. My goal is to shake enough bugs out that it can be used *LIVE* for the Fedora 7 launch.
https://admin.fedoraproject.org/mirrormanager
You need to have a Fedora Account System account. You don't need to sign the Contributor License Agreement, but you will have to accept the Export Compliance statement listed there when you create a new Site.
There will be an app that crawls your web site (either http or ftp), at least once a day, to discover what content you're actually carrying. It uses HTTP HEAD or FTP DIR commands and keep-alives, so it should be fast and not a huge load.
Source Code to mirrormanager is under the MIT/X11 license, and is posted here: https://hosted.fedoraproject.org/projects/mirrormanager
The "Grand Idea" doc is here: http://fedoraproject.org/wiki/Infrastructure/MirrorManagement
Some things you can do: Create a new Site. The Site is the administrative unit. You can add admins for your Site, which additional Fedora Account System usernames, then they can do things with your Site's data.
Create Hosts for your Site. Hosts are the individual systems that you use to serve data. If you're doing a round robin DNS, only list one Host. But you can have many hosts.
Hosts can have their own lists of IP ACLs, local netblocks (so we can eventually try to route your local traffic to you automagically), a list of allowed countries (some places can only serve traffic to a limited number of countries), and the like.
Create Categories (Fedora Core, Fedora Extras, ...) for your Host. More categories will exist as the content becomes available in the masters - e.g. Fedora EPEL and Fedora Releases (for F7).
Create URLs (http, ftp, rsync) for your Categories. You can list URLs which are "Private", meaning they're for use by other mirrors only. In this way we can start to create a tiering system, but with non-publicly-published URLs.
Create SiteToSites. This lets other Sites' admins see the Private URLs.
See the public mirror list.
See the rsync ACL list. The goal is to replace most of ftp@redhat.com with a script that reads this list once a day. :-)
Wanna help? ==> fedora-infrastructure-list@redhat.com.
With your help, this should smooth out the mirror load, and make release day go very easily.
Thanks, Matt Fedora Mirror Wrangler
On Tue, 2007-03-13 at 12:47 -0500, Matt Domsch wrote:
At this point, I'm looking for volunteers to try out the site, enter their data, and let me know if something causes a python backtrace and what you did to get there. My goal is to shake enough bugs out that it can be used *LIVE* for the Fedora 7 launch.
https://admin.fedoraproject.org/mirrormanager/publiclist/2/2
Gives me a 404 (linked from https://admin.fedoraproject.org/mirrormanager/publiclist/)
Jeff
N00b question.
I went to the URL indicated below (https://admin.fedoraproject.org/mirrormanager). Not being able to log in, I followed the "register" link on that page to the accounts home, went to edit group memberships, typed in "infrastructure" (a guess, since I don't know where to get to a list of valid group names), role of user and submitted. I promptly received a mail asking me to approve myself (https://admin.fedora.redhat.com/accounts/groupbox.cgi?_editme=Edit&name=...), but when I follow the URL in said mail it tells me I'm not allowed administer the group :-o
What if anything do I need to do in order to get going here?
-- Toivo Voll Data Network Management Academic Computing University of South Florida -----Original Message----- From: mirror-list-d-bounces@redhat.com [mailto:mirror-list-d-bounces@redhat.com] On Behalf Of Matt Domsch Sent: Tuesday, March 13, 2007 13:48 To: mirror-list-d@redhat.com Cc: fedora-infrastructure-list@redhat.com Subject: Testers needed: mirrormanager
For a while now I've been working on a webapp/database called mirrormanager to track the status of all the various mirror systems that you graciously administer. Right now, all the tracking (IP addresses for the ACLs, yum mirror lists, static web pages listing the mirrors by country, etc) are all manual - someone (lately, me), has to edit a bunch of files for each change (add, remove, move a mirror). So, I got tired and wrote a program to do that for me.
At this point, I'm looking for volunteers to try out the site, enter their data, and let me know if something causes a python backtrace and what you did to get there. My goal is to shake enough bugs out that it can be used *LIVE* for the Fedora 7 launch.
https://admin.fedoraproject.org/mirrormanager
You need to have a Fedora Account System account. You don't need to sign the Contributor License Agreement, but you will have to accept the Export Compliance statement listed there when you create a new Site.
There will be an app that crawls your web site (either http or ftp), at least once a day, to discover what content you're actually carrying. It uses HTTP HEAD or FTP DIR commands and keep-alives, so it should be fast and not a huge load.
Source Code to mirrormanager is under the MIT/X11 license, and is posted here: https://hosted.fedoraproject.org/projects/mirrormanager
The "Grand Idea" doc is here: http://fedoraproject.org/wiki/Infrastructure/MirrorManagement
Some things you can do: Create a new Site. The Site is the administrative unit. You can add admins for your Site, which additional Fedora Account System usernames, then they can do things with your Site's data.
Create Hosts for your Site. Hosts are the individual systems that you use to serve data. If you're doing a round robin DNS, only list one Host. But you can have many hosts.
Hosts can have their own lists of IP ACLs, local netblocks (so we can eventually try to route your local traffic to you automagically), a list of allowed countries (some places can only serve traffic to a limited number of countries), and the like.
Create Categories (Fedora Core, Fedora Extras, ...) for your Host. More categories will exist as the content becomes available in the masters - e.g. Fedora EPEL and Fedora Releases (for F7).
Create URLs (http, ftp, rsync) for your Categories. You can list URLs which are "Private", meaning they're for use by other mirrors only. In this way we can start to create a tiering system, but with non-publicly-published URLs.
Create SiteToSites. This lets other Sites' admins see the Private URLs.
See the public mirror list.
See the rsync ACL list. The goal is to replace most of ftp@redhat.com with a script that reads this list once a day. :-)
Wanna help? ==> fedora-infrastructure-list@redhat.com.
With your help, this should smooth out the mirror load, and make release day go very easily.
Thanks, Matt Fedora Mirror Wrangler
On Tue, Mar 13, 2007 at 12:47:48PM -0500, Matt Domsch wrote:
You need to have a Fedora Account System account. You don't need to sign the Contributor License Agreement, but you will have to accept the Export Compliance statement listed there when you create a new Site.
Isn't the CLA supposed to contain the Export Compliance statement? I had signed the CLA some long time ago, and it still wants the Export Compliance statement from me. Or am I mixing apples and oranges?
Axel Thimm wrote:
On Tue, Mar 13, 2007 at 12:47:48PM -0500, Matt Domsch wrote:
You need to have a Fedora Account System account. You don't need to sign the Contributor License Agreement, but you will have to accept the Export Compliance statement listed there when you create a new Site.
Isn't the CLA supposed to contain the Export Compliance statement? I had signed the CLA some long time ago, and it still wants the Export Compliance statement from me. Or am I mixing apples and oranges?
I'm not sure if the CLA covers this (I don't think it does) but I do know that you don't need a CLA to be a mirror but you do need to sign the export agreement. Really we're not the right people to answer a question like that, anyone know how legal questions like this should be handled?
-Mike
Mike McGrath wrote:
Axel Thimm wrote:
On Tue, Mar 13, 2007 at 12:47:48PM -0500, Matt Domsch wrote:
You need to have a Fedora Account System account. You don't need to sign the Contributor License Agreement, but you will have to accept the Export Compliance statement listed there when you create a new Site.
Isn't the CLA supposed to contain the Export Compliance statement? I had signed the CLA some long time ago, and it still wants the Export Compliance statement from me. Or am I mixing apples and oranges?
I'm not sure if the CLA covers this (I don't think it does) but I do know that you don't need a CLA to be a mirror but you do need to sign the export agreement. Really we're not the right people to answer a question like that, anyone know how legal questions like this should be handled?
You can contact Max Spevack or/and Greg Dek and they should talk to Red Hat legal.
Rahul
On Tue, 2007-03-13 at 14:40 -0500, Mike McGrath wrote:
Axel Thimm wrote:
Isn't the CLA supposed to contain the Export Compliance statement? I had signed the CLA some long time ago, and it still wants the Export Compliance statement from me. Or am I mixing apples and oranges?
I'm not sure if the CLA covers this (I don't think it does) but I do know that you don't need a CLA to be a mirror but you do need to sign the export agreement. Really we're not the right people to answer a question like that, anyone know how legal questions like this should be handled?
I don't think you need a lawyer here. A read of the CLA makes it clear that it does not cover export control:
http://fedoraproject.org/wiki/Legal/Licenses/CLA
I've submitted my plain-English CLA to the appropriate legal counsel for approval -- that means this cannot be relied upon yet -- but it is at least close to the mark, and may help in clarifying for ya'll what the CLA is and is not:
http://fedoraproject.org/wiki/KarstenWade/Drafts/PlainEnglishCLA
IANAL, TINLA, YMMV, caveat emptor, etcetera.
- Karsten
On Tue, 2007-03-13 at 12:47 -0500, Matt Domsch wrote:
I tried to login and got a 500 Internal Error:
500 Internal error The server encountered an unexpected condition which prevented it from fulfilling the request.
Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 103, in _run applyFilters('before_main') File "/usr/lib/python2.4/site-packages/cherrypy/filters/__init__.py", line 151, in applyFilters method() File "/usr/lib/python2.4/site-packages/turbogears/visit/api.py", line 157, in before_main plugin.record_request( visit ) File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 165, in record_request identity= self.identity_from_request(visit.key) File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 89, in identity_from_request identity= source(visit_key) File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 146, in identity_from_form identity= self.provider.validate_identity( user_name, pw, visit_key ) File "/usr/lib/python2.4/site-packages/fedora/tg/identity/safasprovider.py", line 105, in validate_identity except fedora.accounts.AuthError, e: NameError: global name 'fedora' is not defined
I am able to login to the https://admin.fedora.redhat.com/accounts site with a FAS account.
/Brian/
Try again, Toshio fixed it.
-- Matt Domsch Software Architect Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com
-----Original Message----- From: Brian Long [mailto:brilong@cisco.com] Sent: Tuesday, March 13, 2007 2:11 PM To: Domsch, Matt; A private discussion group for official mirrors of ftp.redhat.com Cc: fedora-infrastructure-list@redhat.com Subject: Re: Testers needed: mirrormanager
On Tue, 2007-03-13 at 12:47 -0500, Matt Domsch wrote:
I tried to login and got a 500 Internal Error:
500 Internal error The server encountered an unexpected condition which prevented it from fulfilling the request.
Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 103, in _run applyFilters('before_main') File "/usr/lib/python2.4/site-packages/cherrypy/filters/__init__.py", line 151, in applyFilters method() File "/usr/lib/python2.4/site-packages/turbogears/visit/api.py", line 157, in before_main plugin.record_request( visit ) File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 165, in record_request identity= self.identity_from_request(visit.key) File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 89, in identity_from_request identity= source(visit_key) File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 146, in identity_from_form identity= self.provider.validate_identity( user_name, pw, visit_key ) File "/usr/lib/python2.4/site-packages/fedora/tg/identity/safasprovider.py", line 105, in validate_identity except fedora.accounts.AuthError, e: NameError: global name 'fedora' is not defined
I am able to login to the https://admin.fedora.redhat.com/accounts site with a FAS account.
/Brian/
On Tue, 13 Mar 2007 14:13:27 -0500 Matt_Domsch@Dell.com wrote:
Try again, Toshio fixed it.
Seems to be broken again:
500 Internal error
The server encountered an unexpected condition which prevented it from fulfilling the request. Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 103, in _run applyFilters('before_main') File "/usr/lib/python2.4/site-packages/cherrypy/filters/__init__.py", line 151, in applyFilters method() File "/usr/lib/python2.4/site-packages/turbogears/visit/api.py", line 157, in before_main plugin.record_request( visit ) File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 165, in record_request identity= self.identity_from_request(visit.key) File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 89, in identity_from_request identity= source(visit_key) File "/usr/lib/python2.4/site-packages/turbogears/identity/visitor.py", line 146, in identity_from_form identity= self.provider.validate_identity ( user_name, pw, visit_key ) File "/usr/lib/python2.4/site-packages/fedora/tg/identity/safasprovider.py", line 105, in validate_identity except fedora.accounts.AuthError, e: NameError: global name 'fedora' is not defined
Andreas aka ScottyTM
On Wed, Mar 14, 2007 at 09:59:59PM +0100, Andreas Kupfer wrote:
On Tue, 13 Mar 2007 14:13:27 -0500 Matt_Domsch@Dell.com wrote:
Try again, Toshio fixed it.
Seems to be broken again:
500 Internal error
NameError: global name 'fedora' is not defined
Toshio just found and fixed the FAS authentication code for real. Try now.
Thanks for your patience. -Matt
On Wed, 2007-03-14 at 16:17 -0500, Matt Domsch wrote:
On Wed, Mar 14, 2007 at 09:59:59PM +0100, Andreas Kupfer wrote:
On Tue, 13 Mar 2007 14:13:27 -0500 Matt_Domsch@Dell.com wrote:
Try again, Toshio fixed it.
Seems to be broken again:
500 Internal error
NameError: global name 'fedora' is not defined
Toshio just found and fixed the FAS authentication code for real. Try now.
It looks like another fix I made yesterday caused a regression WRT this traceback. Go ahead and give it another try and let me know if it's still giving you issues.
Thanks, -Toshio
On Thu, 15 Mar 2007 11:10:50 -0700 Toshio Kuratomi a.badger@gmail.com wrote:
It looks like another fix I made yesterday caused a regression WRT this traceback. Go ahead and give it another try and let me know if it's still giving you issues.
Works for me now.
Andreas aka ScottyTM
I'll signup.
Where do I go.. shane@bratnet.net ( my personal account ) but I manage mirrors@tds.net :)
Shane
Matt Domsch wrote:
For a while now I've been working on a webapp/database called mirrormanager to track the status of all the various mirror systems that you graciously administer.[...] At this point, I'm looking for volunteers to try out the site, enter their data, and let me know if something causes a python backtrace and what you did to get there. My goal is to shake enough bugs out that it can be used *LIVE* for the Fedora 7 launch. https://admin.fedoraproject.org/mirrormanager
Is there any reason why a GPG key ID & a SSH public key are needed to create an account ? Several people here manage the FTP mirrors server and we usually create a single account for mirrors management web interfaces. I could create a "fake" account (ie it would not be an official company account) but it is probably not the right way to solve the problem.
François
Francois Petillon wrote:
Is there any reason why a GPG key ID & a SSH public key are needed to create an account ? Several people here manage the FTP mirrors server and we usually create a single account for mirrors management web interfaces. I could create a "fake" account (ie it would not be an official company account) but it is probably not the right way to solve the problem.
For the most part people need the ssh key to access cvs though for general access I think it should be allowed to be blank. The GPG key is part of our current electronic signature process for the CLA, again should be allowed to be blank until the user wants to sign the CLA. We're working on writing a new account system we can examine those needs again.
-Mike
On Wed, Mar 14, 2007 at 08:27:35AM -0500, Mike McGrath wrote:
Francois Petillon wrote:
Is there any reason why a GPG key ID & a SSH public key are needed to create an account ? Several people here manage the FTP mirrors server and we usually create a single account for mirrors management web interfaces. I could create a "fake" account (ie it would not be an official company account) but it is probably not the right way to solve the problem.
For the most part people need the ssh key to access cvs though for general access I think it should be allowed to be blank. The GPG key is part of our current electronic signature process for the CLA, again should be allowed to be blank until the user wants to sign the CLA. We're working on writing a new account system we can examine those needs again.
As you don't need to sign the CLA, you can choose to send your real GPG public key (preferred), or you can create a key which you then don't use. Likewise for the ssh public key - just create a keypair that you'll never use.
And yes, folks working on the new account system know this seems odd and want to take care of it in the next version. :-)
Thanks, Matt
The app worked fine tomorrow. Now I'm getting a 502 Bad Gateway after waiting for about 30 seconds for the main page to load.
Matt Domsch wrote:
With your help, this should smooth out the mirror load, and make release day go very easily.
infrastructure@lists.fedoraproject.org
-
Andreas Kupfer -
Axel Thimm -
Brian Long (brilong) -
Francois Petillon -
Jean-Francois Chevrette -
Jeffrey Ollie -
Karsten Wade -
Matt Domsch -
Matt Domsch
-
Mike McGrath -
Rahul Sundaram -
Shane Brath <mirrors＠tds.net> -
Toshio Kuratomi -
Voll, Toivo