infrastructure June 2015

infrastructure@lists.fedoraproject.org

30 participants
43 discussions

[release] pagure: 0.1.15
by Pierre-Yves Chibon 16 Jun '15

16 Jun '15

Good morning everyone, Another day, another release of pagure: 0.1.15 :) Smaller changelog this time, but the bugs were annoying: * Tue Jun 16 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.1.15-1 - Update 0.1.15 - Use a monospace font for the commit hash - Remove duplicated "commit" id in the HTML (causing a graphical bug in the commit page) - Secure the input using the no_js filter instead of relying on a restrictive regex for PR and issue titles (yangl1996) - Support ',' in the tags field since it's required to specify multiple tags Enjoy! Pierre

1 0

Meeting Agenda Item: Introduction Jean Vicelli
by jean vicelli 15 Jun '15

15 Jun '15

Hello Fedora-Infrastructure! My name is Jean( IRC: jcvicelli ), I'm from Curitiba, Brazil and I've been working with infrastructure for a while, with both Linux and Windows servers. I'm also a bash script and python developer! I can easyly spend 4 to 5 hours a week contributing, out of my work time(utc -3).. Appreciate some help to get started! -- att. *Jean Carlo Vicelli* <http://br.linkedin.com/in/jeanvicelli>

1 0

[release] pagure: 0.1.14
by Pierre-Yves Chibon 15 Jun '15

15 Jun '15

Good morning everyone, I have just released and pushed a new version of pagure: 0.1.14 The changelog is pretty big and quite a number of bugs have been fixed and RFE implemented. Here it is: * Fri Jun 12 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.1.14-1 - Update to 0.1.14 - Remove all new lines characters from the ssh key uploaded - Adjust the URL in the footer to point to https://pagure.io/pagure - Fix displaying the time of a comment - Forbid the use of spaces in group name - Do not get the list of not-merged commits if there is only 1 branch in the repo - Display the error message if pagure.lib.add_group raises an exception - Add a new setting enforcing that all commits in a PR are signed-off by their author - Enforce that all commits are signed-off by the author if the repo is configured for this - Also check for the signed-off status before merging a pull-request - Adjust online-editing to allow specifying which email address to use in the commit - Add an avatar_email field to projects - Change the PullRequest's status from a Boolean to a Text restricted at the DB level (Allows to distinguish Open/Merged/Closed) - Show in the pull-request view who merged the pull-request - Specify who closed the pull-request in the API output - Catch GitError when merging and checking merge status of a PR - Hide the form to create pull-requests if the user is not an admin of the repo - Replace the Pull-Request button by a Compare button if the user it not a repo admin - Set the title of the tab as URL hash to allow directly linking to it - Adjust the API to be able to distinguish API authentication and UI authentication - Fix API documentation to create new issues - Drop the status from the requirements to open a new issue via the API - Expand the list of blacklisted project names - Have the code tags behave like pre tags (html tags) - Allow project to specify an URL and display it on their page - Strip the ssh keys when writing them to the authorized_keys file - Disable javascript in all the markdown fields - Validate early the input submitted in the forms (using more or less strict regex) - If the session timed-out, redirect to the setting page after authentication and inform the user that the action was canceled - Catch PagureException when adjusting the project's settings - Redirect the /api endpoint to the api documentation place - Fix how is retrieved the list of emails to send the notification to - Sanitize the html using bleach to avoid potential XSS exploit - Do not give READ access to everyone on the tickets and pull-requests repos to avoid leaking private tickets - Adjust the unit-tests for all these changes Hope you like it! Pierre

1 0

[release] fedocal 0.14
by Pierre-Yves Chibon 15 Jun '15

15 Jun '15

Good morning everyone, I have just released a new fedocal version: 0.14 Here is its changelog: * Mon Jun 15 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.14-1 - Update to 0.14 - Add an API endpoint presenting a shield if the user is currently in a meeting - Sanitize the text generated by markdown to avoid potential XSS Deployed in both stg and prod. Enjoy! Pierre

1 0

MM related changes
by Adrian Reber 12 Jun '15

12 Jun '15

I have a few MM related ansible changes which I would like to get reviewed before committing them. * The crawler logs are now on mm-crawler01 - relative simple change * Increase cache time for mirrorlist - This tries to increase the cache time from 2 minutes to 6 hours for the mirrorlist. I have tested it on a local varnish installation and it seems to work. * Use the new api interface for Nagios checks - This changes two nagios checks to use the api interface which performs a much simpler database query. Adrian commit 477f2ac02e30be805eacffda47c7c926f25bc892 Author: Adrian Reber <adrian(a)lisas.de> Date: Fri Jun 12 08:20:24 2015 +0000 The crawler logs are now on mm-crawler01 diff --git a/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf b/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf index 44987a1..8ea3700 100644 --- a/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf +++ b/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf @@ -4,8 +4,8 @@ SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1 </Location> -ProxyPass {{localpath}}/crawler http://bapp02/mirrormanager/crawler -ProxyPassReverse {{localpath}}/crawler http://bapp02/mirrormanager/crawler +ProxyPass {{localpath}}/crawler http://mm-crawler01/mirrormanager/crawler +ProxyPassReverse {{localpath}}/crawler http://mm-crawler01/mirrormanager/crawler ProxyPass {{localpath}} {{proxyurl}}{{remotepath}} ProxyPassReverse {{localpath}} {{proxyurl}}{{remotepath}} commit 8e09f5a37620c52265aa83305c99173f9e982d76 Author: Adrian Reber <adrian(a)lisas.de> Date: Fri Jun 12 08:13:17 2015 +0000 Increase cache time for mirrorlist The mirrorlist which can be viewed in the browser used to be generated once or twice per day with MM1. This is listed is now cached in varnish but only for two minutes. This patch increases the cache time to 6 hours. diff --git a/roles/varnish/files/proxy.vcl b/roles/varnish/files/proxy.vcl index 0fd5aa6..6d2f340 100644 --- a/roles/varnish/files/proxy.vcl +++ b/roles/varnish/files/proxy.vcl @@ -303,5 +303,6 @@ sub vcl_recv { sub vcl_backend_response { if (bereq.url ~ "^/mirrormanager/mirrors") { unset beresp.http.set-cookie; + set beresp.ttl = 6h; } } commit 498f16861fbde5190b6d4fbab731185fb01c7662 Author: Adrian Reber <adrian(a)lisas.de> Date: Fri Jun 12 07:28:38 2015 +0000 MM: Use the new api interface for Nagios checks The newly introduced api interface provides a way to query the MM frontend for availability which only performs a minimal database query. Using this interface should reduce the load on the frontend and the database. diff --git a/roles/nagios_server/files/nagios-external/services/websites.cfg b/roles/nagios_server/files/nagios-external/services/websites.cfg index 55ad34a..0787e53 100644 --- a/roles/nagios_server/files/nagios-external/services/websites.cfg +++ b/roles/nagios_server/files/nagios-external/services/websites.cfg @@ -145,7 +145,7 @@ define service { define service { host_name 209.132.181.16-phx2, 85.236.55.6-internetx, proxy03.fedoraproject.org, 152.19.134.142-ibiblio, proxy06.fedoraproject.org, 213.175.193.206-bodhost, 67.203.2.67-coloamerica, 66.135.62.187-serverbeach service_description mirrors.fedoraproject.org - publiclist - check_command check_website_ssl!admin.fedoraproject.org!/mirrormanager/!Fedora Public Active Mirrors + check_command check_website_ssl!admin.fedoraproject.org!/mirrormanager/api/mirroradmins/?name=dl.fedoraproject.org!admins use websitetemplate } diff --git a/roles/nagios_server/files/nagios/services/websites.cfg b/roles/nagios_server/files/nagios/services/websites.cfg index 1beda33..101c115 100644 --- a/roles/nagios_server/files/nagios/services/websites.cfg +++ b/roles/nagios_server/files/nagios/services/websites.cfg @@ -46,7 +46,7 @@ define service { define service { host_name mm-frontend01,mm-frontend01.stg service_description mm-publiclist-internal - check_command check_website!localhost!/mirrormanager/ + check_command check_website!localhost!/mirrormanager/api/mirroradmins/?name=dl.fedoraproj… use internalwebsitetemplate

3 4

[PATCH] distgit: Upload files to both the new and old path
by Mathieu Bridon 12 Jun '15

12 Jun '15

From: Mathieu Bridon <bochecha(a)daitauha.fr> Currently, the CGI script is set to upload files: - to the old path if the upload uses md5 - to the new path if the upload uses sha512 The old path is as follows: /%(srpmname)s/%(filename)s/%(hash)s/%(filename)s The new path is: /%(srpmname)s/%(filename)s/%(hashtype)s/%(hash)s/%(filename)s This was meant to ensure compatibility with current fedpkg which always downloads from the old path, but will eventually download from the new path when we move to sha512. However, working more on this, I now think it would make for a smoother transition if we instead always stored the files at the new path, but just hardlinked to the old path if the upload is using md5. This is what this patch achieves. With this deployed in production, fedpkg could be patched to try downloading from the new path, and fallback to the old one if necessary, which decouples the migration to the new path from the migration to the new hash. --- roles/distgit/files/dist-git-upload.cgi | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/roles/distgit/files/dist-git-upload.cgi b/roles/distgit/files/dist-git-upload.cgi index b4fda74..38c40db 100644 --- a/roles/distgit/files/dist-git-upload.cgi +++ b/roles/distgit/files/dist-git-upload.cgi @@ -112,11 +112,6 @@ def main(): hash_dir = os.path.join(module_dir, filename, hash_type, checksum) msgpath = os.path.join(name, module_dir, filename, hash_type, checksum, filename) - if hash_type == "md5": - # Preserve compatibility with the current folder hierarchy for md5 - hash_dir = os.path.join(module_dir, filename, checksum) - msgpath = os.path.join(name, module_dir, filename, checksum, filename) - unwanted_prefix = '/srv/cache/lookaside/pkgs/' if msgpath.startswith(unwanted_prefix): msgpath = msgpath[len(unwanted_prefix):] @@ -180,6 +175,11 @@ def main(): print >> sys.stderr, '[username=%s] Stored %s (%d bytes)' % (username, dest_file, filesize) print 'File %s size %d %s %s stored OK' % (filename, filesize, hash_type.upper(), checksum) + # Add the file to the old path, where fedpkg is currently looking for it + if hash_type == "md5": + old_path = os.path.join(module_dir, filename, checksum, filename) + os.link(dest_file, old_path) + # Emit a fedmsg message. Load the config to talk to the fedmsg-relay. try: config = fedmsg.config.load_config([], None) -- 2.1.0

7 21

June status update for Fedora Infrastructure Apprentices
by Kevin Fenzi 12 Jun '15

12 Jun '15

Greetings. You are getting this email because you are in the 'fi-apprentice' group in the fedora account system (or are reading this on the infrastructure list). Feel free to reply just directly to me, or cc the infrastructure list for everyone to see and comment on. https://fedoraproject.org/wiki/Infrastructure_Apprentice At the first of every month(or so), I am going to be sending out an email like this one. I would like feedback on how things are going for you. I'd like to ask for everyone to send me a quick reply with the following data or anything related you can think of that might help us make the apprentice program more useful. 0. Whats your fedora account system login? 1. Have you logged in and used your fi-apprentice membership to look at our machines/setup in the last month? Do you plan to? 2. Has it helped you decide any area you wish to focus on or contribute to more? 3. Have you looked at or been able to work on any of the fi-apprentice 'easyfix' tickets? https://fedorahosted.org/fedora-infrastructure/report/14 4. Do you still wish to be a member of the group? If not (for whatever reason) could you provide any hints to help others down the road? 5. Is there any help or communication or ideas you have that would help you do any of the above? 6. What do you find to be the hardest part of getting involved? Finding things to work on? Getting attention from others to help you? Finding tickets in your interest area? 7. Have you been able to make any weekly irc meetings? Do you find them helpful or interesting? 8. Have you logged into our Gobby instance and read/seen/added to our meeting agenda? https://fedoraproject.org/wiki/Gobby 9. If you could go to any place on earth for a vacation, where would it be? Any other general feedback is also quite welcome, including improvements to this email, the wiki page, etc. Any folks I do not hear from in the next week will be removed from the group. (Note that it's easy to be readded when you have time or whatever and it's nothing at all personal, we just want to keep the group up to date with active folks). Thanks, and looking forward to your feedback! kevin

8 7

Plan for tomorrow's Fedora Infrastructure meeting (2015-06-11)
by Kevin Fenzi 10 Jun '15

10 Jun '15

The infrastructure team will be having it's weekly meeting tomorrow, 2015-06-11 at 18:00 UTC in #fedora-meeting on the freenode network. This week we are continuing to try something new. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of this morning is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. kevin -- = Introduction = This shared document is for the next fedora infrastructure meeting. We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2015-06-11) #meetingname infrastructure #topic aloha #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson #topic New folks introductions / Apprentice feedback #topic GSoC student update - kushal = Status / information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info value01/value01.stg reinstalled as rhel7 - kevin #info new s390 koji hub and db setup for testing - kevin #info NFS storage migrated over to new netapp - kevin / smooge / patrick #info value01.stg/value01 moved to rhel7 - kevin #info all builders updated/rebooted - kevin = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus or decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the info in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.) #topic Learn about: = Meeting end stuff = #topic Open Floor #endmeeting

1 0

[release] MirrorManager2: 0.2.0
by Pierre-Yves Chibon 07 Jun '15

07 Jun '15

Good morning everyone, Here is another release of the day, MirrorManager2: 0.2.0 And here is the changelog: * Fri Jun 05 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.2.0-1 - Update to 0.2.0 - Include the background header file in MM2 itself (Adrian Reber) - Support always update hosts which are unreachable in the crawler (Adrian Reber) - Adjust the spec file to the systemd packaging guidelines for Fedora - Multiple improvements to the crawler, including a start of a canary mode (Adrian Reber) - Offer possibility to sort by product, bringing back MM1 behavior (Adrian Reber) - Couple of UI fixes about who is allowed to access what - Fix peer ASNs (in the same spirit, who can access) - Create noauthed master for mirror publiclist so that it can be cached in memcachd (Patrick Uiterwijk) - Fix the report_mirror to correctly catch the xmlrpclib.ProtocolError - Add a new utility script to upgrade repo from -alpha or -beta to release - Adjust the logrotate configuration to fix the permission denied error - Create 2 API endpoints, one for zodbot's .mirroradmin and one for nagios Note: this is currently only in *staging*, we did not feel like pushing this to prod on a Friday afternoon :) We'll likely push on Monday though, assuming nothing burns down over the week-end. Thanks, Pierre

2 1

Followup on koji staging setup
by Ralph Bean 05 Jun '15

05 Jun '15

This is one part a product of discussion from the FAD: https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015 and another part response to the questions about requirements from April: https://lists.fedoraproject.org/pipermail/infrastructure/2015-April/016121.… > We now would like to add some requirements I think. What are they? :) > > Do we need to do rawhide/branched composes? Daily? > > Do we need to be able to do run-pungi (RC/TC) test composes? > > Something else? > > Lets find out our requirements before we adjust things. So, here are some requirements coming out of the FAD: 1) to do rawhide/branched and RC/TC test composes. Hopefully, soon rawhide composes will look more like RC/TC composes so we won't need to make the distinction. 2) specifically we want to do this so we can develop/iterate the composition tools 3) therefore, we need to be able to do them relatively rapidly 4) furthermore, we need it to be as-like production as possible so we know what we're testing. At the FAD, we came up with four different strategies to get there, with different pros/cons: # Leverage koji's secondary volume capabilities. # Apparently, koji can support mounting multiple volumes. We could: - Mount the prod /mnt/koji read-only in staging and tell staging koji that this will be it's "secondary volume". - Dump the prod DB and import it into staging. - Run a sql script that re-points all the rpm entries from what was the prod primary volume to now the staging secondary volume (which is just the prod primary volume, but mounted read-only). We could run full composes in staging here since all the rpm data would be available, just like in prod. The one problem would be that we could not do hardlinks on the read-only mount, so we would need to make that portion of the compose process optional.. or work around it some other way. For what it's worth, I personally like this option the most. Staging koji would drift out of sync with production over time, but we could re-sync it every week or month or so. We could automate that with ansible and cron. # Leverage writable snapshots from the netapp # The netapp we use for storage could potentially provide a "writable snapshot", which could possibly make for an easy-win solution. If we could snapshot the prod mount and then remount it in staging as a writable snapshot, (and then also take a dump of the prod database and import it in staging), then staging could just write on top of that snapshot, without affecting prod. We would just need to take a new snapshot every so often to avoid growing beyond our bounds in staging. If this is even possible (we have to ask), we wouldn't be able to easily automate syncing from prod, since it seems like it would require filing a ticket to get a new snapshot each time. Correct me if I'm wrong, here. A nice aspect of this solution is that it involves zero koji-specific compose-specific knowledge and tooling to pull it off. # Write a custom "koji snapshotter" tool # This approach involves writing a custom tool that knows about koji. It would inspect prod koji and copy over a subset of the content and db relations required to do a compose in staging. This involves no mount fanciness, but it does involve a high degree of special development against koji's API. It could work great, but we'd have to re-do it when koji2.0 comes out (eventually, eventually). # Just fully copy all 30T of prod koji # imcleod say that for less than the cost of the FAD, we could get a storage pod full of disks and have all the space in staging that we want: https://www.backblaze.com/blog/backblaze-storage-pod-4/ With this, our test composes could potentially be faster than in prod, which is nice for testing. However, I can't speak to how easy or not it is or not to get, install, and maintain hardware in our environment. Any input on which of these options is preferable would be appreciated.

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure June 2015