Good Morning Everyone,
I would like to upgrade pagure in prod to 2.15.1, it's a bug fix release from
2.15 that brings upstream some of the hotfix we had to do.
Changelog is something like:
- Fix the requirements on straight.plugin in the requirements.txt file
- Fix typo in the fedmsg hook so it finds the function where it actually is
- Fix and increase the logging when merging a PR
- Fix pushing a merge commit to the original repo
- Use psutil's Process() instead of looping through all processes (Patrick
- Don't email admins for each PR conflicting
- Fix/improve our new locking mechanism (Patrick Uiterwijk)
- Drop making the token required at the database level since pagure-ci doesn't
use one (but do flag pull-requests)
- Fix the watch feature (Matt Prahl)
The changes #2, #3, #4 where hotfix in prod, the last one as well but turned out
the hotfix was incorrect and didn't fix the issue.
My idea, if people agree, is to release 2.15.1 tomorrow morning and push it to
stg then prod before the US wakes up, and rollback if something breaks.
People have noticed that they aren't getting some FMN notifications, so
I spent the afternoon tracking down the issue. It looks as though the
redis instance on notifs-backend01.phx2.fedoraproject.org is configured
to persist to disk (this is apparently the default config, which is
surprising to me). In addition, dogpile.cache is apparently not setting
a TTL on keys despite being configured to do so.
All this means that everything in the FMN redis cache is _really_ stale
and contains invalid objects. For example, the packages a user has ACLs
on is cached as a set currently, but the latest version of FMN expects
a dictionary. This leads to messages not getting sent that should have
I would like to delete the persisted database (/var/lib/redis/dump.rdb)
and restart redis to wipe the cache. FMN does not like redis going down
so the workers and backend will likely need to be restarted, and because
of a different caching issue this will likely take around an hour
(messages won't be lost, they'll just build up in the rabbitmq queue).
The infrastructure team will be having it's weekly meeting tomorrow,
2017-05-18 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of
today is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
= Introduction =
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2017-05-18)
#chair smooge relrod nirik abadger1999 dgilmore threebean pingou
= Let new people say hello =
#topic New folks introductions
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need
(Please use #info <the thing> - your name)
#topic announcements and information
#info we are now in f26 Beta freeze - everyone
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Recap of 2017 CI/Infrastructure hackfest - kevin
#topic Apprentice work day next week (2017-05-23) - kevin
= Apprentice office hours =
#topic Apprentice Open office hours
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
= Learn about some application or gsetup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for
etc. Whoever would like to do this, just add the i/nfo in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about:
= Meeting end stuff =
#topic Open Floor
Stephen J Smoogen.
I'm planning to set up a pagure instance in the cloud as a place to
collect testcases that Red Hat will be releasing for upstream
consumption and collaborate on the tests as they change into a form that
could be accepted by Fedora package maintaners. The request is tracked
in the following ticket:
I'd like to add upstreamfirst.fedorainfracloud.org to DNS, build the
config and push it out to the DNS servers. That hostname would point to
one of the floating IPs we've allocated in openstack.
Good Morning Everyone,
Earlier today I cut a new release of pagure: 2.15.
Here is its changelog:
* Tue May 16 2017 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.15-1
- Update to 2.15
- Improve logic in api/issue.py to reduce code duplication (Martin Basti)
- Fix the download button for attachment (Mark Reynolds)
- Fix our markdown processor for strikethrough
- Add a spinner indicating when we are retrieving the list of branches differing
- Make add_file_to_git use a lock as we do for our other git repositories
- Add the opportunity to enforce a PR-based workflow
- Store in the DB the API token used to flag a pull-request
- Allow people with ticket access to take and drop issues
- Display the users and groups tied to the repo in the API (Matt Prahl)
- Document our markdown in rest so it shows up in our documentation
- Fix comparing the minimal version of flask-wtf required
- Allow the td and th tags to have an align attribute to allow align in html
tables via markdown
- Avoid binaryornot 0.4.3 and chardet 3.0.0 for the time being
- Add group information API that shows group members (Matt Prahl)
- Ensure people with ticket metadata can edit the custom fields
- Add support to create private projects (Farhaan Bukhsh) - Off by default
- Link to the doc when the documentation is activated but has no content
- Enforce project wide flake8 compliance in the tests
- Enforce a linear alembic history in the tests
- Increase logging in pagure.lib.git
- Use custom logger on all module so we can configure finely the logging
- Multiple improvements to the documentation (René Genz)
- Add the ability to query projects by a namespace in the API (Matt Prahl)
- Add the /<repo>/git/branches API endpoint (Matt Prahl)
- Lock the git repo when removing elements from it
- Always remove the lockfile after using it, just check if it is still present
- Implement the `Give Repo` feature
- Allow project-less token to change the status of an issue in the API
- Make the watch feature more granular (Matt Prahl): you can now watch tickets,
commits, both, neither or go back to the default
- Bring the pagure.lib coverage to 100% in the tests (which results to bug fixes
in the code)
- Add locking at the project level using SQL rather than filelock at the git
This is currently happily running in stg and prod.
However, the prod instance has been hotfixed, we found a couple of small typos
(reported by email) as well as couple of issues in the new locking mechanism
(SQL-based) used, so there will be a 2.15.1 release (hopefully tomorrow) so that
we can dropped the hotfix.
2.15.1 will also contain what we believe to fix this annoying bug of PRs
indicated as merged when they weren't.
FMN has finally been updated in production with only a few minor bumps
along the way.
There is currently a performance problem with worker start-up as it
loads all user preferences into memory. This can take around an hour.
Future releases will improve this, but until then restarting workers
is rather costly (messages will be held in a queue until the workers
The changelog is included below:
* Emails now contain headers to indicate to clients that they are auto-
generated. This should stop them from auto-responding (#165).
* New rules for the Module Build Service (#174).
* Be fault-tolerant towards missing 'owner' field in copr msgs (commit
* Messages that can't be sent are now requeued (#169) (hotfixed).
* Update to the generic rule for packages to account for namespaces in
Today I made a Bodhi 2.7.0 beta:
Since there is a beta freeze starting tomorrow, I will not be deploying
Bodhi 2.7.0 to production for a while. However, this release has some
nice CLI improvements so I figured it would be good to get it out there
Jeremy wrote a sweet patch for Bodhi that enables it to run migrations
with BDR enabled without me having to ask admins to run SQL scripts. I
tested his patch with Bodhi 2.7.0 beta on staging today and it seems to
So if you are looking for a nice way to become "BDR compatible" while
still being able to use "alembic upgrade head", this is a nice way to
do it and it should be possible to apply similar code to our other
Jeremy and I also discussed about whether a similar patch should go
upstream. He suggested that upstream might just say that we should do
what we did, so maybe we could just send them a documentation "tip"
about this or something?