freeze break request: filtered file lists
by Adam Williamson
This commit should give us new 'filterlist' files alongside the
'fullfilelist' and 'fullfiletimelist' files in the /fedora , /archive
and /alt folders on the mirrors. The contents of this file are the same
as 'fullfilelist', but with all directories, packages (.rpm or .drpm
files) and device tree boot files (.dtb files) removed. This gives a
massively smaller list which, right now, will be useful for fedfind (it
can parse these lists instead of rsync scraping) and may possibly be
useful for other things in future, I guess.
It would be nice to have this now as this all came out of the work
around improving generation of the mediawriter 'available images' JSON
file: we want to use fedfind to generate that, but the rsync scraping is
pretty heavy for something that'll run quite frequently. This should
improve things quite a bit (especially as I've written fedfind to cache
the files and only re-download them if the Last-Modified header
changes). I have the fedfind changes all written and tested (I tested
against the fullfilelist files).
From 3f61df2f2879c23a7e44271527facce99bc92286 Mon Sep 17 00:00:00 2001
From: Adam Williamson <awilliam(a)redhat.com>
Date: Fri, 18 Nov 2016 16:34:38 -0800
Subject: [PATCH] Generate filtered file lists for fedfind to use
This adds `filterlist` files alongside the `fullfilelist` and
`fullfiletimelist` files. These are much, much shorter lists
which skip the entries for packages, ARM device tree boot files
and directories. They are intended for consumption by fedfind,
so it can stop using rync scraping to discover the image files
it looks for. To enable this, we update to a newer version of
`create-filelist` from upstream `quick-fedora-mirror` and make
`update-fullfiletimelist` create the filterlist files as well.
We also delete a couple of old copies of `create-filelist`;
nirik made the two roles that use it share a common copy a few
months back, but missed deleting the copy each role had in its
`files` directory.
---
files/scripts/create-filelist | 10 ++++++++-
files/scripts/update-fullfiletimelist | 19 ++++++++++++++--
roles/bodhi2/backend/files/create-filelist | 36
------------------------------
roles/releng/files/create-filelist | 36
------------------------------
4 files changed, 26 insertions(+), 75 deletions(-)
delete mode 100644 roles/bodhi2/backend/files/create-filelist
delete mode 100644 roles/releng/files/create-filelist
diff --git a/files/scripts/create-filelist b/files/scripts/create-filelist
index eeba9d0..8fc3367 100755
--- a/files/scripts/create-filelist
+++ b/files/scripts/create-filelist
@@ -57,7 +57,9 @@ def recursedir(path='.', skip=[], alwaysskip=['.~tmp~']):
def parseopts():
null = open(os.devnull, 'w')
p = argparse.ArgumentParser(
- description='Generate a list of files and times, suitable for
consumption by quick-fedora-mirror.')
+ description='Generate a list of files and times, suitable for
consumption by quick-fedora-mirror, '
+ 'and a much smaller list with packages, Device Tree
boot files, HTML files and '
+ 'directories filtered out, for consumption by
fedfind.')
p.add_argument('-c', '--checksum', action='store_true',
help='Include checksums of all repomd.xml files in
the file list.')
p.add_argument('-C', '--checksum-file', action='append',
dest='checksum_files',
@@ -73,6 +75,8 @@ def parseopts():
help='Filename of the file list with times
(default: stdout).')
p.add_argument('-f', '--filelist', type=argparse.FileType('w'),
default=null,
help='Filename of the file list without times
(default: no plain file list is generated).')
+ p.add_argument('-F', '--filterlist', type=argparse.FileType('w'),
default=null,
+ help='Filename of the filtered file list for fedfind
(default: not generated).')
opts = p.parse_args()
@@ -107,6 +111,10 @@ def main():
for entry in recursedir(skip=opts.skip_files):
# opts.filelist.write(entry.path + '\n')
print(entry.path, file=opts.filelist)
+ # write to filtered list if appropriate
+ skips = ('.rpm', '.drpm', '.dtb', '.html')
+ if not any(entry.path.endswith(skip) for skip in skips) and not
(entry.is_dir()):
+ print(entry.path, file=opts.filterlist)
if entry.name in opts.checksum_files:
checksums[entry.path[2:]] = True
info = entry.stat(follow_symlinks=False)
diff --git a/files/scripts/update-fullfiletimelist
b/files/scripts/update-fullfiletimelist
index 016ca8e..e70fadc 100755
--- a/files/scripts/update-fullfiletimelist
+++ b/files/scripts/update-fullfiletimelist
@@ -25,6 +25,7 @@ CREATE=/usr/local/bin/create-filelist
# context.
FILELIST=fullfilelist
TIMELIST='fullfiletimelist-$mod'
+FILTERLIST=filterlist
usage () {
echo
@@ -107,12 +108,14 @@ cd $tmpd
for mod in $MODS; do
currentfl=$TOPD/$mod/${FILELIST/'$mod'/$mod}
currenttl=$TOPD/$mod/${TIMELIST/'$mod'/$mod}
+ currentsl=$TOPD/$mod/${FILTERLIST/'$mod'/$mod}
flname=$(basename $currentfl)
tlname=$(basename $currenttl)
+ slname=$(basename $currentsl)
- $CREATE -c -s -d $TOPD/$mod -f $flname -t $tlname
+ $CREATE -c -s -d $TOPD/$mod -f $flname -t $tlname -F $slname
- # If a file list exsts and doesn't differ from what we just
generated,
+ # If a file list exists and doesn't differ from what we just
generated,
# delete the latter.
if [[ -f $currentfl ]] && diff -q $currentfl $flname >
/dev/null; then
rm -f $flname
@@ -120,6 +123,9 @@ cd $tmpd
if [[ -f $currenttl ]] && diff -q $currenttl $tlname >
/dev/null; then
rm -f $tlname
fi
+ if [[ -f $currentsl ]] && diff -q $currentsl $slname >
/dev/null; then
+ rm -f $slname
+ fi
done
# Now we have the new file lists but in a temporary directory which
@@ -128,10 +134,13 @@ cd $tmpd
for mod in $MODS; do
currentfl=$TOPD/$mod/${FILELIST/'$mod'/$mod}
currenttl=$TOPD/$mod/${TIMELIST/'$mod'/$mod}
+ currentsl=$TOPD/$mod/${FILTERLIST/'$mod'/$mod}
flname=$(basename $currentfl)
fldir=$(dirname $currentfl)
tlname=$(basename $currenttl)
tldir=$(dirname $currenttl)
+ slname=$(basename $currentsl)
+ sldir=$(dirname $currentsl)
if [[ -f $flname ]]; then
tmpf=$(mktemp -p $fldir $flname.XXXXXXXXXX)
@@ -145,6 +154,12 @@ cd $tmpd
chmod 644 $tmpf
mv $tmpf $currenttl
fi
+ if [[ -f $slname ]]; then
+ tmpf=$(mktemp -p $sldir $slname.XXXXXXXXXX)
+ cp -p $slname $tmpf
+ chmod 644 $tmpf
+ mv $tmpf $currentsl
+ fi
done
) 9>$LOCKFILE
diff --git a/roles/bodhi2/backend/files/create-filelist
b/roles/bodhi2/backend/files/create-filelist
deleted file mode 100644
index d95000e..0000000
--- a/roles/bodhi2/backend/files/create-filelist
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/python
-
-# A simple script to generate a file list in a format easily consumable
by a
-# shell script.
-
-# Originally written by Jason Tibbitts <tibbs(a)math.uh.edu> in 2016.
-# Donated to the public domain. If you require a statement of license,
please
-# consider this work to be licensed as "CC0 Universal", any version you
choose.
-
-
-from scandir import scandir
-
-
-def get_ftype(entry):
- """Return a simple indicator of the file type."""
- if entry.is_symlink():
- return 'l'
- if entry.is_dir():
- return 'd'
- return 'f'
-
-
-def recursedir(path):
- """Just like scandir, but recursively."""
- for entry in scandir(path):
- if entry.is_dir(follow_symlinks=False):
- for rentry in recursedir(entry.path):
- yield rentry
- yield entry
-
-
-for entry in recursedir('.'):
- info = entry.stat(follow_symlinks=False)
- modtime = max(info.st_mtime, info.st_ctime)
- ftype = get_ftype(entry)
- print('{} {} {}'.format(modtime, ftype, entry.path[2:]))
diff --git a/roles/releng/files/create-filelist
b/roles/releng/files/create-filelist
deleted file mode 100644
index d95000e..0000000
--- a/roles/releng/files/create-filelist
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/python
-
-# A simple script to generate a file list in a format easily consumable
by a
-# shell script.
-
-# Originally written by Jason Tibbitts <tibbs(a)math.uh.edu> in 2016.
-# Donated to the public domain. If you require a statement of license,
please
-# consider this work to be licensed as "CC0 Universal", any version you
choose.
-
-
-from scandir import scandir
-
-
-def get_ftype(entry):
- """Return a simple indicator of the file type."""
- if entry.is_symlink():
- return 'l'
- if entry.is_dir():
- return 'd'
- return 'f'
-
-
-def recursedir(path):
- """Just like scandir, but recursively."""
- for entry in scandir(path):
- if entry.is_dir(follow_symlinks=False):
- for rentry in recursedir(entry.path):
- yield rentry
- yield entry
-
-
-for entry in recursedir('.'):
- info = entry.stat(follow_symlinks=False)
- modtime = max(info.st_mtime, info.st_ctime)
- ftype = get_ftype(entry)
- print('{} {} {}'.format(modtime, ftype, entry.path[2:]))
--
2.10.2
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
7 years, 4 months
[PATCH] update the pre release policy for f26
by Dennis Gilmore
From: Dennis Gilmore <ausil(a)fedoraproject.org>
Signed-off-by: Dennis Gilmore <ausil(a)fedoraproject.org>
---
roles/bodhi2/base/templates/production.ini.j2 | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/roles/bodhi2/base/templates/production.ini.j2 b/roles/bodhi2/base/templates/production.ini.j2
index a172e25..7864938 100644
--- a/roles/bodhi2/base/templates/production.ini.j2
+++ b/roles/bodhi2/base/templates/production.ini.j2
@@ -10,16 +10,16 @@ filter-with = proxy-prefix
# Release status
# pre-beta enforces the 'Pre Beta' policy defined here:
# https://fedoraproject.org/wiki/Updates_Policy
-f25.status = pre_beta
+f26.status = pre_beta
-f25.post_beta.mandatory_days_in_testing = 7
-f25.post_beta.critpath.num_admin_approvals = 0
-f25.post_beta.critpath.min_karma = 2
-f25.post_beta.critpath.stable_after_days_without_negative_karma = 14
+f26.post_beta.mandatory_days_in_testing = 7
+f26.post_beta.critpath.num_admin_approvals = 0
+f26.post_beta.critpath.min_karma = 2
+f26.post_beta.critpath.stable_after_days_without_negative_karma = 14
-f25.pre_beta.mandatory_days_in_testing = 3
-f25.pre_beta.critpath.num_admin_approvals = 0
-f25.pre_beta.critpath.min_karma = 1
+f26.pre_beta.mandatory_days_in_testing = 3
+f26.pre_beta.critpath.num_admin_approvals = 0
+f26.pre_beta.critpath.min_karma = 1
##
## Atomic OSTree support
--
1.8.3.1
7 years, 4 months
freeze break request: Undo fedfind/releases.json stuff
by Ricky Elrod
It is a static part of the websites build for now.
commit 61902d81d3d2ece72c9fffcef38fb7b28b9b9055
Author: Ricky Elrod <codeblock(a)fedoraproject.org>
Date: Fri Nov 18 18:20:15 2016 +0000
undo these for now
Signed-off-by: Ricky Elrod <codeblock(a)fedoraproject.org>
diff --git a/roles/fedora-web/build/files/syncStatic.stg.sh
b/roles/fedora-web/build/files/syncStatic.stg.sh
index 0ad6072..ac9313b 100644
--- a/roles/fedora-web/build/files/syncStatic.stg.sh
+++ b/roles/fedora-web/build/files/syncStatic.stg.sh
@@ -54,13 +54,6 @@ build arm.fedoraproject.org
build getfedora.org
build alt.fedoraproject.org
-# Temporary: Build releases.json here and copy it over.
-# This will eventually move to a daily cron.
-cd getfedora.org
-make json-releases
-rsync -qa --delete-after --delay-updates out/releases.json
/srv/web/getfedora.org/
-cd ..
-
pushd mirrors.fedoraproject.org > /dev/null
rsync -qa --delete-after --delay-updates .
/srv/web/mirrors.fedoraproject.org/
popd > /dev/null
diff --git a/roles/fedora-web/build/tasks/main.yml
b/roles/fedora-web/build/tasks/main.yml
index 903f341..4fc92e6 100644
--- a/roles/fedora-web/build/tasks/main.yml
+++ b/roles/fedora-web/build/tasks/main.yml
@@ -2,7 +2,6 @@
yum: pkg={{ item }} state=present
with_items:
- babel
- - fedfind
- git
- python-feedparser
- python-genshi
7 years, 4 months
[release] pagure: 2.9
by Pierre-Yves Chibon
Good Morning everyone,
I just cut a new release of pagure: 2.9.
This is a fairly large release, see the changelog:
* Fri Nov 18 2016 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.9-1
- Update to 2.9
- Fix redirecting after updating an issue on a project with namespace (Vivek
Anand)
- Remove take button from Closed Issues (Rahul Bajaj)
- Show the open date/time on issues as we do for PR (Rahul Bajaj)
- When rendering markdown file use the same code path as when rendering comments
- Add documentation for using Markdown in Pagure (Justing W. Flory)
- Fix the behavior of the Cancel button on PR page (Rahul Bajaj)
- Be tolerant to markdown processing error
- Let the notifications render correctly when added by the SSE server
- Fix the URL for pull request on the list of branches of a fork (Rahul Bajaj)
- Adjust the markdown processor to have 1 regex for all cross-project links
- Remove unsued variables (Farhaan Bukhsh)
- Hide the title of private tickets when linking to them in markdown
- Show user activity in pagure on the user's page
- Add the possibility to subscribe to issues
- Do not cache the session in pagure-ci (as we did for pagure-webhook)
- Fix rendering raw file when the sha1 provided is one of a blob
- Include project's custom fields in the JSON representation of a project
- Include the issue's custom fields values in the JSON representation of an
issue
- Include the list of close_status and the milestones in the JSON of a project
- Improve documentation related to unit-tests (Rahul Bajaj)
- Use `project.fullname` in X-Pagure-Project header (Adam Williamson)
- Figure a way to properly support WTF_CSRF_TIME_LIMIT on older version of
flask-wtf
- When updating an issue, if the form does not validate, say so to the user
- Fix the total number of pages when there are no PR/issues/repo (vibhcool)
- Fix forking a repo with a namespace
- Include the namespace in the message returned in pagure.lib.new_project
- Move the metadata-ery area in PR to under the comments tab (Ryan Lerch)
- Update setup instructions in the README.rst (alunux)
- Support namespaced projects when reading json data (clime)
- When uploading a file in a new issue, propagate the namespace info
- Ensure our avatar works with non-ascii email addresses
- Downgrade to emoji 1.3.1, we loose some of the newer emojis we get back
preview and reasonable size (Clément Verna)
- Fix sending notifications email containing non-ascii characters
- Fix using the proper URL in email notifications (Adam Williamson)
- Move the Clear and Cancel buttons to the right hand side of the comment box
- Fix spelling in the PR page (Vibhor Verma)
- Support loading custom fields from JSON when loading issues from git (Vivek
Anand)
- Fix handling namespaced project in the SSE server (Adam Williamson)
- Add a pylintrc configuration file to help with code standards (Adam
Williamson)
- Add go-import meta tag allowing go projects to be hosted on pagure (Patrick
Uiterwijk)
- Fix index overflow when opening remote pull-request (Mark Reynolds)
- Add SSE support for custom fields
- Add a git blame view
- Allow emptying a file when doing online editing
- Only let admins edit the dependency tree of issues
- Fix some spelling errors (Adam Williamson)
- Add SHA256 signature to webhooks notifications (Patrick Uiterwijk)
- Multiple fixes in the API documentation and output
To highlight a few:
- We now present the user's activity on their page using the now well-know
calendar heatmap
- X-pagure-project in the email headers now contains the full name of the
project (ie fork/username/projectname) instead of just the project name, so
beware if you are relying on this field for your email filtering
- We should no longer have the issue with disappearing comments \ó/
Thanks to everyone who contributed on this release, there was quite a few people
involved :)
I will not be online next week, so this release is currently running only in stg
and will remain there at least until November 28th (at which point we may very
well be ready for a 2.9.1).
As usual testers are welcomed.
Happy hacking!
Pierre
7 years, 4 months
[freeze break request] add fedfind for websites build, enable
json-releases build in stg
by Ricky Elrod
I'd like to:
- Add the fedfind dep for the websites build on sundries
- Enable the json-releases build task in syncStatic for stg
- Run the sundries playbook
Commits for the first two tasks follow:
commit d892719a2395cd34d1e3c7e979899461e74e941b
Author: Ricky Elrod <codeblock(a)fedoraproject.org>
Date: Thu Nov 10 19:37:58 2016 +0000
add new fedfind dep for websites - needed for release json
Signed-off-by: Ricky Elrod <codeblock(a)fedoraproject.org>
diff --git a/roles/fedora-web/build/tasks/main.yml
b/roles/fedora-web/build/tasks/main.yml
index 4fc92e6..903f341 100644
--- a/roles/fedora-web/build/tasks/main.yml
+++ b/roles/fedora-web/build/tasks/main.yml
@@ -2,6 +2,7 @@
yum: pkg={{ item }} state=present
with_items:
- babel
+ - fedfind
- git
- python-feedparser
- python-genshi
commit 053c0d75d9a4ca6a01a1ab41185a5fb6fc7e481b
Author: Ricky Elrod <codeblock(a)fedoraproject.org>
Date: Thu Nov 10 19:48:52 2016 +0000
build releases.json in stg
Signed-off-by: Ricky Elrod <codeblock(a)fedoraproject.org>
diff --git a/roles/fedora-web/build/files/syncStatic.stg.sh
b/roles/fedora-web/build/files/syncStatic.stg.sh
index ef2446d..c62036c 100644
--- a/roles/fedora-web/build/files/syncStatic.stg.sh
+++ b/roles/fedora-web/build/files/syncStatic.stg.sh
@@ -53,6 +53,13 @@ build labs.fedoraproject.org
build arm.fedoraproject.org
build getfedora.org
+# Temporary: Build releases.json here and copy it over.
+# This will eventually move to a daily cron.
+cd getfedora.org
+make json-releases
+rsync -qa --delete-after --delay-updates out/releases.json
/srv/web/getfedora.org/
+cd ..
+
/usr/bin/git clean -q -fdx || exit 1
/usr/bin/git reset -q --hard || exit 1
/usr/bin/git checkout -q alt-rewrite || exit 1
7 years, 4 months
Plan for tomorrow's Fedora Infrastructure meeting - 2016-11-17
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2016-11-17 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
kevin
--
= Introduction =
This shared document is for the next fedora infrastructure meeting.
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2016-11-17)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
#topic New folks introductions
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info Fedora 25 Final FREEZE still in effect - everyone
#info more pagure migrations happening - everyone
#info Next week is thanksgiving holiday in US. 24th/25th holidays - everyone
#info NO meeting next week due to holiday - everyone
#info Patrick and Kevin will be out at the PHX2 datacenter 2016-12-05 to 09 - kevin and patrick
#info Holidays in december coming up as well.
#info
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus or decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic on-boarding new folks improvements - kevin / stickster
#topic meeetbot status - kevin/abompard
#topic plans for december/january - kevin
= Apprentice office hours =
#topic Apprentice Open office hours
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for improvement,
etc. Whoever would like to do this, just add the info in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about:
= Meeting end stuff =
#topic Open Floor
#endmeeting
7 years, 4 months
Memory on koschei-backend01.phx2
by Mikolaj Izdebski
Michael Simacek pointed out that koschei-backend01.phx2 has 20 GB of
physical memory, while Ansible inventory vars specify only 4 GB.
Does anyone know where this difference comes from? Did anyone increase
VM settings without committing them to Ansible git repo? If so, what was
the reason?
--
Mikolaj Izdebski
Software Engineer, Red Hat
IRC: mizdebsk
7 years, 4 months
Meeting Agenda Item: Introduction Bradley
by Bradley Sharpe
IRC: threenodedev
Not sure what skills to list here. I’ve been working on computer systems (including Linux) for about 20+ years. I have built physical servers, networks and systems for this time. I have had the pleasure of setting up OpenStack on physical hardware, configuring AWS via Ansible and (unfortunately) doing a lot of Windows system administration. So… I’ve had at least a smattering of the following technologies: Nagios, Ansible, AWS, OpenStack, Linux(various distros), Ruby, Python, Javascript, networking (cisco mostly).
What I’d like to learn is to get better at Python and system infrastructure design (and monitoring).
Mostly I’m excited to contribute to the Fedora team.
Cheers,
Bradley
7 years, 4 months