Flagstate: an index server for containers
by Owen Taylor
The Docker registry protocol has very limited search and index
functionality - you can (if enabled on the server) list all repositories,
list tags for each repository, and then download the manifest for each tag.
This obviously is very inefficient if you want to ask questions like:
* "What containers are available with the org.flatpak.metadata" and what
are their human readable names?
* "Is there a newer version available of any of these 50 installed
applications" (especially without fingerprinting your machine by asking
about each application)
We're planning to distribute desktop applications as Flatpaks on
registry.fedoraproject.org, and to be able make GNOME Software and updates
work properly we need a way to efficiently query the metadata of the
registry.
To handle this, I wrote a new server "Flagstate" -
https://github.com/owtaylor/flagstate - It sits alongside a
docker/distribution registry instance. At initialization, it retrieves all
the metadata from the registry server and stores it in a database. It then
incrementally updates the database in response to webhook notifications
from the registry.
The index server supports queries. A hypothetical URL would be:
https://registry.fedoraproject/index/static?architecture=amd64&annotation...
Which returns a JSON dump of the metadata for matching containers. The
response is designed to be cacheable with consistent ordering and Etag
support. See
https://github.com/owtaylor/flagstate/blob/master/docs/protocol.md for
details.
In addition to the Flatpak use case, this also would be useful for the
Cockpit project that wants to be able to provide a nice interface for
browsing Cockpit plugins that are packaged as container images. Additional
future use cases that could be accomplished with pretty simple extension of
the Flagstate code include providing a backend for CLI searches - docker
search, podman search, ..., and providing a more comprehensive web frontend
than what is currently on registry.fedoraproject.org - allow seeing names,
descriptions, and so forth extracted from container labels.
I'd like to propose that we work toward a deployment of this on Fedora
infrastructure.
Mini-FAQ
=======
Can't this be done by just traversing the registry in a cron job and
writing a static index?
Without the incremental update process, it's going to be impossible to
provide a reasonable update frequency for a medium-large index. It would be
possible to static-generate a fixed set of queries instead of having
dynamic responses, but I think the current approach is more flexible - it
avoids having to hard-code, say, Flatpak specifics in the server
configuration.
Are there existing projects that could be used instead of writing our own?
Not that I'm aware of - 'reg', which is used for the current HTML
frontend for registry.fedoraproject.org, takes the above approach of
traversing the entire registry and writing a static index.
How is the data stored?
Data is stored in a a PostgreSQL database - this is purely a shadow of
the data as canonically stored in the registry, so has minimal backup
needs. PostgreSQL 9.4 or newer is needed for the jsonb functionality,
though this requirement could be removed if necessary.
Why the language/license choice?
Flagstate is writing in Go and license under the Apache License 2.0 to
match the docker/distribution codebase and maximize the chance of creating
a community.
Is it secure?
It's hard to say without some careful checking. The index only indexes
public information and care is taken when building SQL queries. DOS is
always a tricky thing to handle on any unauthenticated API - setting the
statement_timeout postgresql parameter might would make accidental DOS
harder.
6 years, 2 months
Reinstall of staging openshift
by Kevin Fenzi
Greetings.
Our staging openshift is still version 3.5, and still warns/yells about
nodes not having enough disk space (with 3.4 we allocated 20GB to them,
but in 3.5 and later they want at least 40G).
So, I'd like to take our staging openshift down, reprovision it with
more space for the nodes and also get it up to 3.7
Hopefully this won't take too long, and since we have no persistent data
everything should just reprovision via ansible.
I'd like to look at doing this tomorrow.
If you are using staging openshift and would prefer some other
time/date, please let me know.
Thanks,
kevin
6 years, 2 months
What causes newly registered account on fas to be spamcheck_manual?
by Zamir SUN
Hi Infra team,
This week two people reached me asking about logging into Pagure. Both
of them tried to register on FAS, but they are automatically set as
spamcheck_manual immediately after they registered and even cannot login
to sign CLA/FPCA. As far as I know, one of them is using gmail and
another is using a emain from their own domain - I think it's not caused
by main domains.
So I want to know if there are certain rules that we don't know, or
spamcheck rules which users from China might easily get into?
Thanks!
--
Ziqian SUN (Zamir)
GPG : 1D86 6D4A 49CE 4BBD 72CF FCF5 D856 6E11 F2A0 525E
Want to know more about Fedora?
Visit https://fedoraproject.org/wiki/
Ready to contribute? See https://whatcanidoforfedora.org/
6 years, 2 months
Active Fedora releases and the EOL
by Chenxiong Qi
Hi,
When I query active Fedora releases from PDC, I get Fedora 24 and 25,
both of them were already EOL. I'm confused why 24 and 25 are still
active releases? What is the definition of "active release", and how
to understand the relationship between active release and EOL?
Thanks!
--
Regards,
Chenxiong Qi
6 years, 2 months
[PATCH] git/hooks: only use first line as project description
by Todd Zullinger
If a description file contains multiple lines the script reads the whole
file and uses them in the X-Project: header. This leaves subsequent
headers in the message body.
Here's an example from this repository:
Subject: [ansible] Redefine master_repomd settings that don't have defaults.
Keywords: ansible
X-Project: ansible playbook/files/etc repository for fedora infrastructure.
Message-Id: <20180209174002.696C8FB72586(a)batcave01.phx2.fedoraproject.org>
Date: Fri, 9 Feb 2018 17:40:02 +0000 (UTC)
This is the public repository, do not commit sensitive
or confidential information here.
X-Git-Refname: refs/heads/master
X-Git-Oldrev: db817ee0f3bbd99b762a2d6b78d84212a0770935
X-Git-Newrev: d65b46a20984939465a6209205662f0030d80f54
commit d65b46a20984939465a6209205662f0030d80f54
A further potential feature could be added to include the additional lines
from the description file in the beginning of the message body, if they
are desired.
---
Hi all,
This is something I've seen for ages and thought should be fixed (either by
this patch and/or another which supports multi-line description files
properly).
I have an alternate version of this patch which side-steps the issue by adding
the X-Project header last, so if it's multiple lines it won't screw up other
headers. I think this is the proper fix though.
If multi-line description files are really something which are desired, then a
commit on top of this change could add support for reading the remaining lines
from the description file and add them to the message body.
roles/git/hooks/files/git.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/roles/git/hooks/files/git.py b/roles/git/hooks/files/git.py
index 72adff1f7..586fbffd2 100644
--- a/roles/git/hooks/files/git.py
+++ b/roles/git/hooks/files/git.py
@@ -202,7 +202,8 @@ def get_project_description():
description = os.path.join(git_dir, 'description')
if os.path.exists(description):
try:
- projectdesc = open(description).read().strip()
+ with open(description, 'r') as f:
+ projectdesc = f.readline().strip()
except:
pass
if projectdesc.startswith('Unnamed repository;'):
--
2.16.2
6 years, 2 months
[PATCH] remove cron job for atomic-cd repo
by Dusty Mabe
This is not used and we agreed to remove it. See [1] [2].
[1] https://pagure.io/releng/issue/7269
[2] https://pagure.io/atomic-wg/issue/349
Signed-off-by: Dusty Mabe <dusty(a)dustymabe.com>
---
roles/releng/files/atomic-cd | 4 ----
roles/releng/tasks/main.yml | 5 -----
2 files changed, 9 deletions(-)
delete mode 100644 roles/releng/files/atomic-cd
diff --git a/roles/releng/files/atomic-cd b/roles/releng/files/atomic-cd
deleted file mode 100644
index da90f7153..000000000
--- a/roles/releng/files/atomic-cd
+++ /dev/null
@@ -1,4 +0,0 @@
-# continous delivery of atomic host
-MAILTO=releng-cron(a)lists.fedoraproject.org
-*/15 * * * * root TMPDIR=`mktemp -d /tmp/atomic-cd.XXXXXX` && pushd $TMPDIR >& /dev/null && git clone -q https://pagure.io/releng.git && cd releng && LANG=en_US.UTF-8 /usr/local/bin/lock-wrapper atomic-cd ./scripts/build-test-ostree >& /dev/null && popd >& /dev/null && rm -rf $TMPDIR
-
diff --git a/roles/releng/tasks/main.yml b/roles/releng/tasks/main.yml
index 94ba9d814..6a8d81e79 100644
--- a/roles/releng/tasks/main.yml
+++ b/roles/releng/tasks/main.yml
@@ -167,11 +167,6 @@
copy: src="docker-updates" dest=/etc/cron.d/docker-updates
when: inventory_hostname.startswith('compose-x86-01')
-# put cron job in for continuously deliverying atomic host
-- name: atomic-cd compose cron
- copy: src="atomic-cd" dest=/etc/cron.d/atomic-cd
- when: inventory_hostname.startswith('compose-x86-01')
-
# put cron job in for purging nightly amis
- name: purge nightly amis cron
copy: src="purge-amis" dest=/etc/cron.d/purge-amis
--
2.14.3
6 years, 2 months
[release] simple-koji-ci: 0.3.0
by Pierre-Yves Chibon
Good Morning Everyone,
I just cut a new release of simple-koji-ci: 0.3.0.
Here is its changelog:
* Thu Feb 22 2018 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.3.0-1
- Ensure the license info are consistently GPLv2+
- Flag the PR once the build has been submitted
- Support building in the target branch if known, otherwise default to rawhide
- Mention the commit tested in the flag added
- Trigger a new build on PR updates
It's running in prod already, I'm monitoring it to see if it behaves as expected
:)
Happy pull-requesting!
Pierre
6 years, 2 months