sochotni reported a new issue against the project: `releng` that you are following: `` We will need changes/additions to koji configuration to enable imports of Modules through content generator.
Expect future updates once since this is a bit off the top of my head.
1. Add content generator "module-build-service" 2. Add btype "module" 3. Configure whatever user is used by modulebuild service to have permission for the content generator APIs
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: ``
Configure whatever user is used by modulebuild service to have permission for the content generator APIs
The user is `mbs/mbs.fedoraproejct.org`.
For more info on what this is for, see the koji-devel thread(s): https://lists.fedoraproject.org/archives/search?mlist=koji-devel%40lists.fed... ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: `` FYI, this came up in the bodhi stakeholders meeting today: https://meetbot.fedoraproject.org/fedora-meeting-1/2017-05-23/bodhi_stakehol... ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` So a summary of assumed required calls: koji grant-cg-access mbs/mbs.fedoraproejct.org module-build-service --new koji call addBType module
That should be all that's needed from MBS side to work as content generator ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: `` Pull request submitted to Bodhi to look for markers provided by this content generator: https://github.com/fedora-infra/bodhi/pull/1543 ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: `` Blocking https://pagure.io/fm-orchestrator/pull-request/569 ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ausil added a new comment to an issue you are following: `` @rlaliber FYI this is a module config issue we have open. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: `` Adding the meeting label to see if there's anything that needs to be discussed here. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: `` FYI, this was discussed today: https://meetbot.fedoraproject.org/fedora-meeting-2/2017-06-12/releng.2017-06...
My read of the conversation is that this request is "approved", and that this ticket has to date been stuck on a lack of explicit policy for new content generators. releng took an action item to work on documenting that for the future. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: `` Linking some more information about what we want to do with this: https://fedoraproject.org/wiki/Infrastructure/Factory2/Focus/Bodhi ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` Also - offered to write to SOP to @katec, added @ausil to that mail waiting to hear back whether that would help ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` Added the SOP PR: https://pagure.io/releng/pull-request/6849
I am just referencing the koji documentation for content generator requirements since they seem aligned with what I'd expect rel-eng to require.
Alternatively koji documentation could be copied and tweaked but unless specific reservations are made it would be a waste of time. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
mohanboddu added a new comment to an issue you are following: `` As per the policy we have set
https://docs.pagure.org/releng/sop_adding_content_generator.html
Can you answer the following questions:
https://docs.pagure.org/releng/sop_adding_content_generator.html#questions-t...
And one other question I would like to know, where can we access the logs? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` I set myself up didn't I? :-)
The content generator is module build service - mbs.fedoraproject.org I am not 100% sure what the support agreement is around it but given it's running in the infra and there's playbooks I'd say it's in a decent state: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/mbs
Dev team is decently spread between continents at the moment and technically if mbs is down it's only affecting module builds, not rest of koji workflows.
The module build service itself doesn't actually do any builds - it just orchestrates koji so all logs are available in there. There's a possibility that we'd improve logging of the orchestration itself and make it part of the module import itself in the future.
New content type: We are adding "module" content type which is only a modulemd [1] yaml file. File size should be in the order of KBs. The binary content (module rpms) are already in koji and are just referenced in the build metadata.
As for the generic requirements from koji docs: * We record rpms installed on the module build system at the time of the module orchestration * We record version of the MBS itself * Since we call koji to do actual binary builds we don't use host software * For the same reason as above - we won't allow network during builds or use of pip or other content directly
Mocked content generator that would be imported into Koji looks like this: https://fedorapeople.org/~sochotni/mbs-content-generator-v2.json
In addition the mentioned modulemd file would be included as output and all rpms from the module referenced as belonging to this modulemd file.
[1] https://pagure.io/modulemd ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` I should have mentioned - basic information about state of orchestration is visible in mbs itself. For example: https://mbs.fedoraproject.org/module-build-service/1/module-builds/640 ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` I forgot that in addition to the buildroot info and host rpms etc on MBs, we also provide dist-git source URL/hash. This leads to modulemd yaml file which was used by MBS to orchestrate the builds - same way you have source url for rpm builds
Linking of the module rpms into this module build itself provides way to collect logs and track which specific branches/hashes module builds were done. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: `` A picture of MBS in action! http://threebean.org/img/mbs-with-mohan.png ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
mohanboddu added a new comment to an issue you are following: `` Some final quick questions : where does the json file thats imported into koji comes from? How is it generated? And where are the logs on its generation? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: ``
where does the json file thats imported into koji comes from?
It gets generated on the fly by the MBS in this file https://pagure.io/fm-orchestrator/blob/master/f/module_build_service/builder...
How is it generated?
From the modulemd that the user used to drive the module build as a whole (plus information from koji during the module build process).
And where are the logs on its generation?
The logs can be found in the journal on mbs-backend01.phx2.fedoraproject.org. See the 'fedmsg-hub' unit. (that's where the mbs backend runs).
Importantly, see also https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/mbs.h... ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
ralph added a new comment to an issue you are following: `` Any other questions we can answer @mohanboddu? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
The issue: `[koji] Add configuration for module build service content generator` of project: `releng` has been assigned to `mohanboddu` by ausil.
ausil added a new comment to an issue you are following: `` ``` 14:47 < dgilmore> I think the only change I would like to propose here is that mbs also uploads teh logs on teh json generation 14:47 < dgilmore> not sure how invasive that would be 14:47 < dgilmore> threebean: any idea? 14:48 * threebean reads up 14:50 < threebean> eh, should be possible. 14:50 < threebean> but will take a code change. 14:51 < dgilmore> threebean: cool 14:51 < threebean> will file an issue to track it. 14:51 < threebean> can we add it in later? 14:51 < dgilmore> I think it wou,ld be usefult to have as part of the auditability 14:51 < dgilmore> I think so 14:51 < threebean> (for now at least, the logs are accessible on mbs-backend01 in the journal) 14:51 < dgilmore> threebean: thats not public and open and transparent :D 14:52 < threebean> yup, yup. 14:52 < threebean> but it is auditable! 14:53 < dgilmore> #agreed that under the condition we add the logs creating the json to the upload we can add the content generator ```
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` MBS pull request with the addition of build logs to the content generator outputs section: https://pagure.io/fm-orchestrator/pull-request/601 ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` FYI the PR is now merged and @jkaluza is working on releasing new MBS version that will be importing the logs during CG imports ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` FYI the MBS is now deployed - cg imports are disabled until Koji configuration is changed. So ... shall we? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` Can we get an update/plan for this? Boltron is out so I assume folks should have some time now? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
mohanboddu added a new comment to an issue you are following: `` @sochotni Just for an example, can you link one of the logs here? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` By default the build logs are stored in /tmp and it doesn't seem to be overridden in the infra repo. I don't have permissions to ssh into mbs.fedoraproject.org but if you do you should find logs in /tmp/build-X.log ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
sochotni added a new comment to an issue you are following: `` Correction, I looked at the wrong default config variable - by default we don't generate per-build logs unless it's enabled by changing BUILD_LOGS_DIR from empty string. Which would happen once we enabled CG code but it's not there yet. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
mohanboddu added a new comment to an issue you are following: `` $ koji grant-cg-access mbs/mbs.fedoraproject.org module-build-service --new $ koji call addBType module None $
The content generator has been added. We still need the logs which requires a change in koji config. Please let us know when its done or what needs to be done to get the logs and then we can close the ticket.
Thanks.
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6799
rel-eng@lists.fedoraproject.org