On Mon, Nov 27, 2017 at 04:20:23PM -0500, Dusty Mabe wrote:

For Fedora Atomic we release content every two weeks. A lot of ducks have to be in a row in order to do this and often times behind the scenes work is time sensitive. I really need access to releng/infra so that I can be more efficient in doing this. I'd like to be able to:

To whatever degree me vouching for Dusty counts, I vouch for him. :)

For Fedora Atomic we release content every two weeks. A lot of ducks have to be in a row in order to do this and often times behind the scenes work is time sensitive. I really need access to releng/infra so that I can be more efficient in doing this. I'd like to be able to:

Okay, here we go. I'll give an overview of what kind of access you're asking for here.

• restart failed disk image build in koji

This means admin in koji or root on compose boxes (Depending how you want to kick it off). Allows you to either do basically everything in koji or on compose boxes.

• re-kick off daily runs if they fail for some reason (make a .1 of a daily run)

This needs root on compose boxes. Allows you to do basically everything in all of releng.

• merge pull requests for other team members after I review them

Releng group on Pagure. For this I'd be +1.

• perform two week atomic releases

Root on bodhi-backend01.

- I realize this one is hard because of signing keys but seriously
  would be nice to be able to do this one myself 
- If signing keys are a problem then we should just automate the
  signing of the checksums for the atomic runs so that the
  checksums are already signed by the time we do the release.

Access to all signing keys, allowing you to sign anything you'd like. Do note that autosigning this was on the task list, but because the process changed so much the last few months I was told to "hang on".

• push access to infra repo

Sysadmin or sysadmin-veteran group (assuming you mean the ansible repo). I'd be fine with this.

- I often get patches reviewed on the list and then no one merges
  the patches for me because they assume I have access and that I'm
  going to do it myself. Then I bug someone to merge it for me,
  then they forget to run the ansible playbook that is needed to
  apply the change. I probably have to ping/wait 5+ times for every
  change I need.

Running playbooks: depends on what playbook, we can give specific access to run a particular playbook (rbac-playbook). The playbooks you likely want (bodhi* and releng) are somewhat tricky, but since it's all from ansible anyway, fine.

• manipulate ostree repos
  • occasionally we have to reset refs, sync content from one repo to another or otherwise manipulate the repo. Waiting on someone to do this for me every time is quite painful. I do want people to review changes that I propose but waiting more than 2 days is not something I want to have to do.

Root on bodhi-backend01. Do note that right now the reason you need to wait so long is because I am the only one handling these requests. I already asked three months ago for SOP's on how the ostree stuff works, and learning materials for releng, but have so far not seen those (https://pagure.io/releng/issue/6984#comment-460533 and some meetings). I might have missed them, but if you have those resources, other releng folks can help with more things, which would massively speed things up.

• investigate failures by having access to infrastructure and logs
  • fedimg fails often and I have to wait someone to investigate every time

sysadmin-fedimg, including access to the AWS keys on it. Note that Sayan promised me the new one is more reliable and should be deployed this week.

- koji builders can sometimes have random failures. Being able to
  investigate myself would be useful.

sysadmin-releng. Wide access to all builders.

I have read over the following docs:

• https://docs.pagure.org/releng/index.html#join-releng
• https://docs.pagure.org/releng/contributing.html
• https://fedoraproject.org/wiki/Infrastructure/GettingStarted
• https://fedoraproject.org/wiki/Infrastructure_Apprentice

I know of the SOPs and have contributed to at least one of them:

• https://docs.pagure.org/releng/sop.html
• https://infrastructure.fedoraproject.org/infra/docs/

I have helped debug a lot of issues with composes and general infra inconsistencies over the past year. I have spent much time in the releng /admin channels and have even helped others who are new to the channel. I have attended releng/infra meetings somewhat regularly over the past few months.

I am not asking for permissions so that I can willy nilly do what I please. I'd try to get proposed changes reviewed and announce any actions I performed on list. I just want to add a little more sanity to my day-to-day.

The permissions you would need to do the things you requests *do* allow you to do "willy nilly" what you want. E.g. access to modify ostree repos manually require full write access to /mnt/koji and /pub. I have tried to point out the specific pieces of access I would be fine with, but personally I am really not happy with the large set of permissions requested, and a set of them I would really be uncomfortable with. One of the reasons for that feeling from me is the number of things we have needed to do ad-hoc to ostree repositories, like resetting refs etc, because the version numbers got out of step due to ostree bugs. If you have ideas which other permissions would grant the things you requested, I'd be open to them, but this is what I can see would satisfy what you asked for.

I have an outstanding ticket open for adding myself to the releng group:

• https://pagure.io/releng/issue/6994

Not sure if there is an equivalent process for adding people to Infra.

There is none, the normal process is that when you do a lot in a particular piece of infra and we think you have a proven track record and give a good feeling with people, we will ask you if you'd like the permissions.

Do note that we also discussed the releng request over Flock, where I suggested to hold off for a bit and get stuff documented and see if that improves things.

Thanks! Dusty

Regards, Patrick

On 29 November 2017 at 09:49, Dusty Mabe dusty@dustymabe.com wrote:

Thanks patrick for the response. I'm going to cut/snip individual pieces to respond to some more specific questions/comments and then try to summarize at the end.

...

I already asked three months ago for SOP's on how the ostree stuff works, and learning materials for releng, but have so far not seen those (https://pagure.io/releng/issue/6984#comment-460533 and some meetings). I might have missed them, but if you have those resources, other releng folks can help with more things, which would massively speed things up.

I wholeheartedly agree and I plan on getting those SOPs written. I'd actually like to do a workshop (maybe at the next hackfest) for everyone.

...

The permissions you would need to do the things you requests *do* allow you to do "willy nilly" what you want.

What I was trying to say is that I'm not planning to abuse the power. If I abuse power then revoke access. I'm really not the type of person that makes changes without asking people. I would probably make a lot of noise before doing something without waiting for input from others and I would only do it if I thought it was critical.

...

E.g. access to modify ostree repos manually require full write access to /mnt/koji and /pub. I have tried to point out the specific pieces of access I would be fine with, but personally I am really not happy with the large set of permissions requested, and a set of them I would really be uncomfortable with. One of the reasons for that feeling from me is the number of things we have needed to do ad-hoc to ostree repositories, like resetting refs etc, because the version numbers got out of step due to ostree bugs.

I agree that there have been some issues with ostree, but I think there have been a fair number of issues from the releng/infra side that have caused ostrees to get out of sync as well. Fortunately when we switch to smart versioning this will matter less and we can deal with ostree bugs or releng/ infra transient issues with more grace.

So in summary from your response I gather I am requesting:

• Releng group on Pagure
• Running specific playbooks
• admin in koji
• root on compose boxes
• root on bodhi-backend01
• sysadmin-fedimg
• sysadmin-releng
• Access to all signing keys

I'll actually remove two items from that list:

• Access to all signing keys
• admin in koji

I really don't want access to signing keys; I understand why that needs to be locked down as much as possible. Also, I don't really think I need admin in koji assuming root on compose box will allow me to kick off things I need to.

For the rest I really don't see why it's not OK. Basically it comes down to two issues I can see (let me know if there are others):

• trust
• competence

If I can't be trusted by now then that is unsettling and discouraging to me. As for competence, I guess that depends on who you ask, but I'd like to think I would pass that bar as well.

In security, no one is trusted. I am not trusted, Kevin is not trusted, Patrick is not trusted. Dennis and Mohan are not trusted. This is because we are foolable and fallible. Even worse our accounts are not ourselves and could be used to look like us but not be us. This gives us each a potential reliability of 0.8 and a combined reliability of 0.1 (aka 90% of the time our root powers combined will cause things to be worse.)

Second our infrastructure is very much an organically grown ball of kerosene soaked yarn. You get root on box A and magically you can affect something on box C because someone decided that <<fill in funny named service of the week>> needed to be used for our brand new << fill in feature that we try in Fedora N and shoved aside in N+2 >> and both systems use it. And we can't remove that feature or tool because oh someone built it into koji or fas or some other critical tool to cut down development time because it needed to be 'working' now. [Most of our services are 0.95 reliable but combined in their myriad ways it is probably a combined total of 0.4 -> 0.6. In another words 40-60% of the time something is not working right for someone]

The two systems you are wanting root access to are some of the most interconnected in the kerosene level. Everyone who does have root on them has accidentally mangled all of Fedora in some way for several hours or days because we were trying to fix something and broke something else. [I think Dennis made archives completely unreadable twice and I moved all of them to ... using a script which had been looked at by two other sysadmins before I ran it. Those were the easy ones to undo.. there have been others which took going through backups to fix.]

It would be nice if those boxes weren't the worst but every time we redesign things to be saner, we end up with a << must have this new service installed or Fedora will FAIL >> that can only meet the release deadline by being put on them. We try to clean up before the next release but oh look another << if Fedora doesn't have this feature it will FAIL >> comes up.

And when things fail, those of us in root are the ones who collectively get blamed for it. Why did you let smooge have rights to the system? Why did you not check his actions before he did them. How did you not catch a ... in that line of code? It may all sound like reasonable questions but to the people who are dealing with the problem it gets picked up as "How did you let this moron ever do this?" And because things fail so much.. it eventually comes across as "You people are completely inept".

All of this: * High chance that someone as root is going to make a mistake and the more people who have it more likely. * High chance that some systems will catch fire through all the rest of infrastructure * Perceived blame

makes that adding anyone else a very difficult process. We can be prickly because of 3 and should mitigate it, but the real problem are the first two in the list.

Dusty _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org

On 29 November 2017 at 12:29, Dusty Mabe dusty@dustymabe.com wrote:

On 11/29/2017 11:34 AM, Stephen John Smoogen wrote:

...

In security, no one is trusted. I am not trusted, Kevin is not trusted, Patrick is not trusted. Dennis and Mohan are not trusted. This is because we are foolable and fallible. Even worse our accounts are not ourselves and could be used to look like us but not be us. This gives us each a potential reliability of 0.8 and a combined reliability of 0.1 (aka 90% of the time our root powers combined will cause things to be worse.)

Second our infrastructure is very much an organically grown ball of kerosene soaked yarn. You get root on box A and magically you can affect something on box C because someone decided that <<fill in funny named service of the week>> needed to be used for our brand new << fill in feature that we try in Fedora N and shoved aside in N+2 >> and both systems use it. And we can't remove that feature or tool because oh someone built it into koji or fas or some other critical tool to cut down development time because it needed to be 'working' now. [Most of our services are 0.95 reliable but combined in their myriad ways it is probably a combined total of 0.4 -> 0.6. In another words 40-60% of the time something is not working right for someone]

The two systems you are wanting root access to are some of the most interconnected in the kerosene level. Everyone who does have root on them has accidentally mangled all of Fedora in some way for several hours or days because we were trying to fix something and broke something else.

So let's develop a set of sudo rules so I can only run certain commands?

That sounds good but isn't what you asked for. This may sound like a "but it should be clear that isn't what I wanted" but it isn't. Most requests for root from various projects has been for complete root and any amount of sudo rules gets them cranky. Which makes us reflex to assuming that a person is asking for what they know they require.

Writing limited sudo commands is also hard because it is very easy to leave something open which goes BOOM.. we have done it to ourselves a lot and found that there is no shortcut to writing a command. It usually means making it very very limited and then opening it up slower and slower until it does what the person wants while we have a limited amount of risk. We can do this but expect that there will be a long period of "it doesn't do X", "ok try now" "I have lost Y" "ok why do you need Y?" "well it needs to get W working". And then a bunch "oh huh, I never knew it used Z"

That requires a lot of documentation on processes all around.

...

And when things fail, those of us in root are the ones who collectively get blamed for it.

Possible blame is definitely something I worry about when requesting access like this. I do propose that I work with some bumpers (i.e. only doing things after I get someone to review it, announcing things i'm going to do). Basically I would be in FBR mode all the time, except after I get reviewed I can actually do the work myself.

I am not clear on FBR means in this context. I am used to the context Full Blown Root which is not what you are meaning.

Why did you let smooge have rights to

...

the system? Why did you not check his actions before he did them. How did you not catch a ... in that line of code? It may all sound like reasonable questions but to the people who are dealing with the problem it gets picked up as "How did you let this moron ever do this?" And because things fail so much.. it eventually comes across as "You people are completely inept".

All of this:

• High chance that someone as root is going to make a mistake and the

more people who have it more likely.

• High chance that some systems will catch fire through all the rest

of infrastructure

• Perceived blame

makes that adding anyone else a very difficult process. We can be prickly because of 3 and should mitigate it, but the real problem are the first two in the list.

yes, hopefully the infra becomes less kerosene over time, but I'd like to be part of the solution there rather than just someone waiting on it to

I think infrastructure will always be keroseney by the nature of trying to make radical changes every 6 months. The best we can hope for at times is that a contained fire will take out enough old stuff we can build the new refinery on top of it.

happen. I'm mostly here to offer help and also deliver/release atomic host every two weeks. Often times this means I find issues before anyone else does (because I'm releasing tomorrow and not 6 months from now). This translates mostly into: I help fix problems before other people even notice they are problems. This translates into: I think I do more good than harm. The jury is out on that, though.

I am not sure I want the jury's opinion on myself.

Dusty

On 29 November 2017 at 14:25, Justin W. Flory jflory7@gmail.com wrote:

On 11/29/2017 02:04 PM, Stephen John Smoogen wrote:

...

On 29 November 2017 at 12:29, Dusty Mabe dusty@dustymabe.com wrote:

...

Possible blame is definitely something I worry about when requesting access like this. I do propose that I work with some bumpers (i.e. only doing things after I get someone to review it, announcing things i'm going to do). Basically I would be in FBR mode all the time, except after I get reviewed I can actually do the work myself.

I am not clear on FBR means in this context. I am used to the context Full Blown Root which is not what you are meaning.

I read "FBR" as Freeze Break Request.

Thanks. I should have realized that.

-- Cheers, Justin W. Flory jflory7@gmail.com

---

infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org

On Wed, Nov 29, 2017 at 02:35:13PM -0500, Dusty Mabe wrote:

ask for root access. Patrick explained to me that "you're asking for root access", so yes then I said: "sure I guess i'm asking for root access". You voiced concern. To curb that I suggested sudo rules for those things I asked for in the original email.

Some of the things (like restarting composes) will hopefully be solved in the F28 timeline by the releng automation stuff we've asked to be prioritized.

On 11/29/2017 09:49 AM, Dusty Mabe wrote:

So in summary from your response I gather I am requesting:

• root on bodhi-backend01

I've been spending the last couple of weeks working on refactoring the way that mashing (punging?) works[0] for many reasons, but one explicit goal is to take away root on this box from most of the people that currently have it[1].

[0] https://github.com/fedora-infra/bodhi/projects/2 [1] https://github.com/fedora-infra/bodhi/issues/463