On Thu, 2 Apr 2020 at 22:33, Adam Williamson <adamwill@fedoraproject.org> wrote:
On Thu, 2020-04-02 at 21:58 +0200, Clement Verna wrote:

[evolution is still crashing. sigh. such is life. apologies for
formatting, again]

> > So yeah, let's discount the releng folks first, because releng has
> > existed all along, and - as I said - my original statement was not
about
> > "people who are organizationally in the same team as the people who
work
> > on Fedora app stuff" but "people who work on Fedora app stuff". So
that
> > lets out Mohan and Tomas.
> >
>
> I don't think this is fair at all, Tomas and Mohan are doing a lot of
> development and trying to improve a lot of the tooling around releng
so
> that we move from a manual heavy process to a more automated or at
least
> tool assisted process. If you consider that releng tooling is not
> application work, then please explain to me what is bodhi,
> release-monitoring, anitya, mdapi etc ... It seems to me that these
> services are 100% release engineering focused.

I think we've more or less beaten this list thing to death and maybe we
don't actually disagree at all (can we say it actually turns out to be
more or less a wash and move on? I am at this point willing to concede
that my impression that the headcount was *higher* before was
mistaken), but I didn't mean "discount them" in the sense of "discount
their contributions", I meant "discount them from the comparison",
because I didn't include the people working in releng two years ago in
the "before" list. We could add the current releng people to the "now"
list and the previous releng people to the "before" list, but what
would be the point, beyond beating this dead horse even harder? :)

> Here's Vipul's lists:
> > https://github.com/siddharthvipul
> > https://pagure.io/user/siddharthvipul1
> >
> > he seems to work exclusively on CentOS CI. Okay, Fedora *uses*
CentOS
> > CI, but presumably back in the 2018 timeframe, someone (whether
that's
> > Vipul or someone else) was working on CentOS CI who wasn't included
in
> > my list, because I only listed people working on Fedora stuff. So
this
> > still seems like a wash.
> >
>
> Vipul and I have done extensive work to add the OSBS aarch64 cluster
in
> staging, and it might comes as a surprise but yes we are working as a
team
> and even sometimes pair programing and sharing knowledge. But you
will find
> some of Vipul's contribution on the ansible repo git logs. Also this
work
> is directly coming as a request from the council to support the IOT
> objective, so I think it is fair to count it as "development" work
even if
> this was mostly operation work and deploying a new OpenShift cluster
for
> OSBS, since this time could have been spend on other application if
that
> was asked by the council.

Same point: I wasn't meaning to dismiss anyone's contributions, only to
try and keep the *comparison* valid, so it's not valid to include
"people working on CentOS CI" in the 'now' list (as they're now
accounted as part of 'CPE') but not include people who were working on
CentOS CI in the "then" list (because they weren't part of 'Fedora
infra'). But please let's just leave this point now? :)

> Yeah we are even, and we have 2 new persons joining the team next
week with
> a more sysadmin/operation profile because we really need to support
nirik
> and smooge in that area. I think what you are failing to see is that
for
> roughly the same team, there is much more thing to do. The project
keeps
> adding new things, we now have containers, flatpaks, IoT, silverblue,
> CoreOS and this is good thing but it adds more work on the team for
example
> the releng work that was needed 5 years ago has now triple, same
thing on
> the infra side. On the application development, tools and application
have
> to be adapted to take care of these new deliverables. I am pretty
sure you
> know this very well because that must impact you also on the QA side.

Yeah, you're right, so I'm certainly not "failing to see" that. But I
think we may have lost track of where this discussion got started. Let
me quote myself from right back at the start:

"At the very least, if we have somehow reached a point where Red Hat is
no longer willing to provide sufficient resources to run Fedora on the
lines the Fedora community wants it to be run, we need to recognize
that this is a significant problem that needs to be properly aired and
discussed and resolved. In this context I'll note that the apparent
significant headcount reduction of RH people working on Fedora
infrastructure over the last few years is in itself a worrying trend,
particularly if you consider it while reading Clement's email."

*that's* where this whole "headcount" subthread kicked off. As I
mentioned above, I'm now willing to admit that my impression that there
was an "apparent significant headcount reduction" was off the mark, but
to me it seems that in saying "for roughly the same team, there is much
more thing to do", you're *supporting* my initial point here, if
anything, right?

Yes, indeed I see your point and this whole thread made me realize that maybe we have not been good enough at communicating that we need help. I mean one way to see it, is that Red Hat should be investing more in the team but another way is that we are failing to get more support from within our community.
As Neal and Smooge mentioned it is difficult to contribute and help in the infra and releng activities, it is not impossible but there is a very high entry fee to pay which is to understand how all the tools and services interact together + the history of why this was done that way.
In the same time, I honestly think the team does a good job at being transparent and welcoming to people willing to help, we have weekly and daily "standup" style meetings on IRC (infra and releng), we have started to do backlog prioritization on the infra list. We have also tried in the past to have office hours or apprentice day to help people make some contribution. Unfortunately the reality is that we don't have much participation to these initiatives. I am very happy to think that we are doing something wrong, and if anyone has ideas on how we could improve or make it easier to contribute to infra and releng please reach out.
 

> > To be really clear: I don't want the takeaway from this to be "Adam
is
> > very mad and doesn't want CPE to be allowed to work on cool new
projects
> > any more". I like cool new projects! Cool new projects are great! I
> > write them myself sometimes! I'm just having trouble joining up the
dots
> > here in terms of high-level strategy.
> >
>
> We are really starting with prioritization and trying to be better at
how
> we organize our work, so yeah maybe not every is great, but I did not
know
> that perfection was the minimum required.

Sheesh, I thought everyone knew that perfection is the baseline for
anything where LWN and Phoronix might be reading ;)

>  We also share a weekly status
> update of every that is going on and welcome feedback and comments.
Also if
> you think that we are really on the wrong tracks please feel free to
reach
> out to the council or directly to Aoife which is working really hard
to try
> to make sense of all the requests and needs that arrives at our door.

I certainly don't mean to give that impression. I meant what I said:
I'm having trouble trying to reconcile the various different statements
about resource prioritization from Smooge and Randy and yourself and
Leigh with the whole forge decision process, and I'm struggling to make
sense of it. In my head it is adding up to something like "we
absolutely don't have anything like the resources to maintain all the
legacy stuff (including Pagure) let alone take on new stuff BUT
continuing to maintain and host Pagure was absolutely a valid option
and one we needed to go through a lengthy decision process to discount
BUT "24/7 availability in an SLA model and not hosted by the CPE team"
was a top level requirement BUT we do have the resources to work on
*this* new thing..."

I'm not saying it *definitely doesn't* add up. I'm not in a position to
make that assertion and I wouldn't presume to. I'm just saying I
haven't yet found it all laid out somewhere where I can see how it adds
up. That's why I phrased it as "I'm having trouble joining up the
dots".

> I think it is important to note that a lot of people in the team are
quite
> new to being part of a community. Honestly I don't think we are
giving them
> a really good experience. But what strike me the most is the little
trust
> there is in our team. When a group of people is working as hard as
the
> folks in the CPE team, (some being around during the weekends or
waking up
> in the middle of the night to restart a service) is telling that they
> cannot sustain the amount of work they have, there is so little trust
in
> that team that we have to go and check the commit logs of these
people to
> see on what they are working and if this time is well invested.
>
> Honestly I have no words, and no motivation to go and fix any of the
> tickets that are waiting to be fixed.

To be absolutely 100% clear on this: I am absolutely not trying to dunk
on the CPE team here. I deeply appreciate all the work you folks in the
trenches do. I am trying very hard not to denigrate any specific
*people* here, because it's against the CoC and I don't like doing it
anyway: as I've always tried to say in discussions like this, I
absolutely believe good people with good intentions sometimes don't do
things as well as they could have been done, and I absolutely include
myself in that group, and I hope I'm always open to the same kind of
feedback on my decisions (as opposed to me personally) from others. But
the thing I am asking questions about here is the *decision making
process* and the *strategic vision*, which if I have to pin it down is
clearly a responsibility of management.

Yeah I know you meant well and were trying to prove your point. I just felt it was important to remind that these are great people, dedicated and doing a lot of work for the good of Fedora and CentOS.
 
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org