GitHub migration again -- The Plan

Lukas Krejci lkrejci at redhat.com
Fri Mar 1 08:06:53 UTC 2013


On Thursday, February 28, 2013 16:53:33 Stefan Negrea wrote:
> ----- Original Message -----
> 
> > From: "Lukas Krejci" <lkrejci at redhat.com>
> > To: rhq-devel at lists.fedorahosted.org
> > Sent: Wednesday, February 27, 2013 10:37:12 AM
> > Subject: Re: GitHub migration again -- The Plan
> >

... snip ...

> > > Splitting the repository will have important implications on CI
> > > side and
> > > we might also need a plan for Jenkins jobs.
> > 
> > This should be fairly simple, IMHO: each job we have will have to
> > copied for
> > each new repo.
> > 
> > So instead of having rhq-master-compile, we're going to have to have:
> > rhq-core-master-compile
> > rhq-helper-master-compile
> > rhq-plugins-master-compile
> 
> If we do this, then we would need to re-work the maven repository isolation
> method that we currently have in place. Today every single job has it's own
> local maven repository. The separation was implemented because we have a
> total of 7 cascading Jenkins builds on the same branch (and most of the
> times two or three of them run concurrently). If these jobs were to use
> some shared repository they would step on each other's toes. We had huge
> stability issues in the past because two jobs were deleting/updating
> artifacts at the same time.
>

This is why I think it is important to be strict about only depending on 
released and published core-artifacts in non-core modules. That way you don't 
need a shared repo and can keep a repo-per-job as we do now.

> The other challenge that I see with splitting the current jobs into three
> jobs is the explosion on the number of Jenkins jobs. Today we have a chain
> of 7 jobs chained for master. Multiply that with 3 and we would need to
> monitor, update, and fix about 21 jobs. If we have to maintain builds for
> two branches that would be a total of 42 jobs. Think now about the trivial
> task of updating the maven version, it will become non-trivial and error
> prone.
> 
> From my perspective the only way to monitor a chain of about 21 jobs is with
> the help of a build pipeline. Currently we do not have that capability
> implemented in Jenkins (... even though I requested it about 6 months ago).
> And Jenkins support is rudimentary at best when it comes to build pipelines
> anyway.
> 
> 
> I know the hope was to split the current jobs into smaller, nimbler jobs. We
> can still probably achieve this, but we need to think it through more and
> 3x factor is not easy. The main limiting factors here are the build system
> and maven.

I don't think we need to multiply all the jobs by 3. For example, we have a 
job running the tests against an Oracle database. Running a plugins build 
against an Oracle database just doesn't make sense, because the plugin tests 
don't interact with the database (maybe barring the Oracle plugin's 
integration tests if there are any - but itests are another whole story 
orthogonal to this discussion). But generally you concern is very valid - the 
number of builds will increase and we will have a harder job maintaining them.

I don't think that's necessarily a bad thing though. By the split, we're just 
"formally" admitting that we maintain 3 separate projects - RHQ itself, 
plugins for it and then a bunch of helpers that come in handy from time to 
time.

> > The good thing is that those jobs are going to be independent and
> > will not
> > require each other's build artifacts, because the policy we are
> > adopting is
> > that only published modules from the other repos are admissible as
> > deps.
> 
> I am not sure how that will work if you have to fix a bug in a plugin that
> also requires a change in the core code. I had at least a few of those
> while working on the content system or the AS7 plugin.
>

Heiko already mentioned it and I whole-heartedly agree with him. If you need a 
fix in a plugin that also requires a change in core, do the change in core 
first, publish it as a new incremental version and only then change the 
plugin. Basically we need to start thinking about the core server as a 
standalone project.
 
> What would the workflow be for a developer? What about creating a build with
> Jenkins?

Locally, you can do both the changes together, but when committing to the 
repo, you need to split the fix in two phases as outlined above.

While this is more cumbersome for the devs, I just can't see another way of 
providing a disciplined and usable API for the plugin developers in the 
community. If a 3rd party developer wants to develop a plugin for RHQ.Next, 
they are essentially out of luck today because they need build against 
snapshots which will force them to build the whole of RHQ from source just for 
the sake of developing the plugin.

While this change will slow down the progress somewhat, I think it's worth it 
- we will have more fine grained micro releases that will actually track the 
progress in the core.

> > One thing that WILL have to change rather substantially is the API
> > checking,
> > but that's not directly related to Jenkins, it's a build issue. The
> > API checks
> > now assume a uniform version of all RHQ modules, which is no longer
> > going to
> > be true.
> > 
> > > Regards
> > > Thomas
> > > _______________________________________________
> > > rhq-devel mailing list
> > > rhq-devel at lists.fedorahosted.org
> > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > 
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel


More information about the rhq-devel mailing list