#5866: Create new channel for building Java packages

List overview All Threads
Download

newer

older

#6037: Create .xz images for...

RFC: Proposal for Rel-Eng project...

Fedora Release Engineering

11 Mar 2014 11 Mar '14

5:59 a.m.

#5866: Create new channel for building Java packages ----------------------+------------------------ Reporter: mizdebsk | Owner: rel-eng@… Type: task | Status: new Milestone: | Component: koji Keywords: | Blocked By: Blocking: | ----------------------+------------------------ == Request ==

Please create a new Koji channel for building Java packages which would exclude ARM builders, or implement an equivalent solution. My goal is to be able to build noarch Java packages skipping any ARM builders. Using custom koji command is not a problem for me.

== Justification ==

The great majority of Java packages are noarch. As such they are most often pisked up by ARM builders.

Java performance on ARM is very poor. The situation got even worse after JIT for ARM was disabled in JVM. ARM machines are often slower by one to three orders of magnitude than x86_64. Some tests which execute in a few seconds on x86_64 can take over half an hour on ARM.

Current state of Java on ARM combined with Koji configuration makes it difficult to maintain Java packages in Fedora. A substantial amount of human resources is wasted on waiting for builds to complete, debugging tests which fail only on ARM, keeping track on builds and resubmitting them. That resources could be utilised better and I think that justifies my request.

== Alternative solutions ==

I have tried differnt alternative solutions since ARM became a primary architecture and none of them is good enough.

One ad-hoc solution is submitting numerous dummy koji tasks for ARM builders until their capacity is exhausted, then submitting Java package build and canceling ARM tasks. This is obviously a hack and should be avoided.

If Koji configuration is not improved the only remaining solution will be converting Java packages from noarch to arch-specific and adding ExcludeArch. I still hope that ARM situation will eventually improve, so I would like to avoid doing drastic things like that, if possible.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Show replies by date

Fedora Release Engineering

11 Mar 11 Mar

6:07 a.m.

#5866: Create new channel for building Java packages -----------------------+----------------------- Reporter: mizdebsk | Owner: rel-eng@… Type: task | Status: new Milestone: | Component: koji Resolution: | Keywords: Blocked By: | Blocking: -----------------------+----------------------- Changes (by mbooth):

* cc: mbooth (added)

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:1 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

6:08 a.m.

* cc: jerboaa (added)

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:2 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

6:32 a.m.

Comment (by akurtakov):

I second this request as from Eclipse POV it's a nightmare to wait for 20 h when x86 needs 30 min. It's a clean showstopper for doing any serious work on Fedora if one starts build and has to wait for next day before seeing results. Even though Eclipse itself is an arch package some solution needs to be found as this is preventing any serious work on Fedora.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:3 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

6:49 a.m.

Comment (by aph):

This isn't such a terrible idea, given that some ARM boxes are often much less powerful than our faster x86 builders. However, the ARM-32 JIT is (hopefully) about to get fixed, and ARM boxes are due to get quite a lot faster.

Having said that, this is really an infrastructure bug: the Fedora build system really should have some notion of priority when deciding where to place jobs

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:4 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

6:54 a.m.

Comment (by ahughes):

I'd be for a more intelligent build system that knows to allocate noarch builds to the fastest builders, rather than an arch-specific inclusion. That should really have been in place before making ARM32 a primary architecture. You'd have the same problem if some other archs were primary as well. Maybe look at how Debian deals with this?

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:5 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

7:18 a.m.

Comment (by bochecha):

...

the Fedora build system really should have some notion of priority when

deciding where to place jobs

It does.

That's why it allocates noarch builds in priority to the least loaded machines, which turn out to be ARM most of the time as there are many more ARM builders.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:6 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

7:24 a.m.

Comment (by mizdebsk):

Replying to [comment:5 ahughes]:

...

I'd be for a more intelligent build system that knows to allocate noarch

builds to the fastest builders, rather than an arch-specific inclusion. That should really have been in place before making ARM32 a primary architecture. You'd have the same problem if some other archs were primary as well. Maybe look at how Debian deals with this?

I'm not even dreaming about improving Koji, which would require some effort. All I ask for is a few lines of configuration to allow me to tell Koji to exclude ARM builders when I am building Java packages.

Debian developers can build binary packages on their own systems and upload them to FTP.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:7 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

9:06 a.m.

Comment (by rdieter):

I'd suggest discussing options on devel and/or buildsys lists first, and once some consensus is reached, *then* work to implement it (and not before).

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:8 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

12 Mar 12 Mar

3:11 a.m.

* cc: msrb (added)

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:9 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

24 Mar 24 Mar

8:56 a.m.

Comment (by mizdebsk):

Could we please get some reply to this request?

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:10 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

9:21 a.m.

Comment (by rdieter):

there is no specific/actionable request here, just a criticism about the current setup being slower and not-ideal for java-related builds.

See comment 8 about constructive ways forward (e.g. discuss on mailing lists first).

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:11 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

9:38 a.m.

Comment (by mizdebsk):

Replying to [comment:11 rdieter]:

...

there is no specific/actionable request here, just a criticism about the

current setup being slower and not-ideal for java-related builds.

There is a clear request for creating a Koji channel, which is in domain of release engineering. Other packages (like kernel) have their channel and I don't see a reason a channel couldn't be provided for Java packages.

...

See comment 8 about constructive ways forward (e.g. discuss on mailing

lists first).

I would like to get an authoritative answer from a member of release engineering team before trying other means of resolving this issue.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:12 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

23 Apr 23 Apr

6:29 a.m.

Comment (by goldmann):

+1 for this ticket.

A local build of `wildfly` application server takes a few minutes, whereas on koji ARM builder it takes about 3 hours. This makes it impossible to rapidly fix build issues.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:13 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

9:57 a.m.

* cc: jreznik (added)

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:14 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

24 Apr 24 Apr

9:40 a.m.

Comment (by jreznik):

Replying to [comment:4 aph]:

...

This isn't such a terrible idea, given that some ARM boxes are often

much less powerful than our faster x86 builders. However, the ARM-32 JIT is (hopefully) about to get fixed, and ARM boxes are due to get quite a lot faster.

I talked to jvanek, he thinks JIT could be turned on now for the next update, probably happening in the first half of May.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:15 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

28 Apr 28 Apr

10:51 a.m.

Comment (by sochotni):

I believe the JIT will help '''somewhat'''. What I don't believe is that it will magically bridge the whole gap. PPC doesn't have JIT and it's still much much faster then ARM. For comparison, eclipse build times: * x86_64 (JIT): 30 minutes * ppc64le (no JIT): 3 hours * armv7hl (no JIT): 20 hours

So even if JIT sped it up tenfold, it would still be 2 hours vs 30 minutes. Sure, most packages don't take '''that''' much time, but try maintaining 200+ packages like this.

Another point is that quite a number of packages have testsuites that are timed. On arm hardware these tests often fail (sometimes due to race conditions in tests, sometimes because the code is really slow...). In either case we lose a lot of time debugging and fixing testsuites (which are not packaged or executed) instead of actually improving Fedora. I'll also note that a lot of testsuites are "write-once-read-never" deals and '''are''' hacky, but they are not meant to be perfect.

I don't see reason why we should inflict this on people working on Fedora when we clearly have a better way to deal with this.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:16 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

23 May 23 May

11:56 a.m.

Comment (by ausil):

I made a change to the capacites of the builders last night.all the x86 builders have higher capacities than the arm builders. what this should mean is that most of the time an x86 builder will take noarch only builds. but they will go to arm builders in times of high build loads.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:17 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

11 Dec 11 Dec

4:54 a.m.

Comment (by raphgro):

:+1:

Could the koji balancer just parse for any "java" keywords in SRPM/spec file?

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:18 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

6 Jan 6 Jan

10:20 a.m.

* cc: raphgro (added)

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:19 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

29 Apr 29 Apr

12:48 p.m.

#5866: Create new channel for building Java packages -----------------------+----------------------- Reporter: mizdebsk | Owner: rel-eng@… Type: task | Status: closed Milestone: | Component: koji Resolution: fixed | Keywords: Blocked By: | Blocking: -----------------------+----------------------- Changes (by ausil):

* resolution: => fixed * status: new => closed

Comment:

I am going to close this as fixed. seems the ssds speed things up enough

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5866#comment:20 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

3507

Age (days ago)

3921

Last active (days ago)

rel-eng@lists.fedoraproject.org

20 comments

1 participants

tags (0)

participants (1)

Fedora Release Engineering