With MPB, trying to rebuild all Ruby related packages, I'd hoped I'll get the results much faster.
The problem (to me) seems to be that packages are build in batches (as can be seen in attached screenshot). If there is some build which takes ages (this time it seems to be qt5-qtwebkit) in some batch, nothing else is processed unless that build finishes. But to me, the slow package has probably the lowest importance. I'd rather see the biggest chunk done than to endlessly wait for some package.
Can this behavior be modified? E.g. the batches could overlap slightly or rebatched with low number of tasks still running. Dunno. But this makes MPB / Copr useless for this task.
Vít
Dne 01. 12. 23 v 18:00 Vít Ondruch napsal(a):
With MPB, trying to rebuild all Ruby related packages, I'd hoped I'll get the results much faster.
The problem (to me) seems to be that packages are build in batches (as can be seen in attached screenshot). If there is some build which takes ages (this time it seems to be qt5-qtwebkit) in some batch, nothing else is processed unless that build finishes. But to me, the slow package has probably the lowest importance. I'd rather see the biggest chunk done than to endlessly wait for some package.
See this blog post:
https://pavel.raiskup.cz/blog/build-ordering-by-batches-in-copr.html
and this documentation
https://docs.pagure.org/copr.copr/user_documentation.html#build-batches
Can this behavior be modified? E.g. the batches could overlap slightly or rebatched with low number of tasks still
running. Dunno. But this makes MPB / Copr useless for this task.
Likely no. It does its job for most of the people. I.e. when you run two consequent 'copr build' then lots of users told us that they expect that the second starts only when the first one finish. Therefore I think that most people likes this behavior.
If you submit lots of builds at once, I recomend to utilize batches with --with-build-id and --after-build-id. With these option you can easily divide hundreds build into several batches and be sure they will be built in right order. IIRC KDE folks are utilizing this.
Dne 02. 12. 23 v 23:20 Miroslav Suchý napsal(a):
Dne 01. 12. 23 v 18:00 Vít Ondruch napsal(a):
With MPB, trying to rebuild all Ruby related packages, I'd hoped I'll get the results much faster.
The problem (to me) seems to be that packages are build in batches (as can be seen in attached screenshot). If there is some build which takes ages (this time it seems to be qt5-qtwebkit) in some batch, nothing else is processed unless that build finishes. But to me, the slow package has probably the lowest importance. I'd rather see the biggest chunk done than to endlessly wait for some package.
See this blog post:
https://pavel.raiskup.cz/blog/build-ordering-by-batches-in-copr.html
and this documentation
https://docs.pagure.org/copr.copr/user_documentation.html#build-batches
Can this behavior be modified? E.g. the batches could overlap
slightly or rebatched with low number of tasks still running. Dunno. But this makes MPB / Copr useless for this task.
Likely no. It does its job for most of the people. I.e. when you run two consequent 'copr build' then lots of users told us that they expect that the second starts only when the first one finish. Therefore I think that most people likes this behavior.
If you submit lots of builds at once, I recomend to utilize batches with --with-build-id and --after-build-id. With these option you can easily divide hundreds build into several batches and be sure they will be built in right order. IIRC KDE folks are utilizing this.
Ah, so this is how MPB is using batches. Thx for enlightening me. I somehow thought that the batches are used to e.g. limit resource usage if I fired like thousands builds at once.
I have followed up with MPB:
https://gitlab.com/fedora/packager-tools/mass-prebuild/-/issues/90
Vít
copr-devel@lists.fedorahosted.org
-
Miroslav Suchý -
Vít Ondruch