Nice feedback! I've managed to digest a half of it, so let me react to the first half and I will try to follow up with the rest later. :-) 

I don't know Solaris IPS (I'll definitely have a look), but from what I understood, it looks like they are delivering system components as multiple repositories. Each repo delivers a component which you can install in "many ways" using something called facets - like devel facet, docs facet, etc.

In modularity, we are also (kind of) delivering multiple repositories [1] - even though we are still using RPM packages as a delivery mechanism - (we call these modules) and we can also install them in different way using something called "install profiles". Install profiles define a set of packages from a given module that will be installed on your system. So you could have a "default" install profile, "devel" one for -devel packages, "docs" one for -docs packages, "lang-en_GB" for language, etc. You would be able to install one or more install profiles. We would still (at least for now) use RPM as a delivery mechanism, but from my understanding, it could achieve very similar results.

Modules are defined by modulemd [2] - including list of packages, install profiles, etc.

What do you think?

Cheers!
Adam

[1] https://asamalik.fedorapeople.org/modularity-building.jpg
[2] https://pagure.io/modulemd/blob/master/f/spec.yaml




On Thu, Mar 23, 2017 at 9:02 AM, Tomasz Kloczko <kloczko.tomasz@gmail.com> wrote:

OK. Will try to write longer comment :)

Looks like more or less Modularity people are trying to solve problems already solved in Solaris IPS mediators which allows solve
problems of delivery software in different versions/variants.
Trying to solve such dilemmas by manage as well software which is not delivered as regular package probably will break completely as
software in working state as non-packaged software will not have proper requires/provides description.
IPS trying to solve such problems is using know from Linux alternatives management + dependency resolver.
Highly likely solving multiple versions software delivery issues on top of raw rpm packages (as they are now) will end up big mess
behind. In other words such support IMO must be anchored way deeper into PM like it was done in IPS.
Another issue is that some level of flexibility manipulation of versions of some components may be working only in strictly controlled
sets of versions of the software.
Deliver such such baselines is solved in IPS by incorporations idea which is mechanism guarantee consistency on some exact areas where
delivery of alternatives is possible. Single incorporation simple locks all possible to install packages on exact version and releases.

At the moment similar locking is done in rpm based distros like Fedora is done by delivery whole distribution. However all internal
dependencies inside exact distro version are based only on serial:version-release(arch) dependencies and there is no mechanism which
will start delivery alarms or will not allow to install anything from other versions of the distribution.
There is no in current rpm based approaches mechanisms allowing make upgrade from distribution version N to version N+1 with
signalisation that some already installed packages are/will be breaking dependencies encircled on areas on which exact distribution
packages have been tested together,

Generally delivery packaged software in multiple versions on top of packages like rpm will be really hard if not impossible, as long
each variant adds yet another dimensions the same stuff which needs to be delivered.

This is why IPS completely moved away from packages delivered in form archives and switched to serving software in form of
repositories.

IPS mediators + facets ideas really solves Modularity problems and few other things as well.
Surprisingly something like IPS facet idea in some very limited form is available on top of rpm.
For example at the moment is possible to choose install everything with or without documentation (rpm exclude doc mechanism) which is
basing on %doc tokens in %files sections. The same is with choosing languages/locale dependent files basing on %lang() tokens.
In each of those two "dimensions" are used "dimension" specific %files tokens. In other words in rpm world is possible to choose within
 only those two defined "dimensions".
The same possibility of customisation in case of IPS is delivered in more general way in form of facet like doc=[true|false],
locale.<lang_name>=[true|false], The same file in package description can be marked using multiple facets as well. Something ca e
documenttion in exact language.

How this approach may be used on some other areas?
One example: someone on top of typical system want to compile something because additionally software must be tsted "in situ". Choosing
some exact set of -devel packages to install to start compiling some software? No .. just "pkg change-facet devel=true" and within few
minutes ALL already installed pakages will be enreached by adding all files with devel=true facet.
On Solaris there is no separated devel packages!!!!

This beautifully as well interacts with AI (Automated installer) manifest where on specifying installation of the system with exact
attributes is possible nicely described this by:

    <software type="IPS">
      <destination>
        <image>
          <facet set="false">facet.doc.html</facet>
          <facet set="false">facet.doc.pdf</facet>
          <facet set="false">facet.locale.*</facet>
          <facet set="true">facet.devel</facet>
          <facet set="true">facet.locale.en</facet>
          <facet set="true">facet.locale.en_GB</facet>
        </image>
      </destination>

Solving problems of moving around software in form of archives? No problem 'software type="ARCHIVE"' (Unified ARchive). For example
initial version of OpenStack packages on Solaris where provided as UAR.
Example from my first experiments with OpenStack trying to use it by install and setup this software over AI manifest and profile:

    <software type="ARCHIVE">
      <source>
        <file uri="http://<hostname>/AI/sol-11_3-openstack-x86.uar"/>;
      </source>
      <destination>
        <image>
          <facet set="false">facet.devel</facet>
          <facet set="false">facet.doc.html</facet>
          <facet set="false">facet.doc.pdf</facet>
          <facet set="false">facet.locale.*</facet>
          <facet set="true">facet.locale.en</facet>
          <facet set="true">facet.locale.en_GB</facet>
        </image>
      </destination>

Back to example with compiling something on top of regular system ..
So .. software has been compiled and we have now binaries.
OK. Another command "pkg change-facet devel=false" and all devel stuff is removed. Someone want to have an access to documentation
during development process? .. easy to guess "pkg change-facet 'doc*=true'" (there are few doc* facets used in Solaris packages).
It is possible to add dependencies on facet dependent files.

Few days ago bugs in build-id infrastructure which is now integrated within each rpm packages kicked hardly. Good that it has been
already solved (partially).
Generally build-id tries to solve delivery of debuginfo packages/resources for exact and matching versions-releas(arch) for example
core dumping binaries (https://bugzilla.redhat.com/show_bug.cgi?id=1433837 are some of my comments how build-id could be still
simplified if it would be relaying more on packages database). However some bugs are still around (https://bugzilla.redhat.com/show_bug
.cgi?id=1434235
)

So how such problem cold be solved using IPS approach? Simple by debuginfo=true facet on exact set of packages. PM software will find
exact files which needs to be installed on the system. All without hardcoding as it is now within additional ELF section build-id
hashes.
In repository exact packages will exist with debuginfo=true files however on exact system image those files may or may not be installed
depends on facet debuginfo=[true|false] state. By addng dependencies between debuinfo resources in the same way as between regular rpm
packages is possible to 100% reuse dependencies resolver.

As I've mentioned rpm allows use only two facets-like types of tagging. Solaris uses more than two hundreds.
Typical dilemma of cutting system image to absolute minimum is solvable by facets customisation without changing set of installed
packages. No doubts that something like extreme cutting off used disk space will fail because will not allow to new states of facets
with breaking some dependencies.

---- *** ----

What I'm trying to tell by above is that technologies like rpm have been designed on top of quite precise assumptions and used
approaches on solving some scenarios have been architected within scope of those assumptions.
When now original design design needs to be transformed to handle few new assumptions, without breaking scaffold of original design
everything likely will end up in form of growing and breaking apart ball of yeast than something solid.

Providing software in form of only repositories (and effectively breaking paradigm of package as file/archive) solves perfectly on
source side providing software for multiple distro versions of within *single repository*. What it means? No longer updating
repositories addresses on major upgrade -> one less point o fail on whole upgrade procedure.
The same repository is used as well to provide software for multiple architectures and packages on repo side are sharing files with the
same checksums.
As long as new version of the package A delivers only one changed file in new version on new version package everything else on repo
side will be shared between multiple versions of the packages. Packages can be way bigger and as long new versions of the same packages
will be changeing only some subset of files owned by package over network automatically will be transferred only what has been changed.
Solutions like drpm (delta rpm) are completely not needed because delta resources are automatically formed.

If someone is interested some more details about IPS please try to have look on source code repo https://java.net/projects/ips/sources/
pkg-gate/show

IPS code is probably something like +20 times smaller than rpm, dnf and all additional python modules code combined. Aditionally it
provides all repo side services with caching and providing multi layered repos infrastructure services software. IPS is fully written
in python and in many places still it solves a lot of more problems/scenarios which are still ahead of Fedora to solve.

For example on may systems happens something like this that someone installs some additional package as JFDI solution. As it was done
during weekend Monday the same person forgets that temporary solution need to be solved in some clean and tested way. After this
someone else starts using this newly installed software adding to the system software some script. This scrip have been even added to
install profile used in full OS reinstall/DR recovery procedure. However after next cycle of reinstallation such script starts failing
and no one remember why and what is needed.
How to avoid such scenarios? Easy: by locking whole set of packages after initial installation by executing from cron every day
uninstall every package which is not within originally locked set of packages. By this our example script will fail next day after
installation recalling automatically to finish solving JFDI properly much earlier.

Part of the IPS internal simplicity lies as well on top of other OS provided technologies like using snapshots. Even single new package
installation starts from creating on affecter volumes snapshots. If package installation fails and it is usually hard to say how to
roll back all changes on PM layer. So .. no problem. Just roll back everything to checkpointed state in matter of fraction of second.

The same approach is possible to use on Linux. However to solve this the same way all ext, xfs and few other FSes needs to be excluded
from new approach and only btrfs ATM could be used as only fully supported platform. Radical approach .. but 100% it will be working
without breaking internal simplicity.

What is more important is that on top of IPS have been already proven that this new approach is working. In other words IPS it is stash
of tested in combat ideas (I'm not suggesting to switch from rpm to IPS because ~99.99% Linux community ATM is not mentally ready to
start thinking about more radical approaches to PM :) )
IMO it would be really good if people involved in Modularity will have closer look on IPS to avoid reinventing the wheel.
Best would be to switch to IPS but probably this time again it will be not possible to avoid NIH syndrome :)

Whole and so huge IPS simplification on code layer was possible only because about decade ago few people come to conclusion that it is
no longer possible to solve new problems using old paradigms of SySV packages. rpm still sits very hardly on basic SySV packages ideas
which as the set they've been invented *~30 years ago* (a lot of people here was born around the same time when those fundaments have
been lied :) ).

As I'm looking one more time now on Modularity I think that those problems which this project is trying to solve should be handled
exactly in the same way because if not .. whole project IMO has high chance to fail.
IMO it is only matter of time when rpm will be abandoned because no longer would be possible to stretch this software onto new needs
without breaking internal consistency.
rpm still is very strong but already with each day is slightly weaker and weaker.

kloczek
--
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-leave@lists.fedoraproject.org



--

Adam Šamalík
---------------------------
Software Engineer
Red Hat