Hi,

Here's a summary of the Fedora.next and the Modularity Initiative presentation Langdon gave today. It's mostly a log copy/paste job, but I tried to follow the threading of the conversation and organize it a bit differently than a straight chronological log would have done.

---

= Modularization = The idea that we can start to work with modules as a larger granular unit than a package. This does not mean "packages" go away but rather that we start to worry about the things at a higher level. This allows us to recognize some of the changes to the development world and packaging world. This also allows tradeoffs between them as we go.

== Core Module == We want to consider a "core" module that is useful for physical, vm, and container images installs.. key being "binaries available but not all installed in all cases, but we also want to consider a few simple modules that are basically just "applications".."

== Example Modules ==

Simple module examples being considered: * httpd - in fedora but a desire for updates on their own cycle) * howdoi - a little python app not in fedora but fun and useful, also changes a lot)

== Motivations / Goals ==

Look at puppet as an example (which is in my forthcoming blog post).. important to fedora .. but countless problems about maintaining the "correct" version of ruby for it (and I am sure many other examples)..

1) allow software to operate at a difference pace from OS

one of the main goals here is to allow for things to operate on a separate pace from the core OS.. in other words.. less like a "distro is released at point x" rather "an os is released at point x" and a bunch of apps are released on their own lifecycles..

2) allow an application to have an independent lifecycle, including its dependencies, from other applications and the OS but only branch on deps when necessary

another motivation is "quality" a la rings.. but that isn't "quite" right.. as in the quality of a beta isn't "bad" it just isn't ready yet.. the "quality" of puppet isn't bad.. it just hasn't updated to latest ruby yet.. and the latest python isn't "bad" just the rest of the distro isn't ready for it yet.. so .. these things need their own lifecycles.. modularity is attempting to support that

3) make an explicit promise about whether a dep is for an app or meant to be shared

another motivation is the i included this package for my app to work but i am only supporting it for my app.. but, now a few apps use it.. and now we have another lifecycle problem.. does app-a or app-b get to decide on an update? what if we could declare a dep as a shared one or a private-to-app one..

4) move testing from implicit every package to explicit exposed public apis of a module

Make testing at a module level rather than individual or whole-distro level. right now we implicitly promise an api/abi for every rpm for the lifecycle of the release.. but we don't really mean it, i propose moving it to more explicit, limited api/abis at a less granular level

= Questions =

== Q: Module == role? == * roles are possibly one implementation or a subset of modules * roles, as i understand them, are about getting something "running".. a role may require / use one or more modules. but modules are more about the logical set of things.. rather than the use of them..

== Q: Would the core be the same for each use case or would it be tuned specifically via configs / etc for each case? == * the "module" would be the same... but the installation on demand would be different * the core is a set of packages, which may or may not all be installed for each usecase. for example, containers don't need a kernel. but the kernel is probably going to be in the core module * a module is just the software, the specific config is layered on top * If you want to build something and call it Fedora, the foundational pieces would come from that repo (core)

== Q: Q: So different lifecycles this implies that some modules are incompatible with others, right? ==

* yes * that is a great question.. i think interactions between them might be "not supported" (in a sense).. but i think the long term goal is that they can coexist without hurting each other.. aside from deps...as in httpd-24-module deps core-f23 & core-f24 .. but not core-f22 * not trying to go into implementation, but something like SCL's? ** scls are another good example of a "solution" that was never a first-order citizen.. just a way to solve a problem ** in other words.. we have lots of things that deal with this lifecycle symptom.. but not at the core (sorry to reuse the word) of the distro

== Q: how do we avoid balkanization and having to maintain 15 different versions of the same package ? and importantly, keeping things around that have known security issues? (how do we even maintain 2 version in Fedora, given currently you cannot have more than one version per package in repos ?) ==

* That's an open question and one that can *only* lead to complicated implementation discussions. Let's hold that for later. Or I guess we can discuss the high-level maintenance burden issues. * righto.. that is a problem.. but one we currently have.. i think it is just not as explicit.. we need to still foster a community working together... but, enforcement via fiat seems to cause workarounds not solutions * figuring out an implementation for that is part of the immediate work to do #todo * I'd suggest that each module should probably have its own SIG maintaining it. And if the SIG falls into a lack of maintenance, the module is retired. ** SIG is too big a construct ** something smaller than SIG ** maintainers/co-maintainers, like packages probably * People who need it for their module are responsible. If a version maintained to a high standard by someone else exists, great. If not, don't stop them from doing it as long as bar for security responsiblity is met

== Q: are modules "supported" (as in security) by Fedora itself? ==

* I'd like to have levels, ranging from "very centrally important -- critpath" to "gets security attention and help" to "copr-like edge" to "third party"

== Q: so there is a strong incentive to Fedora to keep the modules it supports on the same schedule? at least until/if tooling makes it a non-issue? ==

* Yes, very much so. Although we would at least have the ability to have differing schedules in special cases where we want it --- if Fedora atomic host wants to move kube+docker faster, then that would be possible if we have a distinct "container support" module - But still keep the core on the same schedule.

== Q: It seems like another requirement (or perhaps consequence) is "more maintainers" or more interested parties? ==

* A high-level hope is to move the focus of a lot of contributor activity away from working on individual tiny dep rpms to larger chunks.

== Q: Do we need a new packaging system to improve the app -> module in Fedora path? ==

* Maybe new tooling, but there's some work underway to help with automating the creation of RPMs ** I am thinking of delivery, the problem with rpms is that we'll end up with app-a requiring foo-1.1 and app-b requiring foo1.2 and if they are both build with standard rpms the 2 modules wll conflict *** In my ideal world, I'd like to see this handled with RPMs and groups/metapackages. So we don't lose the flexibility we have with all the little pieces. **** so you won't be able to install both app-a and app-b unless you allow relocation/embedding of packages ***** No: figuring out how to cleanly parallel-install deps is a key point. ****** I am not against, that's why I was thinking a new module-evel packaging system may be required, or may be we simply build a big RPM with the contents fo all the RPMs used to generate the module, a meta-rpm of sort - that exposes only the deps you want to make public, and not every singlle package, but that would require some relocation/smarts ****** i really don't want a new packaging method.. i would much rather embrace all the existing ones ****** it doesn't matter what packaging tool you use underneath, but you have to deliver a module and be able to coherently introspect it - call it a meta-package, you need info on all files, where they go, how to verify if they are still what was delivered, etc... ******* ahh i see what you are saying.. in that sense.. yes.., but.. this "packaging" would not actually package the binary like rpm does.. we are imagining a split channel.. metadata that maps to binaries through two channels *******You might run into the issue that some "modules" have pieces that don't need to be installed (Like the base distribution module) whereas on the app spectrum it's more likely all the packages in the module need to be installed. **** Indeed. So we may need to be able to package those bundled dependencies in a path that belongs to the app that wants them, and not have them share the system path. ***** langdon's app-to-module path. If a dependency is brought in purely to support one particular app, and the maintainer has no real interest in the dependency itself,then it may actually be just _fine_ to support multiple instances of that. Mark those dependencies as belonging just to the app. And if somebody wants to maintain the dependency as an independent thing in its own right, then it belongs in some module of its own, not in the app module. ******* that is kinda what i am thinking (marking as "for that app") **** but that is longer term.. for now.. we are just playing with modules as rpm repos.. the simplest case.. so a repo ~= module.. with some extra metadata and some client side tooling **** make dealing with libraries and dependent modules less time consuming, freeing effort for applications people really care about

== Q: not to be awkward, but has anyone actually convincingly defined a 'core / apps' boundary yet? /me tends to find one person's core is another person's app. which is GTK+? ==

* We've been kind of actively avoiding talking about the desktop stack, because it's the hardest problem. It's one of the reasons we came to Server SIG first: the headless case is a lot more direct. ** fine, which is python-requests ? * i think this has been a huge problem.. so we are somewhat taking the approach of "pull all the things out we can, and what is left is the core" * GTK+ is "stacks" :)

== Q: Should there be levels of importance inside modules? ==

* ie, the 'cockpit' module would have cockpit itself as critical, but 'libfoo' thats pulled in for something as much less so... * i dunno. if cockpit doesn't function without libfoo, then it's important. if it does, why is it in the cockpit module? * They care that it works *for their app* They may not care at all if it works for anyone else's app * well, it may only need some small part of libfoo and not care about anything else it does... so it could be upgraded and as long as that small part works, great * oh, yes. but that doesn't mean we need to mark importance levels - importance might have been the wrong word... * that is very much what i mean.. essentially a module only tests its deps for the api it needs.. if a dep becomes popular enough, it might get its own api test * its bundling in the sense of being declarative about your deps.. but not in the sense an end system owner can't update the deps directly.. even if it may break an app on their machine but module a and module b might both contain package C, at differing verions And yes, bundling/relocation are on the table here, with tooling to reduce the problems inherent there.

== Q: so how do modules map? or is that up to the maintainer(s)? ie, if there's fedora22/23/24 core modules, would there be just one httpd module that has 3 versions depending on which core? ==

* hopefully up to the maintainer.. but in the short term, 1-1 .. moving very quickly to maintainer choice * hopes no one points out having more than 1 core on one system ;) * ideally a module can have a single source base that works on multiple versions of fedora: Rather than having separate sources for fc22, and fc23, and fc24 you instead have sources for your module that get built on fc22, fc23, fc24. ** right, so there might be httpd22 httpd24 and they would be built against all the various core/modules that they have as deps... ** one source, many binaries ** write once, run everywhere! but the other way too ** as in .. core source may offer f22, f23, f24 apis .. even though it is only one source.. but i think that is a ways away - yes definitely a ways off

== Q: do we have a plan on where to start and where we'll see the first artifacts? ==

* We are updating the wiki pages right now (wiki/modularization), renewing the objective in a couple weeks, some prelim initial module attempts are getting written up now.. * also posting some blog posts.. actually doing commblog.. as it seemed the more appropriate ** yeah I was just saying to jzb we could do a user-focused magazine post pointing to your commblog posts, or just link it in 5tftw * nirik thinks this all sounds awesome. The devil is always in the details tho. * Right, the major intention of this meeting was to get a feel for "Does this sound like something we want to do?" vs "This is a terrible idea, go home and leave us alone"

== Q: So is Fedora the first distro to try this with such a broad target (ie: not just "make everything into a container")? ==

* i think so.. and i think 'make it all a container' is not gonna work..

== Q: nirik wonders about some kind of rpm namespacing to support this, but thats way too into the details weeds ==

* langdon mutters relocatable rpms..

== Q: We talked about automatically building the modules for each base, but is there value in trying to avoid doing that entirely when possible? ==

*For example, if we have modules with no shared deps, it makes sense to have the same exact module just tested atop F24 and F25 cores. * i think it is too early to say about optimizing builds.. i think we just "build it all" until we can tell we can short circuit.. i know sct had some thoughts on where can do that... but i would punt for now ** Right. The main thing with builds is that I would like automation to try all builds and tell us what breaks;

== Q: so which WGs/SIGs are we going to ask to spearhead this? == * part of the objective-renewal for the council is to ask base and e&s to make some changes to org and goals.. and then they would spearhead it

== Q: what about FTBFS (fail to build from source) against some base but not another? ==

* it may be ok if new packages fail to build on old cores, but the build system should tell us it fails and reject the build until we waive older builds. Ie. accepting failures should always require manual choice, the tools should tell us exactly what fails by default. ** yeah, and at that point you could decide that you don't want to keep supporting that old core or fix things so it works or whatever.

== Q: as an end user, how do I know that my system is safe from the vulnerability fixed in sec-patch? ==

* a point i am not sure was clearly made.. we don't want to ship anything that hasn't passed testing as a module.. so for example, app-a builds successfully w/ dep-d.sec-patch but app-b does not.. so app-a ships an update.. but app-b does not.. ** so app-b is still vulnerable to --dep-d.sec-patch but.. we need client tooling to allow 1) disable app-b 2) force patch app-b 3) ignore w/ risk ** app-b w/ risk seems like an ok idea on inward facing apps.. like .. what we also are supporting here is the end user system owner making explicit judgements about what risks they want to take..

== Q: from a UX pov, i think overall this is going to be a big win for users, who always want new hotness but can't without an OS update. i'm hoping that making it so you can build against older cores would make it easier to make new apps available for older cores? ==

* Probably, but as Time Passes, more and more stuff will likely end up bundled. * So it will result in an increasing maintenance burden over time. But with automation, maybe not too much, and modules likely will grow over time if they support older cores I bet. * supporting against the versions of Fedora is already really insane ... eg: without automation Cockpit would be impossible - supporting against N modules * M cores will take the already insane state to the next level ** if only it was a requirement before pushing *any* update to *any* package *** but CI fixes ALL THE THINGS! **** adamw looks forward to seeing the container truck fleet and helicopters turn up to deliver the CI cluster ** i guess my concern is that we are forcing a tree of modules and making a graph impossible if fragmentation happens at each/many levels of the tree there is no hope for tools and projects that try to make the result usable, integrated, etc. so there needs to be a balance found, and perhaps very strict CI would be that balance *** If we can get to a place where the API/ABI boundary is really what we care about rather than the package versions, I think it gets more straightforward **** indeed but in reality you have things constantly deprecating and then removing API, a good example is docker dropping things out of the API (with fair warning mostly) from version to version ***** sgallagh: it is a good example of how people do things unfortunately *** i guess i hope for automation to help with this.. as in, module owners have deps suggested to them when they work for them.. so the automation is actually trying to minimize the number of versions of deps ** i guess if we allow modules to require certain packages or versions of API, and as a result make them incompatible with other versions, then we'll at least model the issue quite nicely, and force fixes rather than ignore them *** if we are constantly scanning for where modules' deps overlap, and testing versions for simplification, i think this will be better than you think.. software is much more sophisticated in recent years.. with much better backward/forward compat in libraries.. **** yes it may have been worse ... but from working on Cockpit ... it's pretty bad, people are constantly pushing broken stuff into Fedora, from systemd, to docker, to kubernetes, to NetworkManager, to lvm - it's hard to think of a high level API that hasn't regressed at some point over the last couple years ***** right.. but we block that from coming in.. as in all the modules fail with new dep.. so no one picks it up ***** platform-api problem there.. i think that will be a harder case.. sorta.. ****** it's 2016 ... to me "API" means something remotable, like DBus, REST, spawning a process, etc ******* yeah.. i just mean the libraries that you use.. local or remote ******** right, so you're right that this is a requirement of modularization. it should be a hard requirement as in CI, otherwise we'd be building something that will get out of hand rapidly ********* i almost think it is a by product.. no module is updated unless it passes its tests.. so if you push a broken shared library.. no module will pick it up ********** currently most of the tests are done by things that integrate other things, the libraries and modules themselves don't do near the amount of testing required to make this work *********** yeah... glossing over the current lack of tests ;) but, as with all things software these days, need to start fixing recipes not cakes * I'd like to eventually get to the point where os core and modules can be mixed and match betweeen Fedora and CentOS/RHEL ** im not following - frankenlinux? *** no, because the modules are big, clear deliniation points, rather than organs all sewn together, mattdm remembers something about voltron. **** but i'm thinking the cores for centos vs fedora etc would be different? so how could you voltron modules ***** bring along the appropriate runtime / bundling ***** This is easier in container world.... with docker, use the fedora or centos base image, with xdg-app, use the centos or fedora runtimes ***** the other approach is rebuild, as people were talking about for f23/f24/f25 -- could be centos too ** four basic kinds of modules: base system; system services (e.g. logging); infrastructure (web, database, etc.); and then actual applications ** so by a module spanning fedora / centos you dont literally mean the same module, but the functionality / definition of the module *** I am partly Dreaming Big, here, not suggesting an immediate target, but I do want it kept in mind as we design *** with container approaches, maybe literally the same binary *** something like that, yes. it might be easier to think of it as "the source code for a module can be rebuilt on top of fedora and centos cores" and then with containers, it might map to the same binaries too. but again, that is all an eventuality we may not even try *** although, if we can, it means we really did disconnect the os from the apps.. and honestly, for many apps built on scripting languages.. i bet running on multiple cores is not that hard even today *** Or even running on golang and other bundle-the-world languages * from an end user perspective it is just, you have all current versions of an app avail for either cent or fedora.. but.. an app will start to have to seriously think about LTS style versions.. because the end user will start to care about their version stickiness more than the target OS. and if a patch comes out.. inkscape gets tested with the patch before the patch is shipped.. so.. inkscape may have the patch but openoffice may not.. because oo failed the new sec patch test... ** could also help with the 'can you duplicate this bug against the latest upstream git snapshot?' problem... :) sure, let me install that module, yes, here's the error. ** Probably not in the first blush, but tools to hook up a git repo and do automated nightly builds would be fantastic too ** we can also show to the app/module provider exactly what the end user is using when the bug was filed.. including force-patched fixes.. *** what is force-patched fix? **** that is my idea from above.. where oo failed the rebuild but the end user decided they wanted the security patch enough to force it on the module ***** I think we'll have to come back to that one. I have concerns, but they're too low-level for this discussion * aiming big: forget nightly. do it on each commit! ** Probably unachievable for huge projects like kernel or glibc *** the kernel already has CI that does builds and testing on every commit. unfortunately, the kernel also supports so much diverse hardware that it is impossible to test all those CI builds on all the hardware, so bugs still happen a lot. * but.. suffice to say.. an end user can exactly report what binaries are in use in a particular module, you know that the module has been approved to use with those binaries (or are warned), and so bug reports are rarely "what the h*@&*@k did they have installed besides my app" * https://wiki.gnome.org/Projects/SandboxedApps/NightlyBuilds

= To-Dos =

* a REQUIREMENT of modularity is significant tooling change/automation/improvement * we need to have automated rebuilds of modules.. we need basic testing of a module on security updates automatically.. we need to build all that out for modules.. to a much greater degree than we do with rpms.. because we don't want to update a module on the end system if it hasn't been tested as a unit * we also need to improve the "app -> module in fedora" path.. like really lower the barrier (of effort not quality) in an app developer maintaining their own module.. * We may need to be able to package those bundled dependencies in a path that belongs to the app that wants them, and not have them share the system path. * make dealing with libraries and dependent modules less time consuming, freeing effort for applications people really care about * I'd say a prerequisite for this is going to be Continuous Deployment-based testing of modules right from the start * to simo's point (sorta) earlier.. i would like the language packaging mechanisms to be more directly supported.. reducing app dev effort, simplifying packaging, not creating a new packaging system, relying on app language unbundling, etc