Good morning everyone,
I would like to start gathering our thoughts about python3 in our apps.
To my knowledge, we currently have two applications that are python3 (only): - Mailman3 core (as in hyperkitty is still py2) - mdapi
Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi runs on a Fedora node for the moment.
So, what do we think about python3 app in our infrastructure? Are we ok with it? Do we want to avoid them for the moment? Do we want to split 'backend' vs 'frontend' (ie web-apps)?
What brings me to raise these questions is that I have spent my morning getting pagure to run on pygit2 0.23.0 (the version present on F23) so that I can upgrade my laptop still running F21 atm. The result of this work is not pretty: https://pagure.io/pagure/pull-request/516 (and might still require some work, just the tests are passing at the moment, I would not be surprised if I missed some things in the templates). I am more and more thinking about replacing pygit2 and a friend of mine is developing a potentially really interesting alternative: maggit: https://gitlab.com/maggit/maggit However, maggit is python3 only. But the pain of dealing with pygit2 is such that I have been considering looking more into maggit and maybe porting pagure to it (which means porting pagure to python3, already checked, all of its dependencies are py3 ready).
Before I consider such effort (which would be a 2.0 release), I want to clarify the status of python3 web-apps.
So what do you folks think? :)
Thanks, Pierre
On Wed 02 Dec 2015 12:14:42 PM CET Pierre-Yves Chibon wrote:
Good morning everyone,
I would like to start gathering our thoughts about python3 in our apps.
To my knowledge, we currently have two applications that are python3 (only):
- Mailman3 core (as in hyperkitty is still py2)
- mdapi
Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi runs on a Fedora node for the moment.
So, what do we think about python3 app in our infrastructure? Are we ok with it? Do we want to avoid them for the moment? Do we want to split 'backend' vs 'frontend' (ie web-apps)?
What brings me to raise these questions is that I have spent my morning getting pagure to run on pygit2 0.23.0 (the version present on F23) so that I can upgrade my laptop still running F21 atm. The result of this work is not pretty: https://pagure.io/pagure/pull-request/516 (and might still require some work, just the tests are passing at the moment, I would not be surprised if I missed some things in the templates). I am more and more thinking about replacing pygit2 and a friend of mine is developing a potentially really interesting alternative: maggit: https://gitlab.com/maggit/maggit However, maggit is python3 only. But the pain of dealing with pygit2 is such that I have been considering looking more into maggit and maybe porting pagure to it (which means porting pagure to python3, already checked, all of its dependencies are py3 ready).
Before I consider such effort (which would be a 2.0 release), I want to clarify the status of python3 web-apps.
So what do you folks think? :)
Not part of Fedora infra of course, but my thoughts are that
* Fedora infra was relatively recently migrated to RHEL 7 * RHEL 7 has python 2.7 * Basically all current infra is 2.x * It seems there will be progressive move of Python ecosystem towards py3 over the course of RHEL 7 lifetime
I would venture a guess that we (that is you our lovely Fedora infra folks) will need to come up with py3 strategy/plan sooner or later (whether it's EPEL or RHSCL or...)
-- Stanislav Ochotnicky sochotnicky@redhat.com Business System Analyst, PnT DevOps PMO Team - Brno
PGP: 7B087241 Red Hat Inc. http://cz.redhat.com
Hi,
Good morning everyone,
I would like to start gathering our thoughts about python3 in our apps.
To my knowledge, we currently have two applications that are python3 (only):
- Mailman3 core (as in hyperkitty is still py2)
- mdapi
Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi runs on a Fedora node for the moment.
So, what do we think about python3 app in our infrastructure? Are we ok with it? Do we want to avoid them for the moment? Do we want to split 'backend' vs 'frontend' (ie web-apps)?
I personally think that it doesn't make much sense to split backend vs frontend requirements in this regard, since the distinction here differs per app, and may not be very clear. I'd say we either allow or deny python3 in total.
I also like the idea of having the option to switch to Python 3 in the infra, since more neat features will come to it. I would also prefer to keep running our infra on RHEL7.
That being said, I don't really know the current state of python3 for RHEL7. There is a package for EPEL7 (python34) that has not been touched since it was initially imported, and is still at python 3.4.3. This admittedly has not had a new release, but there are also several bug reports open against it that seem to have been idle for the last few months [1], and there are no co-maintainers.
One advantage of Python 2 at this point is also support. I'm not sure about the rest of the team, but I know that I personally am not really that well versed in the internals of CPython, and if something were to crash there, I'd have no idea what is going on exactly without a lot of debugging and digging. With python 2, we have a support account with which we could ask internal engineers to look into it.
All in all, I think that enabling infra applications to use python 3 would be great, but I'd like to see the current package for CPython improved, or more co-maintainers added, before I would feel completely comfortable. (Perhaps we can get the Fedora python3 maintainers to help with python34 for EPEL7?)
With kind regards, Patrick Uiterwijk
[1]: https://bugzilla.redhat.com/buglist.cgi?quicksearch=component%3Apython34&...
I don't contribute much time to infrastructure these days so I'll try to outline costs::benefits rather than saying whether something is a good idea or not -- I won't be spending much time doing the work so it's not right for me to sign other people up for it :-)
Good morning everyone,
I would like to start gathering our thoughts about python3 in our apps.
To my knowledge, we currently have two applications that are python3 (only):
- Mailman3 core (as in hyperkitty is still py2)
- mdapi
Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi runs on a Fedora node for the moment.
You'll probably either need to have a few load balanced Fedora webservers running mod_wsgi for python3 or the equivalent rhel7 webservers with mod_wsgi for python3 to do this. You'll want both python2 and python3 web servers because it will allow your applications to port one at a time. Doing everything at once will be overwhelming.
* Going with Fedora means a more frequent distro upgrade cycle but Fedora is pushing python3 as default so the support from packagers not in infrastructure is likely higher. * Going with RHEL7 means that you can continue with your current lifecycle for the distro but it means that you'll have to take a more active role in the python3-for-epel effort.
* Note: EPEL was started in large part because mmcgrath knew we needed extra packages for RHEL. Rather than doing it all alone, he pushed for us to share the workload with other people in the wider ecosystem that could also use the packages. So there probably is a bit of "if you build it, they will come here"... Infrastructure might need to contribute a fair amount of packaging time to get python3 on RHEL7 viable but after that there will be a steady stream of packagers who want to contribute to making the set of packages bigger.
So, what do we think about python3 app in our infrastructure? Are we ok with it? Do we want to avoid them for the moment? Do we want to split 'backend' vs 'frontend' (ie web-apps)?
backend-frontend may make sense because fedora is moving to python3. So frontends (command line CLIs and python libraries for people to write their own scripts or apps) that run on python3 and backends that run on python2 would match the existing reality of what people want to do. It doesn't help your pygit2/maggit problem on the backend, though. It also doesn't help if some of your CLIs also have to run in your infrastructure.
However, maggit is python3 only. But the pain of dealing with pygit2 is such that I have been considering looking more into maggit and maybe porting pagure to it (which means porting pagure to python3, already checked, all of its dependencies are py3 ready).
It could be less overall effort to port maggit to run on python2.7+
However, that feels like moving backwards instead of forwards.
Before I consider such effort (which would be a 2.0 release), I want to clarify the status of python3 web-apps.
I think starting to run python3 web apps will contribute more to the greater good than limiting apps to python2. It will help to future-proof the web-apps since python2 probably won't be default in RHEL8 (who knows if it will be available as an add-on, though)... so at some point you do have to pay this price. So the real question is whether you want to pay this price now and which people within infrastructure are going to be doing the work:
Do it when a RHEL releases with python3: * sysadmins will have to manage both RHEL7 and RHEL8 web servers for py2 and py3. * Concentrated work for developers to get applications ported as we'll want to migrate to rhel8 and python2 web apps will be holding us back. * Packagers will have to branch (and sometimes maintain) needed packages from Fedora to EPEL.
Do it now with the EPEL python34 (or a new python35) stack: * packagers will have to create python3x packages in EPEL * developers can start porting their apps now which gives them a longer window to port and perhaps less urgency * sysadmins will have to manage RHEL7 with python2 and RHEL7 with python3 web servers
Do it now with Fedora: * developers can start porting their apps now which gives them a longer window to port and perhaps less urgency * Packagers still have to get packages running on python3 but not as many due to efforts from Fedora itself. * sysadmins will have to manage RHEL7 with python2 and Fedora with python3 web servers. Fedora has a different lifecycle so there's additional work involved here.
Waiting for RHEL is probably the least overall work but there is a drawback there.... porting applications to python3 is probably what will take the most time so the sooner you can get started on that the sooner you'll be able to stop supporting python2.
-Toshio
On Wed, 02 Dec 2015 16:05:00 -0000 Toshio くらとみ toshio@fedoraproject.org wrote:
...snip...
I'll put in my 2cents here. ;)
I think starting to run python3 web apps will contribute more to the greater good than limiting apps to python2. It will help to future-proof the web-apps since python2 probably won't be default in RHEL8 (who knows if it will be available as an add-on, though)... so at some point you do have to pay this price. So the real question is whether you want to pay this price now and which people within infrastructure are going to be doing the work:
Do it when a RHEL releases with python3:
- sysadmins will have to manage both RHEL7 and RHEL8 web servers for
py2 and py3.
- Concentrated work for developers to get applications ported as
we'll want to migrate to rhel8 and python2 web apps will be holding us back.
- Packagers will have to branch (and sometimes maintain) needed
packages from Fedora to EPEL.
I think waiting this long is a mistake... we should get started (well, we already are really).
Do it now with the EPEL python34 (or a new python35) stack:
- packagers will have to create python3x packages in EPEL
- developers can start porting their apps now which gives them a
longer window to port and perhaps less urgency
- sysadmins will have to manage RHEL7 with python2 and RHEL7 with
python3 web servers
Do it now with Fedora:
- developers can start porting their apps now which gives them a
longer window to port and perhaps less urgency
- Packagers still have to get packages running on python3 but not as
many due to efforts from Fedora itself.
- sysadmins will have to manage RHEL7 with python2 and Fedora with
python3 web servers. Fedora has a different lifecycle so there's additional work involved here.
I'm somewhat torn between these two. I think it's going to depend on if we can get some time to really make the python3 stack on epel7 workable. I know the folks who worked on it were waiting to land all the 35 stuff in Fedora before looking at epel, but we likely need to get that done soon and get it moved to 3.5 and see what things we need there.
If it's not going to be workable we can do Fedora, it just means more work for us on the sysadmin side.
One thing that feel through the cracks recently was that when we first started doing non builder Fedora instances, I wanted them to auto update. So we had yum-cron setup on them. However, new instances haven't honored this and we need to move it to dnf-automatic anyhow. Doing that might help reduce the burden of updates flow, but might also increase breakage.
Waiting for RHEL is probably the least overall work but there is a drawback there.... porting applications to python3 is probably what will take the most time so the sooner you can get started on that the sooner you'll be able to stop supporting python2.
Ultimately I think it's important for us to start moving to python3. If we use Fedora or EPEL7 python3 I guess might be more up to the coders working on that.
kevin
On 12/02/2015 08:05 AM, Toshio くらとみ wrote: [snip]
Do it when a RHEL releases with python3:
- sysadmins will have to manage both RHEL7 and RHEL8 web servers for py2 and py3.
- Concentrated work for developers to get applications ported as we'll want to migrate to rhel8 and python2 web apps will be holding us back.
- Packagers will have to branch (and sometimes maintain) needed packages from Fedora to EPEL.
RHEL currently provides python3 via SCL. Is this insufficient for infrastructure's needs?
[snip]
On Fri, Dec 04, 2015 at 10:04:54AM -0800, Brendan Conoboy wrote:
On 12/02/2015 08:05 AM, Toshio くらとみ wrote: [snip]
Do it when a RHEL releases with python3:
- sysadmins will have to manage both RHEL7 and RHEL8 web servers for py2 and py3.
- Concentrated work for developers to get applications ported as we'll want to migrate to rhel8 and python2 web apps will be holding us back.
- Packagers will have to branch (and sometimes maintain) needed packages from Fedora to EPEL.
RHEL currently provides python3 via SCL. Is this insufficient for infrastructure's needs?
Python3 itself isn't the problem, it's the whole stack of dependencies that we need in addition to python. Packaging things for EPEL means it's available to everyone and anyone can contribute/build upon it, we know how the system and the packaging work, it's easy for us to add new components and fix others. Packaging things for SCL means, it's nowhere but our infra repo since we can't push things to RH's SCLs and SCLs are neither in Fedora nor in EPEL, it's more work as we'll need to figure out how SCLs work and how packaging for SCLs work and it makes it a much more closed system from a contribution point of view.
Pierre
On 12/04/2015 10:26 AM, Pierre-Yves Chibon wrote:
On Fri, Dec 04, 2015 at 10:04:54AM -0800, Brendan Conoboy wrote:
On 12/02/2015 08:05 AM, Toshio くらとみ wrote: [snip]
Do it when a RHEL releases with python3:
- sysadmins will have to manage both RHEL7 and RHEL8 web servers for py2 and py3.
- Concentrated work for developers to get applications ported as we'll want to migrate to rhel8 and python2 web apps will be holding us back.
- Packagers will have to branch (and sometimes maintain) needed packages from Fedora to EPEL.
RHEL currently provides python3 via SCL. Is this insufficient for infrastructure's needs?
Python3 itself isn't the problem, it's the whole stack of dependencies that we need in addition to python. Packaging things for EPEL means it's available to everyone and anyone can contribute/build upon it, we know how the system and the packaging work, it's easy for us to add new components and fix others. Packaging things for SCL means, it's nowhere but our infra repo since we can't push things to RH's SCLs and SCLs are neither in Fedora nor in EPEL, it's more work as we'll need to figure out how SCLs work and how packaging for SCLs work and it makes it a much more closed system from a contribution point of view.
For context, my primary interest is there being a smooth course for developers and users to move from Python 2 to 3, preferably *before* a new RHEL major release comes out. If EPEL is meeting your needs that's great, but if you need something from RHEL that'd be good to know. Thanks,
On Fri, 4 Dec 2015 11:27:22 -0800 Brendan Conoboy blc@redhat.com wrote:
On 12/04/2015 10:26 AM, Pierre-Yves Chibon wrote:
On Fri, Dec 04, 2015 at 10:04:54AM -0800, Brendan Conoboy wrote:
RHEL currently provides python3 via SCL. Is this insufficient for infrastructure's needs?
Python3 itself isn't the problem, it's the whole stack of dependencies that we need in addition to python. Packaging things for EPEL means it's available to everyone and anyone can contribute/build upon it, we know how the system and the packaging work, it's easy for us to add new components and fix others. Packaging things for SCL means, it's nowhere but our infra repo since we can't push things to RH's SCLs and SCLs are neither in Fedora nor in EPEL, it's more work as we'll need to figure out how SCLs work and how packaging for SCLs work and it makes it a much more closed system from a contribution point of view.
For context, my primary interest is there being a smooth course for developers and users to move from Python 2 to 3, preferably *before* a new RHEL major release comes out. If EPEL is meeting your needs that's great, but if you need something from RHEL that'd be good to know. Thanks,
To expand on what Pierre said:
* SCL's aren't allowed in use in Fedora/EPEL, so we would have to not use koji or our usual patterns to maintaining things. Nor would it be as easy for us to collaborate with others who have similar goals.
* Mixing SCLs and RPMS is a mess and complex.
As to getting folks to move to python3 before the next RHEL major release, I am not sure. On one hand it would be great if there was a python3 rpm provided and maintained by RHEL folks. On the other hand if we maintain it in EPEL we might have more flexability to change it (ie, less guarentees than RHEL might provide). So, I guess the best world would be some RH/RHEL python folks helping maintain the EPEL python3x version and building up the stack somewhat would be best for us.
kevin
On Wed, Dec 02, 2015 at 12:14:42PM +0100, Pierre-Yves Chibon wrote:
Good morning everyone,
I would like to start gathering our thoughts about python3 in our apps.
To my knowledge, we currently have two applications that are python3 (only):
- Mailman3 core (as in hyperkitty is still py2)
- mdapi
Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi runs on a Fedora node for the moment.
What are they using instead of apache/mod_wsgi?
So, what do we think about python3 app in our infrastructure? Are we ok with it? Do we want to avoid them for the moment? Do we want to split 'backend' vs 'frontend' (ie web-apps)?
I'm definitely in favor of driving as much stuff towards py3 in our infra as possible, but I think we should try and run them the same way we do with all of other other apps (apache+mod_wsgi).
luke
On Wed, Dec 02, 2015 at 01:24:45PM -0700, Luke Macken wrote:
On Wed, Dec 02, 2015 at 12:14:42PM +0100, Pierre-Yves Chibon wrote:
Good morning everyone,
I would like to start gathering our thoughts about python3 in our apps.
To my knowledge, we currently have two applications that are python3 (only):
- Mailman3 core (as in hyperkitty is still py2)
- mdapi
Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi runs on a Fedora node for the moment.
What are they using instead of apache/mod_wsgi?
In both cases (Aurélien correct me if I'm wrong), they are just exposing an API, REST for MM3, JSON for mdapi. I don't know for MM3, but mdapi is async, which cannot be handled by apache afaik, we would need nginx or something if we wanted some sort of proxy. So at the moment mdapi runs with the web-server provided by aiohttp and is started via `systemctl start mdapi`.
What I wanted to highlight with this section is that despite having already 2 apps running python3, neither are "classic" nor following the traditional scheme of web-app/apache/mod_wsgi, which means when we deploy our first py3 apps under this scheme, we'll need to figure these things out, we can't rely on existing implementation.
Pierre
Dne 2.12.2015 v 12:14 Pierre-Yves Chibon napsal(a):
So what do you folks think?
Copr is already migrated to python3 (in upstream) and I'm getting ready to use python3 on production servers soon. But Copr (frontend and backend) is using Fedora it is no big deal anyway. Well we will migrate frontend to python3, backend have to be postponed since it is using ansible modules, which are python2 only. Anything new we recently wrote (e.g. keygen) is python3 only. And if possible run on RHEL7 (again case of keygen).
Conclusion for me - use common sense. Write code python2/3 compatible. Migrate if possible. If libraries are python2 only (openstack, ansible) notify they owners and wait until there is python3 version or if those authors refuse, choose other alternative.
On Dec 3, 2015 1:44 AM, "Miroslav Suchý" msuchy@redhat.com wrote:
Dne 2.12.2015 v 12:14 Pierre-Yves Chibon napsal(a):
So what do you folks think?
Copr is already migrated to python3 (in upstream) and I'm getting ready
to use python3 on production servers soon.
But Copr (frontend and backend) is using Fedora it is no big deal anyway. Well we will migrate frontend to python3, backend have to be postponed
since it is using ansible modules, which are
python2 only.
That shouldn't matter. You just need to have both python 2 and python 3 installed on the box. Ansible doesn't run under mod_wsgi so there is no conflict at runtime.
Unless you mean you have been using the ansible python module (rather than the python scripts ansible runs on the remote machines)? If so, two things.. (1) warning: 2.0 will be out soon and the python module api will be very different. We don't really support a stable api upstream in that sense.... just the apis for writing plugins and modules (and some plugin apis have changed in 2.0 as well. Ansible module api should be almost the same though.) (2) there's been some work on getting the controller code to be python 3 compatible but we're a ways off from that (ansible modules much further than that even). There's some issues about maintaining python 2.4 compat in the controller code we still know needs porting(because that code also runs on the remote hosts). Help welcomed upstream although we're currently in the talking phase about that code (I'm planning some improvements for 2.1 that will make this easier to manage).
Anything new we recently wrote (e.g. keygen) is python3 only. And if
possible run on RHEL7 (again case of keygen).
Sounds like you jumped the gun here since this discussion will lead to infra deciding if they want mixed python 3 and python 2 web application environments :-) but eventually this is inevitable. If not or own upstream, eventually third party upstream will start to support python 3 only. (As is the current case for maggit).
Conclusion for me - use common sense. Write code python2/3 compatible.
Migrate if possible. If libraries are python2
only (openstack, ansible) notify they owners and wait until there is
python3 version or if those authors refuse, choose
other alternative.
Here we're talking about the case of upstream that only support python 3 and whether we want to take on the maintenance burden (separate systems for running python2 and python 3 mod_wsgi apps, packaging work) that implies.
-Toshio
-Toshio
infrastructure@lists.fedoraproject.org