On 27 April 2017 at 11:47, Nico Kadel-Garcia nkadel@gmail.com wrote:

On Wed, Apr 26, 2017 at 7:13 AM, Charalampos Stratakis

...

At the present time, running sudo pip3 in Fedora is not safe. Pip shares its installation directory with dnf, can remove dnf-managed files and generally break the Python 3 interpreter.

This is true of every version of pip, not merely pip3. It was also true of CPAN, and of many gradle, maven, and ant working environments that semi-randomly collate the very latest versions of indeterminate components and spray them on top of your system workspace with their own quite distinct ideas about packaging and versioning.

There is no solution to this problem, except to use tools like "pyvenv" to set aside secluded workspaces for Python modules and their dependencies assembly. So, for most developers, they *should not* use "sudo pip". They should remain in user space, or possibly in shared workspaces, to avoid overwriting system components.

Nothing is changing in terms of *recommended* practices. This change proposal is entirely about harm mitigation for the cases where users *do* run "sudo pip ...", either because that's their instinctive reaction to a permissions error, or because some misguided installation instructions for a 3rd party package advised them to do it.

Debian and derivatives already mitigate the potential harm for these cases by requiring the "--install-layout=deb" option to be passed to get distutils to install into the system directories rather than doing it by default: https://wiki.debian.org/Python#Deviations_from_upstream

Their approach means that any harm caused by "sudo pip install X" can subsequently be fully reversed by doing "sudo pip uninstall X".

At this moment, this is NOT true for Fedora and derivatives. Instead, the remediation step here is "sudo pip uninstall X && sudo dnf reinstall <something>" where you have to:

1. Figure out what "<something>" needs to be 2. Hope that whatever you broke didn't affect your ability to run "sudo dnf reinstall"

...

Our first attempt to make sudo pip safe on Fedora [0] was based on using a different binary (/usr/libexec/system-python) for RPM packaging purposes and changing the behavior of /usr/bin/python3 (and pip that uses its settings) to install under /usr/local by default. Switching all the Python 3 packages

Oh, Oh my. The extent to which this unworkable is... fascinating.

Nico, the contemptuous tone you have adopted in this post is entirely inappropriate. Please give other list participants the courtesy of assuming expertise at least equivalent to your own, rather than engaging in attempted displays of assumed superiority.

You will inevitably install incompatible future versions of critical packages which are incompatible with the Fedora system packages. And as soon as you install one that breaks RPM, if /usr/local/ if ahead of /usr in your search path for modules, you are *screwed*.

This isn't true, as long as the fix is as simple as using pip to uninstall whatever problematic package you just installed.

At the moment, that isn't the case on Fedora and derivatives, since "sudo pip install ..." can actually *corrupt* a system package installation, rather than merely shadowing it at import time the way it does on Debian et al.

And if /usr/local is first, well, an RPM update may introduce a component that is incompatible with the pip installed modules at arbitrary times.

The goal of this change isn't to allow co-existence of arbitrary upstream and downstream versions as supported combinations - it's to make sure that forced installation of an incompatible upstream version is an easily reversible error, rather than a potentially system-breaking mistake.

...

We decided to try a different approach. The main idea is to detect an ongoing RPM build and modify the behavior of the Python 3 executable only when in normal user environment so that RPM builds won't be affected at all.

See https://xkcd.com/1172/ . You're inventing a workflow that isn't supported by *anyone*.

The problem is that the default behaviour needs to change (to mitigate the potential harm caused by a relatively common end user error), but unlike Debian package builds (which know they need to set "--install-layout=deb"), most RPM builds for Python packages currently rely on that default behaviour as part of the build process. Changing the default indiscriminately would thus be a breaking change that not only affects Fedora RPM builds, but also COPR and other third party RPM builds.

So we need a solution that retains the existing behaviour for RPM builds, but changes the behaviour to be less potentially destructive when run interactively at a shell prompt, or as part of a configuration management script.

...

The adjustment of the behavior can be done on different levels. The first option is to set the sys.prefix variable to /usr/local when the interpreter is initialized. This will affect all the install methods, but the solution can cause some problems in applications that rely on the value of sys.prefix. A prototype of this implementation can be seen here [1].

I'm sorry, but.... ouch. Please stop trying to implement "pip install" as a system installation tool.

That is not the goal of the change.

The effort would be better spent upgrading and enhancing "py2pack".

Improving the level of commonality and collaboration between Fedora & Mageia's pyp2rpm and openSUSE's py2pack would indeed be desirable, but it's entirely orthogonal to the harm mitigation exercise being discussed here.

...

The other possibility is to limit the pip install location change to distutils and pip [2]. This is the "safer" option, but does not cover all corner cases. For example, Python software built locally using cmake or similar tools will be installed into /usr/lib and can conflict with system tools. Debian chose to implement similar solution.

It's local environment specific, unstable software sensitive to arbitrary and unmanaged upstream dependency updates, and extremely sensitive to local system pip modules. It *does not belong* in /usr/lib. Because the components are modular and bundled in a non-RPM compatible fashion, it behooves developers to use a tool to segregate the tools as much as feasible from the Fedora underlying infrastructure. i.e., use pyvenv to build them in a contained environment segregated from the system files.

Helping users that are already doing the right thing isn't the problem at hand - this is about making it easier for users to recover from doing the *wrong* thing.

Cheers, Nick.