On Thu, 2017-02-02 at 11:34 -0800, Craig Rodrigues wrote:

...

Which in turn means 'pip3 install Django' actually used the system- wide pip3 command which tried to install Django to the system-wide location and failed.

After you create your Python 3 virtualev, what files are in the bin directory of your  Pythin 3 virtualenv?

[vagrant@tadej-zbook ~]$ ls -al myvenv3/bin/ total 20 drwxrwxr-x. 2 vagrant vagrant 4096 Feb  3 08:31 . drwxrwxr-x. 5 vagrant vagrant 4096 Feb  3 08:31 .. -rw-r--r--. 1 vagrant vagrant 2143 Feb  3 08:31 activate -rw-r--r--. 1 vagrant vagrant 1259 Feb  3 08:31 activate.csh -rw-r--r--. 1 vagrant vagrant 2397 Feb  3 08:31 activate.fish lrwxrwxrwx. 1 vagrant vagrant    7 Feb  3 08:31 python -> python3 lrwxrwxrwx. 1 vagrant vagrant   16 Feb  3 08:31 python3 -> /usr/bin/python3

Is there a pip file?

No.

Once you are inside your virtualenv and sourced the bin/activate script, my advice to you is to either:

• just run the command "pip"

Since there is no pip command in virtualenv's bin/ directory, pip is the global Python 2 version of pip:

[vagrant@tadej-zbook ~]$ pip --version pip 8.1.2 from /usr/lib/python2.7/site-packages (python 2.7)

• run the command "python -m pip"

Yay, this works:

(myvenv3) [vagrant@tadej-zbook ~]$ python -m pip install Django Collecting Django   Using cached Django-1.10.5-py2.py3-none-any.whl Installing collected packages: Django Successfully installed Django-1.10.5

So, this apparently boils down to:

1) If one uses "python3 -m venv --system-site-packages myvenv3", then no pip executable is installed in myvenv3/bin/. As a consequence, "pip install foo" fails since it tries to install foo in system's global site-packages directory. Using "python -m pip install foo" works as expected.

2) If one uses "python3 -m venv myvenv3-no-system", then pip (and pip3) executables are installed in myvenv3-no-system/bin/. Both, "pip install foo" and "python -m pip install foo" work as expected.

Is this something specific to Fedora's Python 3 packaging? If not, I'll post a question to the distutils mailing list.

Thanks, Tadej

Nick,

thanks for your thorough answer.

On Mon, 2017-02-06 at 20:07 +0100, Nick Coghlan wrote:

It's not specific to Fedora's Python 3 packaging as such, but it *is* specific to:

• using --system-site-packages
• having pip install in the system site packages

Yes, understood.

However, the way Fedora handles the native venv support in Python 3 means that this case will *always* apply when creating Python 3 virtual environments that can see the virtual environments - the pip that gets installed into the virtual environment comes from python-pip, rather than being a bundled copy in the Python RPM.

I'm still a bit puzzled by this part.

If I understand stdlib's venv documentation correctly [1], when one creates a virtual environment, it will use ensurepip to bootstrap pip into the virtual environment.

However, if pip is installed in system site packages, then ensurepip will skip the bootstrapping process, leaving the created virtual environment without its bundled copy of pip:

$ python3 -m venv --system-site-packages myvenv3 $ source myvenv3/bin/activate (myvenv3) $ python3 -m ensurepip  Ignoring indexes: https://pypi.python.org/simple Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python3.5/site-packages Requirement already satisfied (use --upgrade to upgrade): pip in /usr/lib/python3.5/site-packages (myvenv3) [vagrant@tadej-zbook ~]$ ls myvenv3/bin activate  activate.csh  activate.fish  python  python3

Do you think this is OK or should stdlib's venv module (or ensurepip) be changed to always create a bundled copy of pip in the virtual environment?

One big issue with not having pip in virtual environment's bin/ is that users will probably use 'pip install foo' inside the activated virtual environment and be surprised by not being able to install anything. Many of them might not know the 'python -m pip install foo' variant.

That means you have to pass "--ignore-installed" to get pip to actually install into the virtual environment properly:

(venv2) $ which pip     /usr/bin/pip     (venv2) $ python -m pip install --ignore-installed pip     Collecting pip      Using cached pip-9.0.1-py2.py3-none-any.whl     Installing collected packages: pip     (venv2) $ which pip     /tmp/venv2/bin/pip

Well, it is not always necessary :). If there would be a newer pip available on PyPI than the installed system site packages version, one could also get away with: (myvenv3) $ python3 -m pip install -U pip Collecting pip   Using cached pip-9.0.1-py2.py3-none-any.whl Installing collected packages: pip   Found existing installation: pip 8.1.2     Not uninstalling pip at /usr/lib/python3.5/site-packages, outside environment /home/vagrant/myvenv3 Successfully installed pip-9.0.1 (myvenv3) $ ls myvenv3/bin activate  activate.csh  activate.fish  pip  pip3  pip3.5  python  pytho n3

I think that makes sense at least as a BZ against the Fedora Python 3 package (to pass --ignore-installed when installing components into the venv), and it may even make sense to file it as a bug against CPython upstream as well, since it will happen any time you have the three way combination of:

• system wide installation of pip
• using native stdlib venv creation
• the created venv is configured to have access to the system site

packages

Agreed, I'll file appropriate bug reports, but first I would like to make the issues clearer in my head :).

As I see it, there are two issues when one has the aforementioned three way combination:

1) Users will probably use 'pip install foo' inside the activated virtual environment and be surprised by not being able to install anything. Many of them might not know the 'python -m pip install foo' variant.

2) Users that know the 'python -m pip install' variant may want to install a package inside the virtual environment that is already available in system site-packages . The installation won't work unless they pass the '--ignore-installed' option or they install a newer version of the same package with '--upgrade' option.

Does this make sense?

Thanks, Tadej

[1] https://docs.python.org/3/library/venv.html

On Feb 10, 2017, at 9:24 AM, Nick Coghlan ncoghlan@gmail.com wrote:

What's supposed to happen is that ensurepip installs a fresh copy into the virtual environment.

It *doesn't* happen under Python 3 on Fedora when system site-packages is visible, because if python3 is installed, then python3-pip will *also* be installed, and you get the behaviour you're seeing.

So this should be filed as a bug against the Fedora python package for doing the wrong thing by default.

However, it isn't clear that this qualifies as an upstream bug, as ensurepip in general is *supposed* to be a no-op when pip is already installed.

It’s not a Fedora bug really, it’s a venv bug, virtualenv has special logic to ensure pip actually gets installed with system site packages and when we Implemented that in venv I forgot to do it. It looks like it’s already been fixed upstream - https://bugs.python.org/issue24875 https://bugs.python.org/issue24875.

— Donald Stufft

On Fri, 2017-02-10 at 09:34 -0500, Donald Stufft wrote:

It’s not a Fedora bug really, it’s a venv bug, virtualenv has special logic to ensure pip actually gets installed with system site packages and when we Implemented that in venv I forgot to do it. It looks like it’s already been fixed upstream - https://bugs.python.org/issue24875.

Ooh, that's great, I'm very glad that this has already been fixed upstream. Donald, thanks for the pointer!

On Fri, 2017-02-10 at 12:09 -0500, Charalampos Stratakis wrote:

Since it is fixed upstream I'll backport the fix for rawhide and the stable Fedora's as soon as http://bugs.python.org/issue29523 for rawhide and the magic number issue for the stable ones are resolved.

Awesome! Charalampos thanks for your effort.

Regards, Tadej

On Fri, 2017-02-10 at 15:28 +0100, Nick Coghlan wrote:

On 8 February 2017 at 13:44, Tadej Janež tadej.j@nez.si wrote:

...

As I see it, there are two issues when one has the aforementioned three way combination:

1. Users will probably use 'pip install foo' inside the activated

virtual environment and be surprised by not being able to install anything. Many of them might not know the 'python -m pip install foo' variant.

Right, that's why I consider the fact that doesn't work by default a bug in Fedora's current approach.

Agreed. And I'm very glad that this has already been fixed upstream, so we just have to wait until this lands in Fedora.

...

2. Users that know the 'python -m pip install' variant may want to

install a package inside the virtual environment that is already available in system site-packages . The installation won't work unless they pass the '--ignore-installed' option or they install a newer version of the same package with '--upgrade' option.

This, on the other hand, is why having the system site-packages visible is discouraged in general - it really is more complicated than using an environment that only shares that standard library and the language runtime.

Agreed. This is a caveat with using the system site-packages and one has to accept such things if he needs a virtual environment with the system site-packages enabled.

Thanks again for clearing this up for me!

Regards, Tadej