Understanding Python virtual environments using venv and virtualenv

Sukul Mahadik
Sukul Mahadik
·
Follow
13 min read
·
Aug 8, 2022

Listen

Share
References:
Book : Mastering Python
YouTube video by Corey Schafer : https://www.youtube.com/watch?v=Kg1Yvry_Ydk&t=3s
YouTube video by teclado: https://www.youtube.com/watch?v=KxvKCSwlUv8&list=PLiGsRbwqSNTOMsQxYzeMFEB7lMCE-u_qr&index=1
Medium Blog: https://python.plainenglish.io/how-to-generate-requirements-txt-for-your-python-project-235183799d2f
Medium Blog: https://medium.com/@rodrigosyscop/the-pyvenv-script-has-been-deprecated-as-of-python-3-6-2f8127456a85

What are virtual environments and why are they a good idea?
A Virtual environment is a light weight python installation with its own package directories and python binary (either copied or linked from the python environment used to create the virtual environment)
There are multiple ways to install and manage packages. Some of them are :
Download and extract code in our project directory.
Use OS package manager.
Use pip to install packages.
However to make sure packages do not collide with system packages and packages for other projects, its recommended we use virtual environments.
Lets first understand why installing packages globally using pip is a bad idea:
Security Risk: Installing packages globally requires elevated privileges and that's a security risk. When executing pip install <package>, the
setup.py of that package is executed as the user that executed the pip install command. That means that if the package contains malware, it now has superuser privileges to do whatever it wants.
Break system packages: Many system packages also use python. Installing new packages globally can mess with existing packages that are installed by your package manager.
Break other projects: We could be working with multiple projects simultaneously with each project having its own set of dependencies. IN-fact we may have multiple projects relying on different versions of the same package. Every global pip install could pull in new/updated dependencies that could break compatibility with other packages and projects.
Pollute list of packages, making it hard to keep track of your project’s dependencies.
One key advantage of using virtual environments is that we can specify the
Python version when creating the virtual environment, allowing us to test and debug our projects in multiple Python versions.
How to use venv and virtualenv to create and manage virtual environments?
Traditionally virtualenv has been the library used to create virtual environments for python. However , starting python 3.3 , module venv has been added to python standard library and can be used as a drop-in replacement for virtualenv. If older version of python is being used, then virtualenv is the way to go.
Apart from some small differences, venv and virtualenv work similarly.
A virtual environment is nothing but a directory that stores our python binary and dependencies. Its recommended that we keep all our environments in a single directory. Many people keep virtual environment in directory named env, .venv, or venv directory within the project, but its recommended to keep virtual environments separate from the project code.
However if we decide to keep our virtual environment inside our project directory, we should make sure that we add that directory to our .gitignore (or similar) for our version control system so that bulky libraries dont get uploaded to our version control repository. With correct dependency tracking, the virtual environment should be easy enough to rebuild.
Note that since Python 3.6, the pyvenv command has been deprecated in favor of python -m venv. (Do not confuse pyvenv with pyenv, which is used to quickly install and switch between multiple Python versions.)
Creating a virtual environment named virtenv-1: Note that here I plan to keep all my environments under a directory named env . Although not mandatory, I prefer this approach.
88665a380870:~ sukulma$ python3 -m venv envs/virtenv-1
Following shows the contents of the virtual environment. As we can see its just a directory.
88665a380870:~ sukulma$ cd envs/virtenv-1/

88665a380870:virtenv-1 sukulma$ ls -lrt
total 8
drwxr-xr-x 2 sukulma staff 64 Aug 7 10:36 include
drwxr-xr-x 3 sukulma staff 96 Aug 7 10:36 lib
-rw-r--r-- 1 sukulma staff 104 Aug 7 10:36 pyvenv.cfg
drwxr-xr-x 13 sukulma staff 416 Aug 7 10:36 bin
Under the bin directory we can see the python3 and pip3 executable. We can also see several activate scripts that can be used to use a activate a virtual environment.(later)
88665a380870:virtenv-1 sukulma$ ls -lrt bin
total 88
lrwxr-xr-x 1 sukulma staff 51 Aug 7 10:36 python3 -> /Library/Developer/CommandLineTools/usr/bin/python3
lrwxr-xr-x 1 sukulma staff 7 Aug 7 10:36 python -> python3
-rwxr-xr-x 1 sukulma staff 255 Aug 7 10:36 easy_install
-rwxr-xr-x 1 sukulma staff 255 Aug 7 10:36 easy_install-3.8
-rwxr-xr-x 1 sukulma staff 246 Aug 7 10:36 pip
-rwxr-xr-x 1 sukulma staff 246 Aug 7 10:36 pip3
-rwxr-xr-x 1 sukulma staff 246 Aug 7 10:36 pip3.8
-rw-r--r-- 1 sukulma staff 2402 Aug 7 10:36 activate.fish
-rw-r--r-- 1 sukulma staff 1250 Aug 7 10:36 activate.csh
-rw-r--r-- 1 sukulma staff 8834 Aug 7 10:36 Activate.ps1
-rw-r--r-- 1 sukulma staff 2198 Aug 7 10:36 activate
In pyvenv.cfg we can see some configuration settings for the virtual environment. The version is nothing but the python version used in this virtual environment (which is same as the python version used to create the virtual environment). The include-system-site-packages indicates whether system packages will be included in this environment.(more on this later in the blog)
88665a380870:virtenv-1 sukulma$ cat pyvenv.cfg
home = /Library/Developer/CommandLineTools/usr/bin
include-system-site-packages = false
version = 3.8.9
Following shows the dependencies included in this virtual environment. Note that a new virtual environment only has pip and setuptools.
88665a380870:virtenv-1 sukulma$ ls -lrt lib/python3.8/site-packages/
total 8
-rw-r--r-- 1 sukulma staff 126 Aug 7 10:36 easy_install.py
drwxr-xr-x 5 sukulma staff 160 Aug 7 10:36 pkg_resources
drwxr-xr-x 44 sukulma staff 1408 Aug 7 10:36 setuptools
drwxr-xr-x 12 sukulma staff 384 Aug 7 10:36 setuptools-49.2.1.dist-info
drwxr-xr-x 6 sukulma staff 192 Aug 7 10:36 pip
drwxr-xr-x 10 sukulma staff 320 Aug 7 10:36 pip-20.2.3.dist-info
Following is how we activate the virtual environment we created above.
88665a380870:~ sukulma$ pwd
/Users/sukulma
88665a380870:~ sukulma$ source envs/virtenv-1/bin/activate
(virtenv-1) 88665a380870:~ sukulma$
Following things happen when we activate an environment:
Commands such as python and pip use the environment-specific versions. So pip install only installs within our virtual environment. Below shows that once the virtual environment is activated, references to python and pip refer to the binaries in the virtual environment. Note that the directory Users/sukulma/envs/virtenv-1/bin is added first in the PATH environment variable. This is why any references to python and pip refer to the ones from our virtual environment.
An useful side effect of activating the environment is the update to the prompt to prefix with the name of our environment, which is (virtenv-1) in this case, indicating that we are working in a virtual environment.
The system site-packages folder is removed from the sys.path and the virtual environment site-packages gets appended to sys.path. The site-packages folder is nothing but a directory where 3rd party packages get installed. The sys.path contains the list of directories that python will search for when we import a 3rd party library/package. This works similar to system environment variable PATH.
Also note that we are not using sudo or other methods of elevating privileges.
(virtenv-1) 88665a380870:~ sukulma$ which python
/Users/sukulma/envs/virtenv-1/bin/python
(virtenv-1) 88665a380870:~ sukulma$ which pip
/Users/sukulma/envs/virtenv-1/bin/pip
(virtenv-1) 88665a380870:~ sukulma$ echo $PATH
/Users/sukulma/envs/virtenv-1/bin:/usr/local/opt/node@14/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/sukulma/.toolbox/bin
(virtenv-1) 88665a380870:~ sukulma$
Can you guess which version of python will be used in the virtual environment? — Yes, that’s right. its the same version of python used to create the virtual environment.
To deactivate a virtual environment, we simply use the deactivate command. Notice that the prompt virtenv-1 is no more. Also notice that once deactivated, the path /Users/sukulma/envs/virtenv-1/bin is no more part of the PATH.
(virtenv-1) 88665a380870:~ sukulma$
(virtenv-1) 88665a380870:~ sukulma$ deactivate
88665a380870:~ sukulma$
88665a380870:~ sukulma$ echo $PATH
/usr/local/opt/node@14/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/sukulma/.toolbox/bin
88665a380870:~ sukulma$
Following is how the sys.path changes before and after activating a virtual environment.
Before activating: Notice that sys.path includes the system site packages /Users/sukulma/Library/Python/3.8/lib/python/site-packages
88665a380870:envs sukulma$ python3
Python 3.8.9 (default, Apr 13 2022, 08:48:07)
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys

>>> for name in sys.path:
... print(name)
...
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload
/Users/sukulma/Library/Python/3.8/lib/python/site-packages
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages
After activating: Notice that the system site-packages is not more part of the sys.path, but virtual environment specific site-packages /Users/sukulma/envs/virtenv-1/lib/python3.8/site-packages is part of sys.path. This means that after activating a virtual environment when we import a 3rd party package, python looks for it in the virtual environment specific site-packages.
(virtenv-1) 88665a380870:~ sukulma$ python
Python 3.8.9 (default, Apr 13 2022, 08:48:07)
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for name in sys.path:
... print(name)
...
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload
/Users/sukulma/envs/virtenv-1/lib/python3.8/site-packages
Now lets see how we can do the same virtualenv
First lets install virtualenv if we dont already have.
88665a380870:~ sukulma$ pip3 install virtualenv
Defaulting to user installation because normal site-packages is not writeable
Collecting virtualenv
Downloading virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB)
|████████████████████████████████| 8.8 MB 2.5 MB/s
Collecting platformdirs<3,>=2.4
Downloading platformdirs-2.5.2-py3-none-any.whl (14 kB)
Collecting distlib<1,>=0.3.5
Downloading distlib-0.3.5-py2.py3-none-any.whl (466 kB)
|████████████████████████████████| 466 kB 19.5 MB/s
Collecting filelock<4,>=3.4.1
Downloading filelock-3.7.1-py3-none-any.whl (10 kB)
Installing collected packages: platformdirs, distlib, filelock, virtualenv
WARNING: The script virtualenv is installed in '/Users/sukulma/Library/Python/3.8/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed distlib-0.3.5 filelock-3.7.1 platformdirs-2.5.2 virtualenv-20.16.3
WARNING: You are using pip version 20.2.3; however, version 22.2.2 is available.
You should consider upgrading via the '/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip' command.
88665a380870:~ sukulma$ virtualenv envs/virtenv-2
-bash: virtualenv: command not found
88665a380870:~ sukulma$
Note that initially when I ran virtualenv , I ran into an issue. While I try to figure out the reason , following is workaround I used. Since both venv and virtualenv are python modules, we can use the python3 -m to execute a module. Only difference is that virtualenv is a 3rd party module that needs to be installed and venv is part of python standard library.
88665a380870:~ sukulma$ python3 -m virtualenv envs/virtenv-2
created virtual environment CPython3.8.9.final.0-64 in 1289ms
creator CPython3macOsFramework(dest=/Users/sukulma/envs/virtenv-2, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/sukulma/Library/Application Support/virtualenv)
added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
88665a380870:~ sukulma$
88665a380870:~ sukulma$ source envs/virtenv-2/bin/activate
(virtenv-2) 88665a380870:~ sukulma$
An additional advantage of using virtualenv instead of venv, in that case, is that you can specify the Python interpreter:
88665a380870:~ sukulma$ python3 -m virtualenv -p python3.8 envs/virtenv-2
created virtual environment CPython3.8.9.final.0-64 in 856ms
creator CPython3macOsFramework(dest=/Users/sukulma/envs/virtenv-2, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/sukulma/Library/Application Support/virtualenv)
added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Note that to use a python version, we first need to have it installed on our system. If we don’t have the python version to start with, we could get an error like below:
88665a380870:~ sukulma$ python3 -m virtualenv -p python3.3 envs/virtenv-2
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.3'
Whereas with the venv command, it uses the currently running Python installation, so we need to change it through the following invocation —
python3.8 -m venv envs/your_env
Removing a virtual environment is as simple as deleting the folder specific to the virtual environment. Remember a virtual environment is nothing more than a directory. Be careful not to add any project files inside the virtual environment directories. Environment should be something that can be easily discarded and rebuilt.
88665a380870:~ sukulma$ ls -lrt envs
total 0
drwxr-xr-x 6 sukulma staff 192 Aug 7 10:36 virtenv-1
drwxr-xr-x 7 sukulma staff 224 Aug 7 12:00 virtenv-2
88665a380870:~ sukulma$ rm -rf envs/virtenv-2
How to manage packages in virtual environment using pip?
Use pip list command to get a list of packages installed globally or in a virtual environment.
Following shows the result of invoking pip list outside of any virtual environments.
88665a380870:~ sukulma$ pip3 list
Package Version
------------ -------
distlib 0.3.5
filelock 3.7.1
pip 20.2.3
platformdirs 2.5.2
setuptools 49.2.1
six 1.15.0
virtualenv 20.16.3
wheel 0.36.2
and how we activate a virtual environment and run pip list again. Because our environment is isolated from the system, we only see the packages and dependencies that we have explicitly installed (which right now is nothing)
88665a380870:~ sukulma$ source envs/virtenv-1/bin/activate
(virtenv-1) 88665a380870:~ sukulma$ pip list
Package Version
---------- -------
pip 20.2.3
setuptools 49.2.1
2. Use pip install command to install the packages. Following installs requests module in our virtual environment. Notice that requests module along with its dependencies like urllib3 are installed. The pip package manager takes care of downloading transitive dependencies as well. Also note that we can specify multiple libraries separated by space on the same install command.