Python multi-version coexistence problem 2
This article discusses the coexistence and invocation of multiple versions of Python modules, and attempts to provide a refreshing solution.
Multiple versions of the Python module?
With Python becoming the main development language in data science, more and more machine learning libraries are releasing Python versions, including the famous scikit-learn, tensorflow, keras, etc. It is due to the rapid development of data science in recent years in terms of underlying algorithms and computational architectures that new versions of these Python modules have been released frequently.
The most disgusting thing about Tensorflow was that before the stable version 1.0.0 was released, almost every sub-version of Tensorflow had a different API. Of course this is mostly due to the endless array of algorithms in the deep learning space, on the other hand I personally suspect it is due to the Tensorflow code constantly making evolutions based on changes in google's computing platform.
The problem then becomes that the tf code for Seq2Seq, which you may have written yourself three months ago, no longer works in the new version of Tensorflow, which is caused by the API changes brought about by the version change. So some say, as long as I don't update TF all the time, won't it be fine?
The ideal is rich, but the reality is bleak!
How do I configure and use multiple versions of the mod?
A simple idea is to create, for each development task, a separate environment that has modules independent of the outside world. As soon as we get into this separate environment, we can install and use the mission-corresponding version of the module in it.
Fortunately, there are many ways to do this, such as docker. But here we don't use such a high level technique for now, we just need to use the virtualenv command.
1. Create a standalone Python environment with the virtualenv command
Installation of virtualenv
# Install virtualenv under Python 2.7
sudo pip install virtualenv
Create a separate Python environment with the virtualenv command
# Create a space for as many environments as I may create
mkdir ~/my_lib/py_env cd ~/my_lib/py_env
# Assuming that the system default Python is Python 2.7
# Create a standalone environment for Python 2.7
How do you get into that standalone Python environment?
Notice that by entering the standalone Python environment here you don't necessarily have to cd to the folder where the Python standalone environment was created in the previous step, you can start the environment almost anywhere.
A boot environment means that in this environment, commands such as cd, ls, etc. are no different from the outside world except for Python related commands, and only Python related commands such as pip, python, etc. are independent of the outside world.
The Python environment can be accessed anywhere
# In fact, the above command can be interpreted as adding ~/my_lib/py_env/py27_env/bin to the system path with the highest priority. However, the ~/my_lib/py_env/py27_env/bin folder only contains commands related to Python, so only the execution of Python related commands will be affected by this environment
How to exit the standalone Python environment
2. Install any version of the module in the corresponding standalone environment
Our task here is to create a Python 3.5 environment and then install the 1.0.1 gpu version of tensorflow using pip
Create the Python 3.5 environment first and activate it
# Create a Python 3.5 environment
# -p PYTHON_EXE_PATH argument, which can also be replaced with --python=PYTHON_EXE_PATH
# represents the system Python command from which the standalone environment is created, change the parameter to get a different version of Python standalone environment virtualenv -p /usr/bin/python3.5 ~/my_lib/py_env/py35_env
# Activate it source ~/my_lib/py_env/py35_env/bin/activate
Install tensorflow 1.0.1 gpu version
pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.0.1-cp34-cp34m-linux_x86_64.whl
# Here the command pip is replaced by pip3 or pip3.5, all three commands are in ~/my_lib/py_env/py35_env/bin/
# It's worth noting that you can run pip2 here, which is the system command, /usr/bin/pip2. You can use which pip2 to verify which pip2 which pip which pip3 which pip3.5
You can go into the Python interpreter and see
# The following three commands all access the standalone Python 3.5 environment python python3 python3.5
After entering the interpreter
import tensorflow as tf# See where this tensorflow is installed print(tf.__path__)
More options for virtualenv
The system Python environment here is the -p parameter that was used to create the standalone environment
1) The --no-site-packages option means that all third-party packages installed into the system Python environment will not be copied over, currently I see this implemented by default
2) The --system-site-packages option means that the virtual environment can access third-party packages in the system Python environment
3) More options can be viewed directly from the shell by typing virtualenv