When developing Modin, it is a good idea to use a virtual environment. Doing so allows you to develop in multiple versions of python and separate the dependencies of Modin from the python libraries you have currently installed on your computer.
Setting up the virtual environment
To begin, we first make sure we have the
virtualenv library so that we can create virtual environments with
pip install virtualenv
Then, we create a virtual environment in the Modin repo directory using
cd <PATH/TO/MODIN/DIRECTORY> virtualenv venv
which creates a virtual environment folder called
venv in the Modin directory. To create it using a different python version, simply do
virtualenv -p <PATH/TO/PYTHON/INTERPRETER> venv
Having created a virtual environment to develop Modin in, we install the requirements and Modin by running
cd <PATH/TO/MODIN/DIRECTORY> pip install -e .
This runs the Modin installer to install both Modin and all of its dependencies. We include the
-e flag to let python know that we will be editing the file and that python should use the edited versions and not the version when we ran this command.
Using Jupyter Notebooks with the virtual environment
Having created a virtual environment, we can also use Jupyter Notebooks with this virtual environment to provide a playground to hand test Modin.
To do this, we first install Jupyter Notebooks and IPython Kernels within the virtual environment as follows
pip install jupyter ipykernel
We need to install Jupyter Notebooks within the virtual environment so that you can launch Jupyter notebook within the virtual environment for the ray workers will use the virtual environment kernel instead of the default python kernel. IPython Kernels is to allow Jupyter Notebooks to recognize the virtual environment as a python kernel.
To link the virtual environment to Jupyter Notebooks, we simply run
ipython kernel install --user --name=modin_env
NOTE: You have to have also be within the virtual environment when launching the Jupyter Notebook instance for ray to find the right environment