Developing Modin within a virtual environment

When developing Modin, it is a good idea to use a virtual environment. Doing so allows you to develop in multiple versions of python and separate the dependencies of Modin from the python libraries you have currently installed on your computer.

Setting up the virtual environment

To begin, we first make sure we have the virtualenv library so that we can create virtual environments with

pip install virtualenv

Then, we create a virtual environment in the Modin repo directory using

virtualenv venv

which creates a virtual environment folder called venv in the Modin directory. To create it using a different python version, simply do

virtualenv -p <PATH/TO/PYTHON/INTERPRETER> venv

Having created a virtual environment to develop Modin in, we install the requirements and Modin by running

pip install -e .

This runs the Modin installer to install both Modin and all of its dependencies. We include the -e flag to let python know that we will be editing the file and that python should use the edited versions and not the version when we ran this command.

Using Jupyter Notebooks with the virtual environment

Having created a virtual environment, we can also use Jupyter Notebooks with this virtual environment to provide a playground to hand test Modin.

To do this, we first install Jupyter Notebooks and IPython Kernels within the virtual environment as follows

pip install jupyter ipykernel

We need to install Jupyter Notebooks within the virtual environment so that you can launch Jupyter notebook within the virtual environment for the ray workers will use the virtual environment kernel instead of the default python kernel. IPython Kernels is to allow Jupyter Notebooks to recognize the virtual environment as a python kernel.

To link the virtual environment to Jupyter Notebooks, we simply run

ipython kernel install --user --name=modin_env

NOTE: You have to have also be within the virtual environment when launching the Jupyter Notebook instance for ray to find the right environment

1 Like