HPC usage with access to GPU tutorial

How to create your own conda environment and use jupyter on HPC with access to GPU.

Connect to cluster

Simply connect using SSH if you have an authorized account

> ssh hpc.igbmc.fr #connect to cluster via SSH

You will then have access to the HPC terminal.

Useful HPC commands

• module load something (conda, cellpose etc..) -> to load modules that require terminal commands
• squeue -> to know what jobs are running on the cluster at that moment

Create conda directory where to store environments

By default conda creates everything in the home/username/ directory which is limited to 10GB. The environment might grow larger than this in time, so it is reccomended to install them on space2.

1. Create a directory. For the imaging facility, we have already a directory that we use for this: space2/mic-photon/conda_cluster_leave_here.
2. Tell conda to use that folder for future environments:

> module load conda #we can use conda commands
> conda config --show #will show you the default paths
> ls -a #will show all files in the folder
> nano ./condarc #will open a text editor

3. We need to add the envs and pkgs paths to the conda settings inside the .condarc file

Create conda environment that you want

Now that conda knows where to store the environments, you can create new ones and install packages as normal, and they will be stored in the established folders.

> conda create -n environment-bla-bla python=some.version
> conda install -c anaconda jupyter #install jupyter notebook

Connect GPU before starting Jupyter

Because we moved the conda folders outside the default location, sometimes linux struggles to find a library path needed to detect the GPU drive. Run the line below pointing at the conda-env of interest to fix the problem. (Replace conda-env with your conda env name).

> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/conda/folder/conda-env/lib/

Access environment through JupyterHub

• If in the environment you are going to use you don’t have the package ipykernel, install it. From ssh terminal connection:

> conda env list # to see the list of installed packages
> conda install -c anaconda ipykernel # to install it if you don't have it

• Now you need to connect the ipykernel to the JupyterHub to be able to use the environment:

> /path/to/conda/folder/conda_envs/conda-env/bin/python -m \
    ipykernel install --user --name 'kernel-name' --display-name "display-name"

• Check the IT documentation for more info in French.
• Connect to JupyterHub and login. Select the cluster settings you want from the dropdown menu (GPU is you need one).
• You should now see the new kernel you installed in the home page of JupyterHub. Select it and it will create a new notebook using the conda environment you have selected!

Call jupyter to open a notebook in the environment from terminal

Instead of calling jupyter notebook ourselves we need to use a special command to indicate that the notebook should be opened with a cluster node with the requirements we want (GPU, memory etc..). To do this:

> module load sjupyter
> which jupyter-notebook #check that sjupyter points at the correct env
> sjupyter --mem 50G --cpus-per-task 12 -p gpu --gres gpu:1

All the sjupyter parameters above can be changed according to what cluster node you need for the task at hand.