Revision 2 as of 2019-05-10 18:20:13

Clear message

Set up a python development environment for data science

The following procedure shows how to set up a python development environment with the conda packet manager and install pytorch and tensorflow.

Install conda

To provide conda, the minimal anaconda distribution miniconda can be installed and configured for the D-ITET infrastructure with the following bash script:

#!/bin/bash

# Locations to store environments
LOCAL_SCRATCH="/scratch/${USER}"
NET_SCRATCH="/itet-stor/${USER}/net_scratch"

# Installer of choice for conda
CONDA_INSTALLER_URL='https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh'

# Unset pre-existing python paths
[[ -z ${PYTHONPATH} ]] || unset PYTHONPATH

# Downlad latest version of miniconda and install it
wget -O miniconda.sh "${CONDA_INSTALLER_URL}" \
    && chmod +x miniconda.sh \
    && ./miniconda.sh -b -p "${NET_SCRATCH}/conda" \
    && rm ./miniconda.sh

# Configure conda
eval "$(${NET_SCRATCH}/conda/bin/conda shell.bash hook)"
conda config --add pkgs_dirs "${NET_SCRATCH}/conda_pkgs" --system
conda config --add envs_dirs "${LOCAL_SCRATCH}/conda_envs" --system
conda config --add envs_dirs "${NET_SCRATCH}/conda_envs" --system
conda config --set auto_activate_base false
conda deactivate

# Show how to initialize conda
echo
echo 'Initialize conda immediately:'
echo "eval \"\$(${NET_SCRATCH}/conda/bin/conda shell.bash hook)\""
echo
echo 'Automatically initialize conda for furure shell sessions:'
echo "echo 'eval \"\$(${NET_SCRATCH}/conda/bin/conda shell.bash hook)\"' >> ${HOME}/.bashrc"

# Show how to remove conda
echo
echo 'Completely remove conda:'
echo "rm -r ${NET_SCRATCH}/conda ${NET_SCRATCH}/conda_pkgs ${NET_SCRATCH}/conda_envs ${LOCAL_SCRATCH}/conda_envs ${HOME}/.conda"

Save this script as install_conda.sh, make it executable with

chmod +x install_conda.sh

and execute the script by issuing

./install_conda

Choose your preferred method of initializing conda as recommended by the script.

Conda storage locations

The directories listed in the command for complete conda removal contain the following data:

/itet-stor/$USER/net_scratch/conda

The miniconda installation

/itet-stor/$USER/net_scratch/conda_pkgs

Downloaded packages

/itet-stor/$USER/net_scratch/conda_envs

Virtual environments on NAS

/scratch/$USER/conda_envs

Virtual environments on local disk

/home/$USER/.conda

Personal conda configuration

The purpose of this configuration is to store reproducible and space consuming data outside of your $HOME to prevent using up your quota.

Using Conda

conda allows to seperate installed software packages from each other by creating so-called environments. Using environments is best practice to generate deterministic and reproducible tools.

conda takes care of dependencies common to the packages it is asked to install. If two packages have a common dependency but define a differing range of version requirements of said dependency, conda chooses the highest common version number. This means the dependency installed in an environment with both packages together might have a lower version number than in environments separating both packages.

It is best practice to seperate packages in different environments if they don't need to interact.

For a complete guide to conda see the official documentation.

Common commands

Common commands to get you started are listed here:

The name of the default environment is base.

Installation examples

The following examples show how to install pytorch for GPU with CUDA toolkit 9, 10 and without CUDA toolkit for CPU, as well as tensorflow in the same three variants:

conda create --name pytcu9  pytorch torchvision cudatoolkit=9.0 --channel pytorch
conda create --name pytcu10 pytorch torchvision cudatoolkit=10.0 --channel pytorch
conda create --name pytcpu  pytorch-cpu torchvision-cpu --channel pytorch
conda create --name tencu9  tensorflow-gpu cudatoolkit=9.0
conda create --name tencu10 tensorflow-gpu cudatoolkit=10.0
conda create --name tencpu  tensorflow

Testing installations

Testing pytorch

To verify if your installation of pytorch is working, run the following python code:

from __future__ import print_function
import torch
x = torch.rand(5, 3)
print(x)

The output should look something like:

tensor([[0.4813, 0.8839, 0.1568],
        [0.0485, 0.9338, 0.1582],
        [0.1453, 0.5322, 0.8509],
        [0.2104, 0.4154, 0.9658],
        [0.6050, 0.9571, 0.3570]])

To verify if CUDA is available for pytorch, run the following code:

import torch
torch.cuda.is_available()

It should return True.

Testing TensorFlow

The following code prints information about your tensorflow installation:

import tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

Look for lines containing device: XLA_, they show which CPU/GPU devices are available.

A line containing cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version means the NVIDIA driver is not compatible with the CUDA toolkit (see below).

NVIDIA CUDA Toolkit

Which version of the CUDA toolkit is usable depends on the version of the NVIDIA driver installed on the machine you run your programs. The version can be checked by issuing the command nvidia-smi and looking for the number next to the text Driver Version.

The CUDA compatibility document by NVIDIA shows a dependency matrix matching driver and toolkit versions.