#rev 2020-09-10 bonaccos <> = Python = We provide some packages that come with the current Debian GNU/Linux stable release, but usually this is because they are dependencies of an installed software. For python we strongly recommend to build own python environments with the desired python versions and packages. Our recommended way to install such environments is trough `conda`, especially if you want to build a tool or toolchain where the setup will possibly be published in a paper. Alternatively, building an environment via `pyenv` is possible. For just quickly trying out some python tool a local installation of `pip` is recommended. == Installing your own python environment with Conda == For a detailed overview of `conda` please follow the [[Programming/Languages/Conda|Conda documentation]]. == Installing your own python versions with pyenv == `pyenv` is a collection of tools that allow users to manage different versions of python. The simplest case is to install python in your user space. Using this custom python installation, you will be able to install additional packages in a comfortable way, since you can install them in the "system path" (which is then somewhere within your home). The documentation on `pyenv` can be found on its Github page at [[https://github.com/pyenv/pyenv|github.com/pyenv/pyenv]]. Here is a small howto for installing python 3.9.1 in your home: * Install pyenv: {{{#!highlight bash numbers=disable curl https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer -o pyenv-installer }}} Check what the script is doing and then execute it: {{{#!highlight bash numbers=disable bash ./pyenv-installer }}} You can remove the installer file afterwards. * Add the following lines to your `~/.profile` before sourcing `~/.bashrc`: {{{#!highlight bash numbers=disable export PYENV_ROOT="$HOME/.pyenv" export PATH="$PYENV_ROOT/bin:$PATH" eval "$(pyenv init --path)" }}} * In the `~/.bashrc`: {{{#!highlight bash numbers=disable eval "$(pyenv init -)" }}} * If you want to pyenv-virtualenv automatically (in the `~/.bashrc`): {{{#!highlight bash numbers=disable eval "$(pyenv virtualenv-init -)" }}} * You need a new login shell for all settings to take effect (when logged in on a Desktop environment logoff and login again) * Install some python version, e.g. for python 3.9.1: {{{#!highlight bash numbers=disable env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.1 pyenv rehash}}} Note, that settting of `PYTHON_CONFIGURE_OPTS="--enable-shared"` is needed if you need to link against the libpython shared library. * Make sure that this new python version will be used when you run python. You only need to run this command once: {{{#!highlight bash numbers=disable pyenv global 3.9.1}}} * In order to update `pyenv` run: {{{#!highlight bash numbers=disable pyenv update}}} === Documentation of pyenv === || Website of pyenv || https://github.com/pyenv/pyenv || || Website of pyenv installer || https://github.com/pyenv/pyenv-installer || == Installation of a local pip == `pip` can be installed in a user's environment and work with the `python` version installed on the system. Every package will be installed for the user only in one location, there is no separation with virtual environments.<
> The following commands set up a local `pip` in a location of your choice. As example `/scratch/$USER/local` is used. You may use a location of your choice, preferrably outside your $HOME as not to impact your quota. {{{#!highlight bash numbers=disable export PYTHONUSERBASE=/scratch/$USER/local mkdir -p $PYTHONUSERBASE/bin export PIP_USER=true export PATH=$PYTHONUSERBASE/bin:$PATH wget https://bootstrap.pypa.io/get-pip.py -O ~/.local/bin/get-pip.py python3 ~/.local/bin/get-pip.py -vvv --user }}} Set default installations to the user's environment permanently (stored in `~/.config/pip/pip.conf`): {{{#!highlight bash numbers=disable pip config set install.user true }}} The exported environment variables will be lost after closing the shell. To enable local pip on demand, add the following function to your `.bashrc`: {{{#!highlight bash numbers=disable function localpip { PYTHONUSERBASE=/scratch/$USER/local PATH=$PYTHONUSERBASE/bin:$PATH export PYTHONUSERBASE PATH } }}} When you open a new shell, entering the command `localpip` will call the function and initialize your local pip installation. === pip cache === `pip` uses a cache which is by default stored under `~/.cache/pip` or `$XDG_CACHE_HOME/pip` if it is set to a non-default location. This cache tends to fill up quickly and should occasionally be cleared with {{{#!highlight bash numbers=disable pip cache purge }}} It is advisable to set the cache's location to the local scratch disk to avoid using up quota: 1. Create a directory for the cache: {{{#!highlight bash numbers=disable mkdir -p /scratch/$USER/pip_cache }}} 1. Temporarily set the environment variable to tell `pip` to use a different cache location: {{{#!highlight bash numbers=disable export PIP_CACHE_DIR=/scratch/$USER/pip_cache/ }}} or store the location permanently (in `~/.config/pip/pip.conf`): {{{#!highlight bash numbers=disable pip config set global.cache-dir /scratch/$USER/pip_cache }}} 1. Check if the cache location is correct: {{{#!highlight bash numbers=disable pip cache info }}} === Installation of additional or newer packages with pip === Once you installed your custom python with the explanations given above, you are ready to install additional or newer packages the easy way. The usage of `pip` is very easy. The following command installs the package `numpy`. {{{#!highlight bash numbers=disable pip install numpy }}} while the next command would upgrade an existing installation of `numpy` {{{#!highlight bash numbers=disable pip install --upgrade numpy }}} === pip package management === Show installed packages: {{{#!highlight bash numbers=disable pip list --user }}} Show installed packages with their dependencies: {{{#!highlight bash numbers=disable pip freeze user | cut -d '=' -f 1 | xargs -r pip show | grep -E '^(Name|Required-by):' }}} Show installed packages without dependencies: {{{#!highlight bash numbers=disable pip list --user --not-required --format freeze --exclude pip --exclude setuptools }}} Show outdated packages {{{#!highlight bash numbers=disable pip list --user --outdated }}} Update all outdated packages: {{{#!highlight bash numbers=disable pip list --user --outdated | awk '{if ($2 ~ /[0-9\.]+/) print $1}' | xargs -r pip install --user --upgrade }}} For advanced usage of `pip`, please consult the [[https://pip.pypa.io/en/stable/user_guide/|official user guide]]. == Installation of Python packages that are not available in the archives of pip == Here we provide some shell script snippets for installing frequently asked packages which cannot be installed through `pip`. These scripts just provide an example installation. You might have to adapt some paths in order to make the package work correctly with the version of python you are using (e.g. if you run your custom python provided through `pyenv`). === nlopt === {{{#!highlight bash numbers=disable #!/bin/bash # Installation script for nlopt library VERSION=2.3 INSTALLDIR=$HOME/.local BUILDDIR=/scratch/$USER/nlopt mkdir -p $BUILDDIR cd $BUILDDIR wget "http://ab-initio.mit.edu/nlopt/nlopt-${VERSION}.tar.gz" tar -xvvzkf nlopt-${VERSION}.tar.gz cd nlopt-${VERSION} ./configure \ --enable-shared \ --prefix=$INSTALLDIR \ OCT_INSTALL_DIR=$INSTALLDIR/octave/oct \ M_INSTALL_DIR=$INSTALLDIR/octave/m/ \ MEX_INSTALL_DIR=$INSTALLDIR/mex \ GUILE_INSTALL_DIR=$INSTALLDIR/guile make make install }}} == Debugging python code == To debug `python` code with [[FAQ/VS-Code_Server|Visual Studio Code]], Microsoft provides the [[https://microsoft.github.io/debug-adapter-protocol/|Debug Adapter Protocol (DAP)]] and a `python` implementation [[https://github.com/microsoft/debugpy|debug.py]]. For details consult the section [[https://code.visualstudio.com/docs/python/debugging|Python debugging in VS Code]] in the official VS Code manual ---- [[CategoryLXSW]]