Differences between revisions 30 and 73 (spanning 43 versions)
Revision 30 as of 2014-06-20 14:53:06
Size: 5477
Editor: alders
Comment:
Revision 73 as of 2023-11-06 08:33:58
Size: 8419
Editor: stroth
Comment: Link to internal VS Code article
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#rev 2020-09-10 bonaccos

<<TableOfContents()>>
Line 3: Line 7:
We provide as many modules as possible that come with the current Debian GNU/Linux stable release. Nevertheless, that might not be enough for your needs since you may want to use the newest version of some module or one that is not part of Debian. Furthermore you might want to use a different version of Python. We provide some packages that come with the current Debian GNU/Linux stable release, but usually this is because they are dependencies of an installed software. For python we strongly recommend to build own python environments with the desired python versions and packages.
Line 5: Line 9:
The recommended way to install additional python modules is through {{{pip}}}. Unfortunately, {{{pip}}} does not allow users to install modules in the user context. That means that you will first have to install your own version of python in your home. From that moment on, you can install modules through {{{pip}}}. Our recommended way to install such environments is trough `conda`, especially if you want to build a tool or toolchain where the setup will possibly be published in a paper. Alternatively, building an environment via `pyenv` is possible.
Line 7: Line 11:
For just quickly trying out some python tool a local installation of `pip` is recommended.

== Installing your own python environment with Conda ==

For a detailed overview of `conda` please follow the [[Programming/Languages/Conda|Conda documentation]].
Line 10: Line 19:
{{{Pyenv}}} is a collection of tools that allow users to manage different versions of python. In the simplest case you will need it to simply get an installation of python in your user space. Using that custom python installation, you will then be able to install additional modules in a very comfortable way, since you can install them in the "system path" (which is then somewhere within your home). `pyenv` is a collection of tools that allow users to manage different versions of python. The simplest case is to install python in your user space. Using this custom python installation, you will be able to install additional packages in a comfortable way, since you can install them in the "system path" (which is then somewhere within your home).
Line 12: Line 21:
Here is a small howto for installing python 2.7.7 in your home: The documentation on `pyenv` can be found on its Github page at [[https://github.com/pyenv/pyenv|github.com/pyenv/pyenv]].
Line 14: Line 23:
 * Install pyenv:
 {{{
curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash}}}
Here is a small howto for installing python 3.9.1 in your home:
Line 18: Line 25:
 * Add the following three lines to your ~/.bashrc:
 {{{
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"}}}
 * Install pyenv: {{{#!highlight bash numbers=disable
curl https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer -o pyenv-installer
}}}
 Check what the script is doing and then execute it: {{{#!highlight bash numbers=disable
bash ./pyenv-installer
}}}
 You can remove the installer file afterwards.
 * Add the following lines to your `~/.profile` before sourcing `~/.bashrc`: {{{#!highlight bash numbers=disable
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
}}}
 * In the `~/.bashrc`: {{{#!highlight bash numbers=disable
eval "$(pyenv init -)"
}}}
 * If you want to pyenv-virtualenv automatically (in the `~/.bashrc`): {{{#!highlight bash numbers=disable
eval "$(pyenv virtualenv-init -)"
}}}
Line 24: Line 44:
 * Restart your shell so the path changes take effect:
 {{{
exec $SHELL}}}
 * You need a new login shell for all settings to take effect (when logged in on a Desktop environment logoff and login again)
Line 28: Line 46:
 * Install some python version, e.g. for python 2.7.7:
{{{
pyenv install 2.7.7
 * Install some python version, e.g. for python 3.9.1: {{{#!highlight bash numbers=disable
env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.1
Line 32: Line 49:
 Note, that settting of `PYTHON_CONFIGURE_OPTS="--enable-shared"` is needed if you need to link against the libpython shared library.
Line 33: Line 51:
 * Make sure that this new python version will be used when you run python. You only need to run this command once:
{{{
pyenv global 2.7.7}}}
 * Make sure that this new python version will be used when you run python. You only need to run this command once: {{{#!highlight bash numbers=disable
pyenv global 3.9.1}}}
Line 37: Line 54:
 * In order to update {{{pyenv}}} run:
{{{
 * In order to update `pyenv` run: {{{#!highlight bash numbers=disable
Line 42: Line 58:
 || Website of pyenv || https://github.com/yyuu/pyenv/ ||
 || Website of pyenv installer || https://github.com/yyuu/pyenv-installer ||
 || Website of pyenv || https://github.com/pyenv/pyenv ||
 || Website of pyenv installer || https://github.com/pyenv/pyenv-installer ||
Line 45: Line 61:
== Installation of a local pip ==
`pip` can be installed in a user's environment and work with the `python` version installed on the system. Every package will be installed for the user only in one location, there is no separation with virtual environments.<<BR>>
Line 46: Line 64:
== Installation of additional or newer modules with pip == The following commands set up a local `pip` in a location of your choice. As example `/scratch/$USER/local` is used. You may use a location of your choice, preferrably outside your $HOME as not to impact your quota.
Line 48: Line 66:
Once you installed your custom python with the explanations given above, you are ready to install additional or newer modules the easy way. As an example, you can just run
{{{
pip install numpy}}}
to install {{{numpy}}} within your custom python installation.

For advanced usage of {{{pip}}}, please consult the manuals: http://pip.readthedocs.org/en/latest/


== Installation of custom (non easy_install-able) Python modules in the home directory of a user ==

We are sometimes asked for newer version of Python modules. We do no longer build Python modules in SEPP as the requests for modules and their versions is too widespread to keep these modules maintainable.

On this page we will list some bash-snippets that install some often requested modules in a users home.

== numpy ==

{{{#!highlight bash
#!/bin/bash

VERSION_NUMPY=1.7.1
builddir="/scratch/${USER}/build/numpy"

mkdir -p ${builddir}

cd ${builddir}
wget --output-document=numpy-${VERSION_NUMPY}.tar.gz \
    http://sourceforge.net/projects/numpy/files/NumPy/${VERSION_NUMPY}/numpy-${VERSION_NUMPY}.tar.gz/download
tar -xvvzkf numpy-${VERSION_NUMPY}.tar.gz
cd numpy-${VERSION_NUMPY}
python setup.py build --fcompiler=gnu95
python setup.py install --user
{{{#!highlight bash numbers=disable
export PYTHONUSERBASE=/scratch/$USER/local
mkdir -p $PYTHONUSERBASE/bin
export PIP_USER=true
export PATH=$PYTHONUSERBASE/bin:$PATH
wget https://bootstrap.pypa.io/get-pip.py -O ~/.local/bin/get-pip.py
python3 ~/.local/bin/get-pip.py -vvv --user
}}}
Set default installations to the user's environment permanently (stored in `~/.config/pip/pip.conf`):
{{{#!highlight bash numbers=disable
pip config set install.user true
Line 81: Line 79:
== scipy == The exported environment variables will be lost after closing the shell. To enable local pip on demand, add the following function to your `.bashrc`: {{{#!highlight bash numbers=disable
function localpip {
    PYTHONUSERBASE=/scratch/$USER/local
    PATH=$PYTHONUSERBASE/bin:$PATH
    export PYTHONUSERBASE PATH
}
}}}
When you open a new shell, entering the command `localpip` will call the function and initialize your local pip installation.
Line 83: Line 88:
|| '''Depends on''' ||
|| numpy ||
=== pip cache ===
`pip` uses a cache which is by default stored under `~/.cache/pip` or `$XDG_CACHE_HOME/pip` if it is set to a non-default location. This cache tends to fill up quickly and should occasionally be cleared with
{{{#!highlight bash numbers=disable
pip cache purge
}}}
It is advisable to set the cache's location to the local scratch disk to avoid using up quota:
Line 86: Line 95:
{{{#!highlight bash
#!/bin/bash

VERSION_SCIPY=0.13.0b1
builddir="/scratch/${USER}/build/scipy"

mkdir -p ${builddir}

cd ${builddir}
wget --output-document=scipy-${VERSION_SCIPY}.tar.gz \
    http://downloads.sourceforge.net/project/scipy/scipy/${VERSION_SCIPY}/scipy-${VERSION_SCIPY}.tar.gz
tar -xvvzkf scipy-${VERSION_SCIPY}.tar.gz
cd scipy-${VERSION_SCIPY}
python setup.py build
python setup.py install --user
 1. Create a directory for the cache: {{{#!highlight bash numbers=disable
mkdir -p /scratch/$USER/pip_cache
}}}
 1. Temporarily set the environment variable to tell `pip` to use a different cache location: {{{#!highlight bash numbers=disable
export PIP_CACHE_DIR=/scratch/$USER/pip_cache/
}}}
 or store the location permanently (in `~/.config/pip/pip.conf`): {{{#!highlight bash numbers=disable
pip config set global.cache-dir /scratch/$USER/pip_cache
}}}
 1. Check if the cache location is correct: {{{#!highlight bash numbers=disable
pip cache info
Line 103: Line 108:
== matplotlib == === Installation of additional or newer packages with pip ===
Line 105: Line 110:
|| '''Depends on''' ||
|| numpy ||

{{{#!highlight bash
#!/bin/bash

VERSION_MATPLOTLIB=1.3.0
builddir="/scratch/${USER}/build/matplotlib"

mkdir -p ${builddir}

cd ${builddir}
wget --output-document=matplotlib-${VERSION_MATPLOTLIB}.tar.gz \
    http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-${VERSION_MATPLOTLIB}/matplotlib-${VERSION_MATPLOTLIB}.tar.gz/download
tar -xvvzkf matplotlib-${VERSION_MATPLOTLIB}.tar.gz
cd matplotlib-${VERSION_MATPLOTLIB}
python setup.py build
python setup.py install --user
Once you installed your custom python with the explanations given above, you are ready to install additional or newer packages the easy way. The usage of `pip` is very easy. The following command installs the package `numpy`.
{{{#!highlight bash numbers=disable
pip install numpy
}}}
while the next command would upgrade an existing installation of `numpy`
{{{#!highlight bash numbers=disable
pip install --upgrade numpy
Line 125: Line 119:
== nose ==

|| '''Depends on''' ||
|| numpy ||
|| scipy ||

{{{#!highlight bash
#!/bin/bash

VERSION_NOSE=1.0.0
builddir="/scratch/${USER}/build/nose"

mkdir -p ${builddir}

cd ${builddir}
wget http://somethingaboutorange.com/mrl/projects/nose/nose-${VERSION_NOSE}.tar.gz
tar -xvvzkf nose-${VERSION_NOSE}.tar.gz
cd nose-${VERSION_NOSE}
python setup.py build
python setup.py install --user
=== pip package management ===
Show installed packages:
{{{#!highlight bash numbers=disable
pip list --user
Line 147: Line 125:
Show installed packages with their dependencies:
{{{#!highlight bash numbers=disable
pip freeze user | cut -d '=' -f 1 | xargs -r pip show | grep -E '^(Name|Required-by):'
}}}

Show installed packages without dependencies:
{{{#!highlight bash numbers=disable
pip list --user --not-required --format freeze --exclude pip --exclude setuptools
}}}

Show outdated packages
{{{#!highlight bash numbers=disable
pip list --user --outdated
}}}

Update all outdated packages:
{{{#!highlight bash numbers=disable
pip list --user --outdated |
    awk '{if ($2 ~ /[0-9\.]+/) print $1}' |
    xargs -r pip install --user --upgrade
}}}

For advanced usage of `pip`, please consult the [[https://pip.pypa.io/en/stable/user_guide/|official user guide]].

== Installation of Python packages that are not available in the archives of pip ==

Here we provide some shell script snippets for installing frequently asked packages which cannot be installed through `pip`. These scripts just provide an example installation. You might have to adapt some paths in order to make the package work correctly with the version of python you are using (e.g. if you run your custom python provided through `pyenv`).
Line 148: Line 154:
{{{#!highlight bash {{{#!highlight bash numbers=disable
Line 176: Line 182:
== Debugging python code ==
To debug `python` code with [[FAQ/VS-Code_Server|Visual Studio Code]], Microsoft provides the [[https://microsoft.github.io/debug-adapter-protocol/|Debug Adapter Protocol (DAP)]] and a `python` implementation [[https://github.com/microsoft/debugpy|debug.py]].

For details consult the section [[https://code.visualstudio.com/docs/python/debugging|Python debugging in VS Code]] in the official VS Code manual

Python

We provide some packages that come with the current Debian GNU/Linux stable release, but usually this is because they are dependencies of an installed software. For python we strongly recommend to build own python environments with the desired python versions and packages.

Our recommended way to install such environments is trough conda, especially if you want to build a tool or toolchain where the setup will possibly be published in a paper. Alternatively, building an environment via pyenv is possible.

For just quickly trying out some python tool a local installation of pip is recommended.

Installing your own python environment with Conda

For a detailed overview of conda please follow the Conda documentation.

Installing your own python versions with pyenv

pyenv is a collection of tools that allow users to manage different versions of python. The simplest case is to install python in your user space. Using this custom python installation, you will be able to install additional packages in a comfortable way, since you can install them in the "system path" (which is then somewhere within your home).

The documentation on pyenv can be found on its Github page at github.com/pyenv/pyenv.

Here is a small howto for installing python 3.9.1 in your home:

  • Install pyenv:

    curl https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer -o pyenv-installer
    

    Check what the script is doing and then execute it:

    bash ./pyenv-installer
    
    You can remove the installer file afterwards.
  • Add the following lines to your ~/.profile before sourcing ~/.bashrc:

    export PYENV_ROOT="$HOME/.pyenv"
    export PATH="$PYENV_ROOT/bin:$PATH"
    eval "$(pyenv init --path)"
    
  • In the ~/.bashrc:

    eval "$(pyenv init -)"
    
  • If you want to pyenv-virtualenv automatically (in the ~/.bashrc):

    eval "$(pyenv virtualenv-init -)"
    
  • You need a new login shell for all settings to take effect (when logged in on a Desktop environment logoff and login again)
  • Install some python version, e.g. for python 3.9.1:

    env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.1
    pyenv rehash
    

    Note, that settting of PYTHON_CONFIGURE_OPTS="--enable-shared" is needed if you need to link against the libpython shared library.

  • Make sure that this new python version will be used when you run python. You only need to run this command once:

    pyenv global 3.9.1
    
  • In order to update pyenv run:

    pyenv update
    

Documentation of pyenv

Installation of a local pip

pip can be installed in a user's environment and work with the python version installed on the system. Every package will be installed for the user only in one location, there is no separation with virtual environments.

The following commands set up a local pip in a location of your choice. As example /scratch/$USER/local is used. You may use a location of your choice, preferrably outside your $HOME as not to impact your quota.

export PYTHONUSERBASE=/scratch/$USER/local
mkdir -p $PYTHONUSERBASE/bin
export PIP_USER=true
export PATH=$PYTHONUSERBASE/bin:$PATH
wget https://bootstrap.pypa.io/get-pip.py -O ~/.local/bin/get-pip.py
python3 ~/.local/bin/get-pip.py -vvv --user

Set default installations to the user's environment permanently (stored in ~/.config/pip/pip.conf):

pip config set install.user true

The exported environment variables will be lost after closing the shell. To enable local pip on demand, add the following function to your .bashrc:

function localpip {
    PYTHONUSERBASE=/scratch/$USER/local
    PATH=$PYTHONUSERBASE/bin:$PATH
    export PYTHONUSERBASE PATH
}

When you open a new shell, entering the command localpip will call the function and initialize your local pip installation.

pip cache

pip uses a cache which is by default stored under ~/.cache/pip or $XDG_CACHE_HOME/pip if it is set to a non-default location. This cache tends to fill up quickly and should occasionally be cleared with

pip cache purge

It is advisable to set the cache's location to the local scratch disk to avoid using up quota:

  1. Create a directory for the cache:

    mkdir -p /scratch/$USER/pip_cache
    
  2. Temporarily set the environment variable to tell pip to use a different cache location:

    export PIP_CACHE_DIR=/scratch/$USER/pip_cache/
    

    or store the location permanently (in ~/.config/pip/pip.conf):

    pip config set global.cache-dir /scratch/$USER/pip_cache
    
  3. Check if the cache location is correct:

    pip cache info
    

Installation of additional or newer packages with pip

Once you installed your custom python with the explanations given above, you are ready to install additional or newer packages the easy way. The usage of pip is very easy. The following command installs the package numpy.

pip install numpy

while the next command would upgrade an existing installation of numpy

pip install --upgrade numpy

pip package management

Show installed packages:

pip list --user 

Show installed packages with their dependencies:

pip freeze user | cut -d '=' -f 1 | xargs -r pip show | grep -E '^(Name|Required-by):'

Show installed packages without dependencies:

pip list --user --not-required --format freeze --exclude pip --exclude setuptools

Show outdated packages

pip list --user --outdated

Update all outdated packages:

pip list --user --outdated |
    awk '{if ($2 ~ /[0-9\.]+/) print $1}' |
    xargs -r pip install --user --upgrade

For advanced usage of pip, please consult the official user guide.

Installation of Python packages that are not available in the archives of pip

Here we provide some shell script snippets for installing frequently asked packages which cannot be installed through pip. These scripts just provide an example installation. You might have to adapt some paths in order to make the package work correctly with the version of python you are using (e.g. if you run your custom python provided through pyenv).

nlopt

#!/bin/bash

# Installation script for nlopt library

VERSION=2.3
INSTALLDIR=$HOME/.local
BUILDDIR=/scratch/$USER/nlopt

mkdir -p $BUILDDIR
cd $BUILDDIR

wget "http://ab-initio.mit.edu/nlopt/nlopt-${VERSION}.tar.gz"
tar -xvvzkf nlopt-${VERSION}.tar.gz
cd nlopt-${VERSION}

./configure \
        --enable-shared \
        --prefix=$INSTALLDIR \
        OCT_INSTALL_DIR=$INSTALLDIR/octave/oct \
        M_INSTALL_DIR=$INSTALLDIR/octave/m/ \
        MEX_INSTALL_DIR=$INSTALLDIR/mex \
        GUILE_INSTALL_DIR=$INSTALLDIR/guile

make
make install

Debugging python code

To debug python code with Visual Studio Code, Microsoft provides the Debug Adapter Protocol (DAP) and a python implementation debug.py.

For details consult the section Python debugging in VS Code in the official VS Code manual


CategoryLXSW

Programming/Languages/Python (last edited 2023-11-06 08:33:58 by stroth)