Numerical reproducibility

In general Xsuite does not not guarantee the numerical reproducibilty of the computation, in the sense that results obtained on different CPUs/GPUs or with different compilers will be different at the level of the machine precision. This is mostly due to the fact that the underlying python libraries, and in particular numpy and scipy are not numerically portable. Xsuite compiled code is observed to be numerically portable, if compiled with the same set of compilers. CPU and GPU contexts are expected to give results that differ at the level of the machine precision.

We have identified a recipe that allows obtaining numerically reproducible results from Xsuite on CPU , which is reported in the following. Notably this requires compiling numpy and scipy in a special way, disabling vectorization optimizations and using unoptimized BLAS and LAPACK libraries (expect significant impact on numpy and scipy performance). We underline that such a recipe is observed to yield numerically portable results in the analyzed cases of interest and on the CPUs that we had available but is not guaranteed to do so in all possible cases. We cannot commit on keeping such a recipe in the future, as this depends on characteristics of underlying libraries that we do not control.

In general we suggest, whenever possible, to avoid relying on numerical portability in your workflow.

Using conda to create a numerically reproducible environment

A numerical reproducible environment can be installed using the following instructions:

# CLEAN
export LD_LIBRARY_PATH=

# Download, install and activate miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.10.3-Linux-x86_64.sh
bash Miniconda3-py39_4.10.3-Linux-x86_64.sh
source miniconda3/bin/activate

# Install conda compiler packages
conda install -c anaconda gcc_linux-64
conda install -c anaconda gxx_linux-64
conda install -c anaconda gfortran_linux-64

# Install cython
pip install cython

# Install BLAS and LAPACK from netlib (avoid optimized versions like MKL or openblas)
conda install -c conda-forge "libblas=*=*netlib" # pinning netlib, no fancy implementations
conda install -c conda-forge "liblapack=*=*netlib" # pinning netlib, no fancy implementations

# Install numpy using the libraries installed above and disabling vectorization on CPU
git clone https://github.com/numpy/numpy.git --single-branch main
cd main
NPY_BLAS_ORDER=blas NPY_LAPACK_ORDER=lapack python setup.py build_ext --cpu-dispatch="none" --cpu-baseline="none" build_clib --cpu-dispatch="none" --cpu-baseline="none" install
cd ..

# Install scipy using the libraries installed above and disabling vectorization on CPU
mkdir scipysrc
cd scipysrc/
git clone https://github.com/scipy/scipy --single-branch main
cd main
git submodule update --init
pip install pythran
pip install pybind11
NPY_BLAS_ORDER=blas NPY_LAPACK_ORDER=lapack python setup.py build_ext --cpu-dispatch="none" --cpu-baseline="none" build_clib --cpu-dispatch="none" --cpu-baseline="none" install
cd ..

# Continue installation normally

# pymask dependencies
pip install ruamel.yaml
pip install pandas matplotlib ipython
pip install pyarrow
conda install -c conda-forge orjson
pip install cpymad

# Install tree_maker
git clone https://gitlab.cern.ch/abpcomputing/sandbox/tree_maker.git
cd tree_maker
python -m pip install -e .
cd ..

# Install Xsuite
git clone https://github.com/xsuite/xobjects
pip install -e xobjects

git clone https://github.com/xsuite/xpart
pip install -e xpart

git clone https://github.com/xsuite/xtrack
pip install -e xtrack

git clone https://github.com/xsuite/xfields
pip install -e xfields

Some extra information:

https://conda-forge.org/docs/maintainer/knowledge_base.html?highlight=mesa https://numpy.org/devdocs/reference/simd/simd-optimizations.html https://numpy.org/doc/stable/user/building.html