=========================
Numerical reproducibility
=========================

In general Xsuite does not not guarantee the numerical reproducibilty of the computation, in the sense that results obtained on different CPUs/GPUs or with different compilers will be different at the level of the machine precision.
This is mostly due to the fact that the underlying python libraries, and in particular numpy and scipy are not numerically portable. Xsuite compiled code is observed to be numerically portable, if compiled with the same set of compilers. CPU and GPU contexts are expected to give results that differ at the level of the machine precision.

We have identified a recipe that allows obtaining numerically reproducible results from Xsuite on CPU , which is reported in the following. Notably this requires compiling numpy and scipy in a special way, disabling vectorization optimizations and using unoptimized BLAS and LAPACK libraries (expect significant impact on numpy and scipy performance).
We underline that such a recipe is observed to yield numerically portable results in the analyzed cases of interest and on the CPUs that we had available but **is not guaranteed to do so in all possible cases**.
We cannot commit on keeping such a recipe in the future, as this depends on characteristics of underlying libraries that we do not control.

In general we suggest, whenever possible, to avoid relying on numerical portability in your workflow.

Using conda to create a numerically reproducible environment
============================================================

A numerical reproducible environment can be installed using the following instructions:

.. code-block:: bash

    # CLEAN
    export LD_LIBRARY_PATH=

    # Download, install and activate miniconda
    wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.10.3-Linux-x86_64.sh
    bash Miniconda3-py39_4.10.3-Linux-x86_64.sh
    source miniconda3/bin/activate

    # Install conda compiler packages
    conda install -c anaconda gcc_linux-64
    conda install -c anaconda gxx_linux-64
    conda install -c anaconda gfortran_linux-64

    # Install cython
    pip install cython

    # Install BLAS and LAPACK from netlib (avoid optimized versions like MKL or openblas)
    conda install -c conda-forge "libblas=*=*netlib" # pinning netlib, no fancy implementations
    conda install -c conda-forge "liblapack=*=*netlib" # pinning netlib, no fancy implementations

    # Install numpy using the libraries installed above and disabling vectorization on CPU
    git clone https://github.com/numpy/numpy.git --single-branch main
    cd main
    NPY_BLAS_ORDER=blas NPY_LAPACK_ORDER=lapack python setup.py build_ext --cpu-dispatch="none" --cpu-baseline="none" build_clib --cpu-dispatch="none" --cpu-baseline="none" install
    cd ..

    # Install scipy using the libraries installed above and disabling vectorization on CPU
    mkdir scipysrc
    cd scipysrc/
    git clone https://github.com/scipy/scipy --single-branch main
    cd main
    git submodule update --init
    pip install pythran
    pip install pybind11
    NPY_BLAS_ORDER=blas NPY_LAPACK_ORDER=lapack python setup.py build_ext --cpu-dispatch="none" --cpu-baseline="none" build_clib --cpu-dispatch="none" --cpu-baseline="none" install
    cd ..

    # Continue installation normally

    # pymask dependencies
    pip install ruamel.yaml
    pip install pandas matplotlib ipython
    pip install pyarrow
    conda install -c conda-forge orjson
    pip install cpymad

    # Install tree_maker
    git clone https://gitlab.cern.ch/abpcomputing/sandbox/tree_maker.git
    cd tree_maker
    python -m pip install -e .
    cd ..

    # Install Xsuite
    git clone https://github.com/xsuite/xobjects
    pip install -e xobjects

    git clone https://github.com/xsuite/xpart
    pip install -e xpart

    git clone https://github.com/xsuite/xtrack
    pip install -e xtrack

    git clone https://github.com/xsuite/xfields
    pip install -e xfields

Some extra information:

https://conda-forge.org/docs/maintainer/knowledge_base.html?highlight=mesa
https://numpy.org/devdocs/reference/simd/simd-optimizations.html
https://numpy.org/doc/stable/user/building.html