Rationale

This small project aims at mocking MANTA (https://gitlab.com/manta-project/manta) to prepare works on MANTA to enable performance portability. While simplifying it as much as possible, it reproduces the general functioning of MANTA as well as its main data structure.

It may have two purposes: - It proposes a way to approach performance portability with Kokkos for manta. If this way ends up looking appropriate, it gives the base grounds to do it on MANTA. - It may be used to study the performances of this proposed approach combined to the data structures of MANTA.

Build

The program requires Kokkos.

Install Kokkos

A simple way to have a working kokkos is to use spack.

The following show how to install kokkos for openmp and cuda backends (it is assumed that you have a nvidia GPU).

First, for the cuda backend you need to know about the architecture of your GPU and cuda version supported.

For the cuda version supported, just type:

nvidia-smi

The cuda version should be in the top-right corner of the output. On my machine it is 12.2.

For the architecture (or "Compute Capability"), you should type:

nvidia-smi --query-gpu=compute_cap --format=csv,noheader

On my machine it gives 8.9.

At this time, I do not manage to have the program to compile and work when the kokkos installation deals with several backends (other than the 'serial'). I then install with spack several kokkos, each one with either the openmp or the cuda backend.

spack install kokkos%gcc@12 +cuda +wrapper cuda_arch=89 ^cuda@12.2+dev
spack install kokkos%gcc@12 +openmp

Note that for the cuda backend, the cuda_arch and cuda version must be carefully set accordingly to the values got with nividia-smi.

Build the program with CMake

In an empty build directory, with a clean environment.

Build for cuda backend:

spack load kokkos+cuda
cmake ${MINIMANTA_REPO} -DCMAKE_BUILD_TYPE=RelWithDebInfo -DKokkos_ENABLE_CUDA=ON
make

Build for openmp backend:

spack load kokkos+openmp
cmake ${MINIMANTA_REPO} -DCMAKE_BUILD_TYPE=RelWithDebInfo -DKokkos_ENABLE_OPENMP=ON
make

Run

The build creates in the build directory: - An executable sod which compute a sod shock tube problem (https://en.wikipedia.org/wiki/Sod_shock_tube) with a 1rst order finite-volume method (using a HLLC Riemann solver). - A file of input data input_data.json in which a few input data can be modified. The most important one is n which defines the spatial discretization (it is the number of cells defining the tube). - A python script for the post treatment plot.py. This is used to check the solution at the end of the calculation. The execution of sod generates an output file ouput.dat which contains the discretized solution at the final time. The script plot.py reads this file and create plots of the solution.

To run the executable with the cuda backend, once the appropriate Kokkos is loaded (spack load kokkos+cuda), just run in the build directory:

./sod
python plot.py

To run the executable with the openMP backend, once the appropriate Kokkos is loaded (spack load kokkos+openmp), it is suggested by openMP on my configuration to set OMP_PROC_BIND=spread and OMP_PLACES=threads, so (adapt the number of threads to your machine):

OMP_PROC_BIND=spread OMP_PLACES=threads OMP_NUM_THREADS=16 ./sod
python plot.py

The screen output gives a computation time. This time includes: - synchronize the initial conditions data to the device - the time loop - synchronize the data from the device to the host for writing the output

The initialization times and output writing times are not accounted.

Under the hood

The "clone on device" pattern

This prototype make uses of a "pattern" that I called "clone on device".

In manta one use c++ encapsulation of the raw data on which calculation are performed (e.g. the "Fields"). To cope with the genericity of the code, the internal functioning of these classes that the developer uses when developing a new functionality involves several indirections: the instance of a class holds a reference to an instance of another class (e.g. for the fields: the field classes hold a reference to the FieldFieldDataStore. When a value is queried from a field, the underlying FieldFieldDataStore is queried through its reference).

This is apriori problematic for GPU computation with kokkos: let assume that one has a class C1 which has as data member a reference to a class C2, and that this reference is used in a member function of the class C1. When an instance of the class C1 is copied on the GPU (it is used in a kokkos_lambda) and the member function is used, it crashes as the reference points to an instance of the class C2 still in the host memory.

A way to cope with this would be

1. to considerer that the data member of every class can be splitted into
   • "Heavy" data dynamically allocated. Typically at this time these data are managed by some std::vector
   • "Light" metadata
2. to replace every std::vector (the heavy data) by a kokkos::View, as a kokkos::View has a shallow copy mechanism
3. to replace every class the references to another class by copy of it. As only metadata (which are summed to be light) are copied, this should be not dramatically costly in memory and copying time.

What seems to me to be difficult with this approach is that as the metadata are copied everywhere, when for any reason these metadata are updated such a modification must be propagated everywhere (which occurs automatically when using references instead of copies). Then one must add another point: 4) keep consistency between all the copies of the classes

This seems to be very cumbersome. Another approach would be to have the instances of all the classes that are actually used in a GPU kernel constructed in the GPU memory (and not constructed on the CPU and then copied to the GPU as it would be with the first approach).

A difficulty with this approach is that one cannot use dynamic memory allocation in a GPU kernel. The dynamically allocated memory must be allocated with instructions (such as Kokkos::kokkos_malloc, or construction of Kokkos::View<DeviceSpace>) called by the CPU. A way to cope with that is to construct first an instance of a class on the CPU, and then, once the size of the data dynamically allocated involved known, to "clone" it in the GPU memory.

template<typename MemorySpace = HostSpace>
class MeshSet {
public:
  void destructClone()
  {
    MeshSet<DeviceSpace> *devicePtr = this->clone_;
    Kokkos::parallel_for("cleanup", 1, KOKKOS_LAMBDA(const int) { devicePtr->~MeshSet<DeviceSpace>(); });
    Kokkos::fence();
    Kokkos::kokkos_free<DeviceSpace>(devicePtr);
  }

  ~MeshSet()
    requires std::is_same_v<MemorySpace, HostSpace>
  {
    std::cout << "DTOR HOST" << std::endl;
    if (this->clone_) {
      destructClone();
    }
  }

  ~MeshSet()
    requires std::is_same_v<MemorySpace, DeviceSpace>
  {
    printf("DTOR DEVICE\n");
  }

  MeshSet(const std::vector<EntityHandle> &items)
    requires std::is_same_v<MemorySpace, HostSpace>
      : items_("items", items.size())
  {
    for (size_t i = 0; i < items.size(); ++i) {
      items_[i] = items[i];
    }
  }

  MeshSet<DeviceSpace> *cloneOnDevice()
    requires std::is_same_v<MemorySpace, HostSpace>
  {
    MeshSet<DeviceSpace> *devicePtr = (MeshSet<DeviceSpace> *)Kokkos::kokkos_malloc<DeviceSpace>(sizeof(MeshSet));

    auto deviceView    = Kokkos::create_mirror_view_and_copy(Kokkos::DefaultExecutionSpace(), this->items_);
    this->clonedItems_ = deviceView;

    Kokkos::parallel_for(
        "construct meshset", 1, KOKKOS_LAMBDA(const int) { new (devicePtr) MeshSet<DeviceSpace>(deviceView); });
    Kokkos::fence();
    this->clone_ = devicePtr;
    return this->clone_;
  }

  MeshSet<DeviceSpace> *getClone() {
    return this->clone_;
  }

  KOKKOS_FUNCTION
  MeshSet(Kokkos::View<EntityHandle *, MemorySpace> items) : items_(items) {}

  KOKKOS_INLINE_FUNCTION
  const EntityHandle &getItem(int i) const { return this->items_[i]; }

private:
  Kokkos::View<EntityHandle *, MemorySpace> items_;
  Kokkos::View<EntityHandle *, DeviceSpace>
      clonedItems_;    // this is used only for a host when a clone on the device is created
  MeshSet<DeviceSpace> *clone_ = nullptr;
};

With this, the lifetime of the "clone" is tied to the one of the CPU original instance: when this CPU instance is destroyed, the clone is destroyed to. When the "device" shares the same memory space as the host, there is actually a single allocation of the "heavy" data, as in this case the Kokkos::View in the clone points to the same data as the Kokkos::View in the original instance.

Relationship with MANTA

In this prototype, the following features are replicated while being simplified:

• Mesh: matches the manta::Mesh. The mesh in this prototype can only represent a 1D mesh. It provides the function getConnection to get the 2 adjacent points of a segment, and the 1 or 2 adjacent segment to a point. As it is very simple, it does not store any connectivities but give them on the fly.

As it does not hold Kokkos::View, it is not templated by the MEMORYSPACE, but conforms nevertheless to the "clone on device" pattern.

• MeshSet: matches the manta::MeshSet. It just handles a list of entities. It does not handle "localItemRank" with a MeshTag as manta::MeshSet does. It does not handle the nodes of the entities neither.

This class conforms to the "clone on device" pattern: its instance are first created an initialized on the host, and then "cloned" on the device (warning: here "cloned" does not means "memcopied". When "cloned", the constructor of the class is called on the device). It is templated by the MEMORYSPACE (which is HostSpace for the initial version on the host, and DeviceSpace for the clone).

• MeshTag: matches the manta::MeshTag.

It conforms to the "clone on device" pattern as the class MeshSet.

• Field: matches the manta::ElementField. It is constructed on a MeshSet as manta::ElementField. It just provides a getValue function matching the manta::ElementField::getValue which is the main function of the `manta::XXXField. As in MANTA, the field instances do not manage the data but rely on a FieldDataStore.

It conforms to the "clone on device" pattern as the class MeshSet.

• FieldDataStore: matches the manta::FieldDataStore. It only works if all the fields are defined on the same MeshSet.

It conforms to the "clone on device" pattern as the class MeshSet.

• ResourcesManager: matches the manta::ResourcesManger. As for MANTA the field must be declared before ResourcesManager::init and used after.

This class is only meant to be used as a factory to construct the other class on the host. Its instances only exist on the host.

• Integrand: this matches both manta::Integrand and manta::Assembler. There is no loop over integration points as only constant value per entities are considered. It (or its derived class) provides the functions
  • addOn: this matches manta::Integrand::AddOn function expect that it is called at the entity level instead of at the integration point level.
  • insertRHS: this matches manta::Assembler::insertRHS function. There is no insertLHS function here as for the actual problem considered (1st order explicit finite-volume problem) there is no need to consider global matrices.

The derived class of Integrand (which provide the addOn function) are required to exhibit a static getElementaryDataInfo function, which does not match any function of MANTA but is required to avoid data races in the buffers storing the elementary data.

The instance of the derived class of this class are only meant to exist on the device. It then does not conforms to the "clone on device" pattern, but the instances are nevertheless always constructed on the device (inside a "kernel").

• EquationTerm: this matches at the same time the manta::Formulation, manta::EquationTerm and manta::LinearSystem. Instead of having a manta::EquationTerm registered into a manta::Formulation registered into a manta::LinearSystem registered into a manta::ResourcesManger, here the EquationTerm is directly registered in the ResourcesManager. As the actual problem considered is an explicit finite-volume problem, there is no need for a linear system resolution and so the connection with PETSc is not handled in this prototype. Here it is the EquationTerm which provides a function assemble matching the Manta::LinearSystem::computeElementaryEntities. It also provides a function updateField which just transfer the elementary values computed to a field. It then somewhat substitutes for the Manta::LinearSystem::setUp, Manta::LinearSystem::solve and Manta::LinearSystem::updateField.

An EquationTerm is always constructed on the host only. Nevertheless: - It derives from EquationTermBase which holds some Kokkos::Views and conform to the "clone on device" pattern as the class MeshSet. Then the "base part" of an instance of an EquationTerm is cloned (and then constructed) on the device. - It is in charge of constructing/destructing the Integrand it relates to on the device. When constructed, it forward the argument to the constructor of its related Integrand (as in MANTA), but to ease the writing of the applicative code in the "main.cpp", for the arguments whose type conforms to the "clone on device" pattern, it automatically passes the clone of the argument instead of the host instance.