MPI types with runtime size

[Hespian]

I just built a quick example of something that's possible with the C interface but (afaik) not with kamping. (Pretty much the same as the top k example by @niklas-uhl )

#include <iostream>
#include <mpi.h>
#include <vector>
#include <cassert>

class TopK {
public:
    explicit TopK(size_t k) : elements(k) {}
    int& operator[](size_t i) {
        return elements[i];
    }
    int const& operator[](size_t i) const {
        return elements[i];
    }
    int* data() {
      return elements.data();
    }
    size_t size() const {
        return elements.size();
    }

private:
    std::vector<int> elements;
};

std::ostream& operator<<(std::ostream& os, TopK const& top_k) {
    os << "TopK(";
    for (size_t i = 0; i < top_k.size(); ++i) {
        if (i != 0) {
            os << ", ";
        }
        os << top_k[i];
    }
    os << ")";
    return os;
}

TopK mpi_top_k(TopK& local_top_k, MPI_Comm comm) {
    int k = local_top_k.size();
    // create a custom datatype
    MPI_Datatype topK_type;
    MPI_Type_contiguous(local_top_k.size(), MPI_INT, &topK_type);
    MPI_Type_commit(&topK_type);

    // create a custom reduce operation
    MPI_Op             topK_merge_op;
    MPI_User_function* merge_op = [](void* invec, void* inoutvec, int* len, MPI_Datatype* mpi_type) {
        int* lhs    = static_cast<int*>(invec);
        int* rhs = static_cast<int*>(inoutvec);
        assert(*len == 1);
        int k;
        MPI_Type_size(*mpi_type, &k);
        k /= sizeof(int);
        
        size_t          lhs_current = 0;
        size_t          rhs_current = 0;
        std::vector<int> merged(k);
        for (size_t i = 0; i < k; i++) {
            if (lhs[lhs_current] < rhs[rhs_current]) {
                merged[i] = lhs[lhs_current];
                lhs_current++;
            } else {
                merged[i] = rhs[rhs_current];
                rhs_current++;
            }
        }

        for (size_t i = 0; i < k; i++) {
            rhs[i] = merged[i];
        }

    };
    MPI_Op_create(merge_op, true, &topK_merge_op);

    // the actual MPI call
    TopK global_top_k(k);
    MPI_Reduce(local_top_k.data(), global_top_k.data(), 1, topK_type, topK_merge_op, 0, comm);

    // cleanup
    MPI_Op_free(&topK_merge_op);
    MPI_Type_free(&topK_type);

    return global_top_k;
}

int main(int argc, char** argv) 
{
    MPI_Init(&argc, &argv);
    int k = 5;
    TopK  input(k);
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    for (size_t i = 0; i < k; ++i) {
        input[i] = rank + i * size;
    }
    std::cout << "[R" << rank << "] local_input=" << input << std::endl;

    TopK mpi_result = mpi_top_k(input, MPI_COMM_WORLD);
    if (rank == 0) {
        std::cout << "global_result_mpi=" << mpi_result << std::endl;
    }

    MPI_Finalize();
    return 0;
}

[niklas-uhl]

As we discussed during lunch break, the problem here is that topK_type may not be mapped to a C++ datatype. We therefore need to introduce some additional means of representing such MPI datatype. MPL introduces the concept of layouts and each wrapped MPI function may optionally receive such layout. This does not seem like the KaMPIng™ way.
We would introduce a ContiguousView<T>(size_t count), which we use internally to tell MPI how to access the data and which datatypes to create (the view may also encapsulate a custom MPI Datatype for reuse). We would have to provide a ContiguousViewElement which is the operand of a reduce operation, which encapsulates a single element of a view. With this in place, our example would look like the following:

std::vector<ValueType> kamping_top_k(std::vector<ValueType> const& local_top_k, Communicator const& comm) {
  using namespace kamping;
  auto result = comm.reduce(
    send_buf(contiguous_view(local_top_k, local_top_k.size())), 
    op(merge<ValueType>, commutative))
  .extract_recv_buffer();
}

where merge has the following signature:

template<typename ValueType>
ContiguousViewElement<ValueType> merge(ContiguousViewElement<ValueType> const& lhs, ContiguousViewElement<ValueType> const& rhs);

[Hespian]

I think we also talked about this being chainable? I.e. putting this into another view. The template parameter to contiguous_view seems to prohibit that.

[niklas-uhl]

Yes, I just wanted to give a simplified example to show the basic idea.