Currently, all operations that take a count argument assert if the value is larger than INT32_MAX then downcast to int.

All such routines should be extended to support count up to INT64_MAX using MPI contiguous datatypes using the method demonstrated in https://github.com/jeffhammond/HPCGloves/blob/master/examples/mpi64/type_contiguous_x.c.

Currently, SMP optimizations only works when MPI_COMM_WORLD is an SMP. This is easy enough to fix and should be implemented.

The source TODO comment describes this issue succinctly.

/* 8.3: Accessibility Query Routines */
int shmem_pe_accessible(int pe) 
{ 
    /* TODO: detect MPMD launching, i.e. if PE is running same binary as me.
     *       MPI_APPNUM attribute of MPI_COMM_WORLD (MPI-3 10.5.3) is the way.
     *       Create a window containing these values so that any PE can Get it
     *       and compare against the local value. */
    return ( 0<=pe && pe<=shmem_world_size ); 
} 

I have a barrier microbenchmark that scales the number of PEs from 2 to NPES. When OSHMPI is built with --enable-comm-caching, this benchmark always breaks (deadlock) at PE_Size=18 (after performing PE_Size={2..17} correctly).

I've reproduced this bug with Intel MPI and MPICH on two different systems including multi-node and SMP configurations.

This bug is not exhibited when OSHMPI is built without --enable-comm-caching. It also may not repro if NPES=18. NPES=18 and PE_Size=18 may convert a shmem_barrier() to a shmem_barrier_all() and not repro this bug. For those implementations, it has to be at least NPES=19.

It calls MPI unconditionally right now. Optimize for shared memory and use atomic load when possible.

OSHMPI collectives do not require symmetric variables. This permits users to run incorrect code with OSHMPI, only to fail with other implementations. OSHMPI should check the arguments and emit a warning message when OpenSHMEM semantics are violated.

From Jim Dinan:

Initial report:

It looks like this implementation of wait_until violates the MPI spec with regard to polling on a location in memory. Am I missing something, or are you relying on the MPI implementation supporting this as an extension?

Elaboration:

I’m thinking of rule #2 on page 456:

1. Updates from a remote process will appear in the memory of the target, but there are no atomicity or ordering guarantees if more than one byte is updated. Updates are stable in the sense that once data appears in memory of the target, the data remains until replaced by another update. This permits polling on a location for a change from zero to non-zero or for a particular value, but not polling and comparing the relative magnitude of values. Users are cautioned that polling on one memory location and then accessing a different memory location has defined behavior only if the other rules given here and in this chapter are followed.

This effectively allows you to poll on only a byte. If you are expecting bytes to all show up at once, then you’re relying on an extension. For instance, assume a user asks to wait until the value of an integer is 1. Let’s say the user puts 257 = 256 + 1 = 0x 01 01 and least significant byte arrives first. The buffer will temporarily contain 1, even though the integer value should be 257.

Since we are not redistributing the entire OMB, we need to inline the full license in such source files. Otherwise, we should remove them from OSHMPI and just provide a pointer to the OSU tests of interest in test/README (for example).

/*
 * Copyright (C) 2002-2013 the Network-Based Computing Laboratory
 * (NBCL), The Ohio State University. 
 *
 * Contact: Dr. D. K. Panda ([email protected])
 *
 * For detailed copyright and licensing information, please refer to the
 * copyright file COPYRIGHT in the top level OMB directory.
 */

We convert put and get to load-store within an SMP (when SMP optimizations are active) but not atomics. Using GCC intrinsics (or OPA) will improve the performance of SHMEM atomics.

Profiling interface was requested by a user...