Coarray Teams, Events, Collectives and Atomics

The multi-image features are the part of Fortran that compilers support least, so this page is explicit about it. Every example was compiled with GNU Fortran 16.2 as gfortran -fcoarray=lib -std=f2023 -Wall -Wextra -pedantic -fcheck=all file.f90 -lcaf_shmem (the library built with GCC 16, shared-memory images in separate processes; the number of images is set by the environment variable GFORTRAN_NUM_IMAGES, here 4) with no warning and run for real, except where an example is labelled as rejected. The same sources were also tried with OpenCoarrays 2.10.3 (caf and cafrun, built on MPICH) and with LLVM Flang 23. The result, per feature:

Flang says, by default, "Multi-image features are experimental and are disabled by default, use '-fcoarray' to enable". With -fcoarray it prints a warning that multi-image support "is still experimental and in development" and then stops at the first construct it cannot lower. The diagnostics are quoted where each feature is discussed. Without a -fcoarray option gfortran refuses coarray syntax (Fatal Error: Coarrays disabled at (1), use '-fcoarray=' to enable), and with -fcoarray=single it accepts the syntax but there is only one image: the event handshake below stops at its own error stop 'needs two images'. Multi-image runs need -fcoarray=lib and a library, here libcaf_shmem.

A collective subroutine (16.6) computes one value from the contributions of all the images of the current team, without the program having to synchronize them by hand. Every image of the team calls it, in unordered segments (a collective is not an image control statement), and the calls correspond: the same collective, in the same order, with arguments of the same type, kind and shape. The five standard collectives are:

The common rules:

The program below does the collective work of one heatflow time step: the hottest and coldest temperatures, the total heat, the value of largest magnitude by a user-defined reduction, and the configuration read on image 1 and broadcast to the others. The reduction operation is in a module, which keeps the example free of the nested-procedure trampoline that an internal procedure passed as an actual argument can require.

Output (4 images):

Source: J3/23-007r1: 16.6 Collective subroutines, 16.9.54 CO_BROADCAST to 16.9.58 CO_SUM

The same program, built with caf and run with cafrun -n 4, printed the same lines. Flang 23 stops on co_reduce: with -fcoarray it reports not yet implemented: coarray: intrinsic co_reduce, and without it the first coarray-related statement is refused as above.

A team is a subset of the images of the program that acts, for a stretch of the execution, as if it were the whole program (Annex C.1 describes it as letting a subset of the images act together as if it were the set of all images). The images at the start form the initial team. A team formed from the current team by form team is a child of it; the current team is its parent; teams with the same parent are siblings. Each team has a team number: positive for a child team, and -1 for the initial team (3.142.5 and 16.9.207). Inside a team, the images are numbered 1 to the number of images in the team.

The pieces:

form team and the change team statement must be executed by all active images of the team concerned, the same number of times. A coarray established before change team stays accessible inside it; its cosubscripts then count images of the current team. A named construct entity can re-declare the codimensions with coarray-association-list (codimension-decl => selector), which works as in associate. A child team can allocate and deallocate allocatable coarrays collectively among its own images. An allocatable coarray that was unallocated when the construct started and is allocated at end team is deallocated by it (11.1.5.2).

The example splits four images into a left and a right half, which is how a plate would be handed to two groups, and then makes each half compute its own total with a plain co_sum:

Output (4 images):

Source: J3/23-007r1: 11.1.5 CHANGE TEAM construct, 11.7.6 SYNC TEAM statement, 11.7.9 FORM TEAM statement, 16.9.95 GET_TEAM, 16.9.207 TEAM_NUMBER

The two sums are for images 1 and 2 (1 + 2 = 3) and images 3 and 4 (3 + 4 = 7): the co_sum inside change team never saw the other half. The statement end team restored the initial team, in which team_number() is -1 again. Three details:

An event is a counter on one image that other images increment, with the waiting image able to block until the count is high enough. It is the Fortran way to say "this data is ready" without a global barrier, and it orders segments in the same way as sync images (11.7.8: the segments before k posts precede the segments after the wait that satisfies them).

Because event wait subtracts the threshold, the count is the number of posts not yet consumed: ten posts and two plain waits leave a count of eight (the example in 16.9.82). A count is only a snapshot when other images may still post.

Segments are ordered by an event: an image that wrote data, then posted, precedes the receiver’s segment that follows the wait. This is the guarantee that makes the typical hand-off safe, with no sync memory or barrier needed. The sequence below is the handshake the next program performs; it also shows an acknowledgement in the opposite direction, which the producer needs before it may reuse the buffer.

Output (4 images, images 3 and 4 do nothing):

Source: J3/23-007r1: 11.7.7 EVENT POST statement, 11.7.8 EVENT WAIT statement, 16.9.82 EVENT_QUERY, 16.10.2.10 EVENT_TYPE

The data is written before the post, and the consumer reads it after its wait, so it sees 1.0 2.0 3.0 4.0. The count is back to zero after the wait. OpenCoarrays 2.10.3 printed the same two lines; Flang 23 with -fcoarray stopped with not yet implemented: coarray: lowering a reference to a coarray object.

until_count gathers several posts with one wait, which is how a leader learns that a whole group is ready:

Output (4 images):

Source: J3/23-007r1: 11.7.8 EVENT WAIT statement, 16.9.82 EVENT_QUERY

Notes:

A put followed by an event post is two operations: the data transfer and the signal. Fortran 2023 fuses them: an assignment to a coindexed variable can carry notify= in its image selector, which atomically increments the notify count of a notify_type variable on the target image, and the receiver blocks with notify wait. Introduced in Fortran 2023 (Introduction of J3/23-007r1), described in 11.6, in 9.6 (C933: the notify= specifier may appear only in the designator of the variable of an intrinsic assignment statement) and in 16.10.2.22 (notify_type, whose count behaves like an atomic_int_kind integer updated as if by atomic_add).

The intended form of the handshake of the previous section, for the data direction only, is:

Output of gfortran -fcoarray=lib -std=f2023 (GNU Fortran 16.2): rejected, because neither the type nor the syntax exists in this release:

Source: J3/23-007r1: 9.6 Image selectors (C933), 11.6 NOTIFY WAIT statement, 16.10.2 ISO_FORTRAN_ENV (NOTIFY_TYPE)

This example was not run: the text is checked against the standard (11.6, 9.6 and 16.10.2.22), not against a compiler. None of the compilers available for this page accepts it. Intel ifx, NAG and the other compilers were not available, so nothing is claimed about them. Until your compiler supports it, use the event handshake above: event post after the put, which orders the same two segments.

An atomic subroutine (16.5) performs an action on its atom argument indivisibly. Any two atomic actions on the same object in unordered segments behave as if one finished before the other began, in a processor-dependent order, without needing image control statements. They are the building block for custom synchronization (counters, flags, tickets). The atom is a scalar coarray or coindexed object of type integer of kind atomic_int_kind or logical of kind atomic_logical_kind (both from iso_fortran_env, 16.10.2.2 and 16.10.2.3). Only atom is accessed atomically; the other arguments are ordinary.

atomic_define and atomic_ref are older (their stat argument is among the Fortran 2018 additions from TS 18508, Annex C.1); atomic_add and the rest of the family arrived in Fortran 2018 (Annex C.1). stat is zero on success; stat_failed_image if atom is on a failed image; otherwise a processor-dependent positive value; without it an error terminates the program (16.5).

Atomics synchronize nothing by themselves: an atomic_define of the initial value on one image and an atomic_add on another, in unordered segments, are in an undetermined order. The rules of the standard (16.5) are the ones that matter: atomic operations shall make asynchronous progress (a loop that polls with atomic_ref will eventually see the value), and the sequence of changes seen by another image can appear in a different order from the one the defining image performed. The program below does what the named plain coarray accesses cannot: a shared counter, a ticket dispenser and an election, all by atomic_* on image 1, followed by a flag polled by the last image.

Output (4 images):

Source: J3/23-007r1: 16.5 Atomic subroutines, 16.9.28 ATOMIC_ADD to 16.9.38 ATOMIC_XOR, 16.10.2.2 ATOMIC_INT_KIND

Four images each added 10 (40), took a different ticket (the check prints T when every number from 0 to 3 was handed out exactly once; which image got which number differs from run to run), and one won the election. The tickets are published with ordinary assignments to distinct elements of a coarray, and sync all orders them before they are read. With OpenCoarrays 2.10.3 the counter and the election were right, but the ticket check printed F (the published element values were garbage); that was not investigated, so the example is portable across the standard, not shown to be portable across these libraries. Flang 23 stops with not yet implemented: coarray: intrinsic atomic_define.

An image stops when it begins normal termination (stop, or the end of the program), and fails when it terminates abnormally in a way the program may survive, which a program can simulate with the fail image statement (11.5). The rest of the program can then learn what happened and continue with the remaining images. Detecting this relies on stat= in image control statements and collectives:

Image failure is a model: the standard defines what a program can observe, and the implementation decides what can fail. The two programs below follow it with the libcaf_shmem library.

Output (4 images; the line STOP 0 comes from image 2 as it stops, and the position of that line may differ):

Source: J3/23-007r1: 11.7.11 STAT= and ERRMSG= specifiers in image control statements

The statement fail image is the failure case:

Output (4 images):

Source: J3/23-007r1: 11.5 FAIL IMAGE statement

Findings that apply to GNU Fortran 16.2 with libcaf_shmem, observed here:

Up to Fortran 2018 a variable whose type has a coarray component had to be a scalar, neither allocatable nor a pointer nor a coarray (the GNU Fortran message below still says so). The Introduction of J3/23-007r1 lists the relaxation: "A data object with a coarray component can be an array or allocatable." What remains in Fortran 2023 is that the variable shall not be a pointer, a coarray or a function result (C825), shall be an associate name, a dummy argument or have the allocatable or save attribute (C826), and that a coarray component shall be allocatable with a deferred coshape (C751). A type that holds the slab of the plate can then be used in an array of slabs. The Fortran 2023 form:

Output of gfortran -fcoarray=lib -std=f2023 (GNU Fortran 16.2): rejected:

Source: J3/23-007r1: 7.5.4.3 Coarray components, 8.5.6 CODIMENSION attribute

The message states the older restriction, so this release does not implement the relaxation. Not run; the example is checked against the standard (7.5.4.3, 8.5.6 and the Introduction), not against a compiler. Until a compiler catches up, declare one scalar variable of the type per slab, or keep an allocatable coarray of the data itself next to an ordinary array of descriptors.

Fortran 2028 is at the committee-draft stage; ISO/IEC 1539-1 for it is not published, and this section is based on the J3 working draft J3/26-007r1 of 3 March 2026 (the WG5 committee draft N2258 is not a public document). Anything below can still change. The relevant changes in the draft’s Introduction and 17.6 are:

A program that uses team= today is rejected, which I checked for completeness:

Output of gfortran -fcoarray=lib -std=f2023 (GNU Fortran 16.2):

Source: J3/26-007r1, the Fortran 2028 working draft: 17.6 Collective subroutines

Not run, not available: the interface is shown only as "upcoming". Do not write production code against it; where you need a team-scoped collective now, put it in a change team construct as shown above.

Not checked here, because no compiler for them is installed: Intel ifx, NAG, HPE Cray and NVIDIA HPC compilers. See History, Standards and Compilers for the per-compiler support table.

See also do concurrent and GPU Offload, OpenMP and OpenACC and MPI with mpi_f08 for the other ways to the same parallelism, and Error Handling and Program Termination for error stop and the termination of images.