Coarrays, Images and Synchronization

Four facts define the model.

The standard says nothing about how the images are implemented. Note 2 of 5.4.7 describes two possibilities: a shared-memory machine, where a coarray on every image is a sequence of arrays with evenly spaced starting addresses, and a distributed-memory machine, where each image stores the coarray at the same virtual address in its own memory. In practice the images are operating-system processes (OpenCoarrays over MPI, or GCC’s libcaf_shmem over shared memory) or, in other implementations, threads.

The same idea in other languages is the process-and-shared-array model of MPI one-sided communication (covered in MPI with mpi_f08) and, in the PGAS family, UPC and Chapel. In Fortran the default is that every image owns its data and the other images may look at it only through the brackets.

Coarrays need compiler and runtime support, and what is available differs a lot between toolchains. The table lists what was tried for this page; the details follow.

num_images() and this_image() are transformational functions with no required arguments (16.9.156 and 16.9.208; both also accept a team argument, covered in the teams page). Together they are the whole of the language’s process management:

Run with four images:

Source: J3/23-007r1: 16.9.156 NUM_IMAGES and 16.9.208 THIS_IMAGE

The four lines are in an arbitrary order, and the order changes from run to run: the images are not synchronized, and the standard does not say how the files they write to are shared. Output is the first thing that shows the model. The practical rule is that lines of different images interleave arbitrarily, so a program that wants ordered output either gathers the data on one image and prints it there (the next example) or serializes the printing, which sync images does with a chain of sync images statements.

Run with a single image the program prints hello from image 1 of 1, so a coarray program can always be tested without a parallel runtime.

A coarray is declared with the codimension attribute (8.5.6), usually written as a bracketed coarray-spec after the name or after the array bounds. Compare the declaration of an array, whose extents are in parentheses, with that of a coarray, whose cobounds are in brackets. Everything in the brackets describes how the images are arranged, not how big the data is:

The rules, with their clauses, are:

A coarray that is a derived type with allocatable or pointer components is allowed; the target of such a pointer is always on the same image as the pointer (Note 1 of 8.5.6.1). In Fortran 2023 an object whose type has a coarray component may also be an array or allocatable (see the provenance table at the end); that rule is not exercised here, and the ragged array below uses only the older rule, a coarray with an allocatable component.

The rule that makes coarray programs correct or wrong is the segment rule of 11.7.2. Execution of an image control statement divides the execution sequence on an image into segments: the statements before the first image control statement, between two of them, and after the last. The image control statements are (11.7.1):

Segments on different images are either ordered (one precedes the other, because of image control statements) or unordered, in which case they may run concurrently. The rule on shared data is (11.7.2): if a variable is defined, or becomes undefined, on an image in a segment, it shall not be referenced, defined, or become undefined in a segment on another image unless the segments are ordered. In other words, a read in one image of a value written by another image is valid only if a chain of image control statements guarantees that the write segment precedes the read segment, and the same for write after read and write after write. Two reads of the same variable need no ordering. Violating the rule is a data race: the program is not conforming, and nothing diagnoses it. The set of all segments on all images is a partial order (Note 1 of 11.7.2): segment P_i precedes Q_j if and only if there is a sequence of segments from the first to the second in which each precedes the next, either because they are consecutive on the same image or because of image control statements.

Two more consequences follow from the definition. First, an image may keep a copy of a coarray in a register or cache inside a segment (Note 4 of 11.7.2), so a loop that polls a coarray written by another image, without any image control statement, may never see the change and cannot be made to work with plain coindexed reads; use sync memory, an atomic subroutine (next page) or an event. Second, the compiler may move code within a segment as if it were the only image executing (Note 3), so the assumption that statements run in source order across images is wrong unless ordered by an image control statement.

Two statements serialize access to a resource shared by images. They differ in what they protect: a critical construct is a block that one image at a time may execute, and a lock variable is a data object that images acquire and release explicitly.

An allocatable coarray (8.5.6.2) is the form to use for data whose size is only known at run time, as in heatflow where the number of rows per image depends on the number of images. Its allocate statement carries the cobounds in the brackets, the last one always *:

allocate and deallocate of a coarray are image control statements with an implicit synchronization of all active images in the current team (9.7.1.2, 9.7.3.2): execution after the statement on any image is delayed until all other images have executed the same statement the same number of times, and the segments before it precede the segments after it on all images. So the first access to a freshly allocated coarray by another image is always safe. The price is that the allocation is collective, with rules the programmer must keep, none of which is checked by the compiler:

Source: J3/23-007r1: C826 in 8.5.6.1 and C848 in 8.5.10

The same implicit synchronization happens when a block or procedure ends and an unsaved allocatable coarray declared in it is deallocated automatically, so an innocent-looking end block can deadlock a program if only some images reach it (an implicit deallocation is an image control statement, 11.7.1). The example shows an allocation with cobounds, a two-dimensional coshape, a coarray of a type whose allocatable component has a different length on every image, an explicit deallocate, and a block whose coarray is deallocated at end block:

Three images:

Source: J3/23-007r1: 5.4.10 Allocatable variables, 9.7.1.2 Execution of an ALLOCATE statement and 9.7.3.2 Deallocation of allocatable variables

The allocatable component r%v is not subject to the rule that all images must agree: its allocation is local, only the coarray r itself has to be the same everywhere (Note 1 of 9.7.1.1, "a pointer or allocatable component of a coarray can only be allocated by its own image"). The sync all after the allocate (r%v(me)) is what makes the reads r[k]%v safe: the allocation status of an allocatable subobject of a coarray is changed on one image, so no other image may inquire about it until the segments are ordered (11.7.2). Such a ragged coarray, an array of images each holding its own length, is how irregular distributions are expressed (see ragged_t above). The standard also forbids, for the same reason, source= or mold= of a type that is lock_type, event_type, notify_type or team_type, or has such a component, in the allocate of a coarray (C954, 9.7.1.1).

A coarray can be a dummy argument, a module variable or a saved local, never an automatic or non-saved local (C826). The rules for dummy coarrays are in 15.5.2.9:

Source: J3/23-007r1: 15.5.2.9 Coarray dummy variables

The module below puts the rules in a runnable form: an explicit-shape coarray dummy used to read a neighbour, an assumed-shape coarray dummy passed a contiguous section, and a saved coarray in a module (total_calls), which is legal because save makes it established in the initial team:

Three images:

Source: J3/23-007r1: 15.5.2.9 Coarray dummy variables

The call fill_assumed(b(4:7), 0.0_dp) passes a section of a coarray, which is itself a coarray because it has no cosubscript and no vector subscript (5.4.7), so it can be a coarray actual argument. In the output, b(4)[2] is zero and b(3)[2] and b(8)[2] are untouched. Two points about the program: the sync all that fill_assumed and tally execute is legal because the procedures are not pure, and means that every image must call them, in the same number of times: calling tally from an if (me == 1) would be a deadlock. And a first draft of 11_procedures.f90 contained a data race: image 1 printed b(4)[2] and, with no barrier, image 2 zeroed it. The printed value was 0.0 instead of 200.0: the second sync all in the program is the fix. A parallel program’s first bug is usually of this kind.

Two limitations of gfortran 16.2 with -fcoarray=lib were met in the process. Both were reproduced with the smallest program shown, and neither is explained here; they are reported as observed so that a reader who meets them recognizes them.

A coindexed reference in a module procedure with a dummy coarray of non-constant extent is an internal compiler error. The extent n is a dummy argument:

Source: J3/23-007r1: 15.5.2.9 Coarray dummy variables and 9.6 Image selectors

The same procedure compiles with -fcoarray=single, with an extent that is a constant (as in 11_procedures.f90, where slab is a module parameter), or outside a module. The workaround is a constant extent.

A coindexed reference through an assumed-shape dummy coarray crashes at run time. This module procedure reads x(1)[k]; with image 1 asking for itself it crashes as well:

Three images with libcaf_shmem (the output is abridged to the cause; a first draft of 11_procedures.f90 that used this form crashed the same way with OpenCoarrays and with -O2, and the program above runs correctly with -fcoarray=single):

Source: J3/23-007r1: 15.5.2.9 Coarray dummy variables and 9.6 Image selectors

Assumed-shape dummy coarrays work when used without cosubscripts (the fill_assumed routine above), and explicit-shape and assumed-size dummies with constant extent work with coindexed references. Prefer the explicit-shape form with a module parameter extent until the defects are fixed, and test the exact compiler version: nothing in the standard makes these forms invalid.

An image terminates normally when it executes stop or reaches the end of the main program, and with error termination when it executes error stop or in some other cases defined by the standard (5.3.7). The two models are different, and both matter to a coarray program:

An image can also be in a failed state (5.3.6): it has stopped participating without initiating termination, as when its process is killed. Whether a processor can detect that is processor dependent; the standard makes stat_failed_image positive if it can and negative if not. After a failure the data of nonfailed images become undefined if the failed image was running a segment that might have defined them, except lock, event and notify variables. Image failure, the intrinsic functions failed_images, stopped_images and image_status and the fail image statement belong to the next page.

For the image control statements the standard offers two ways to handle a stopped or failed image. Without stat=, an error condition in an image control statement initiates error termination (11.7.11). With stat=, execution continues and the variable is set: to stat_stopped_image (positive) if one of the images involved has stopped, otherwise to stat_failed_image if one has failed and no other error occurred, otherwise to a processor-dependent positive value. errmsg= receives an explanatory message. For sync all, sync images and sync team with a stopped image the effect is that of a sync memory (11.7.11). Both specifiers use a noncoindexed variable (C1180).

The two examples are the textbook ones. First error stop on image 3 while the others wait in sync all:

Source: J3/23-007r1: 5.3.6 Image execution states

With GCC’s libcaf_shmem (four images, 20 s time limit added by the test script):

The message was printed but the other three images stayed in sync all until the time limit killed them (exit=124 is the timeout exit status): this runtime did not propagate the error termination. With OpenCoarrays the whole job ended:

Second, stop on image 2 only while the others execute sync all (stat=...), which is the situation the standard’s stat_stopped_image is for:

Source: J3/23-007r1: 11.7.11 STAT= and ERRMSG= specifiers in image control statements and 16.10.2.31 STAT_STOPPED_IMAGE

This program is correct Fortran 2023, and in this toolchain it does not work as the standard describes. With libcaf_shmem it printed STOP and then hung until the 20-second time limit killed it. With OpenCoarrays 2.10.3 the stop of image 2 aborted the whole MPI job:

Do not build a recovery strategy for stopped or failed images on these runtimes without testing the exact installation; the language feature exists, and the most complete implementations are the commercial compilers and, in the future, Flang with a PRIF runtime. What can always be relied on is that all the images should stop together, that a stop of a single image is valid when the others do not need it any more, and that error stop terminates the program.

The halo exchange puts the earlier sections together. The plate of the running example has nx = 41 columns and ny = 40 rows. The rows are divided into slabs of nloc = ny/np consecutive rows, one per image; each image keeps its slab in the coarray u(nx, 0:nloc + 1)[*], where row 0 and row nloc + 1 are the halo rows: copies of the last row of the image below and the first row of the image above, which the stencil needs to update the edge rows of the slab.

Each time step has two segments separated by two barriers (the lower half of the figure): segment A reads the neighbours' edge rows into the halo rows; segment B updates the owned rows from the local data. The barrier between A and B is needed so that nobody overwrites an edge row in B before all neighbours have read it in A; the barrier after B is needed so that the new edge rows are complete before any neighbour pulls them in the next A. The hot spot and the boundary rows are placed by global row number, so that the result does not depend on the number of images:

Four images:

Source: J3/23-007r1: 5.4.7 Coarray, 9.6 Image selectors, 11.7.3 SYNC ALL statement and 9.7.1.2 Execution of an ALLOCATE statement

The decomposition is the point of the result: with 1, 2, 4, 5 and 8 images (the divisors of 40 that were tried) the program printed the same images=N max= 0.512 sum= 99.811976 apart from N, so the parallel program agrees with the serial one to the printed digits. The same was true under OpenCoarrays with four images. The program ends with co_max and co_sum, collective subroutines that every image calls and that are the subject of the next page; they replace a manual loop of coindexed reads. The two if statements in segment A and the choice of jlo and jhi keep the boundary images correct: image 1 has no lower neighbour and global row 1 is a fixed boundary, and image np likewise for row ny. The check mod(ny, np) /= 0 is done by every image before any synchronization, so all images stop together; when the number of images does not divide the number of rows a real code would give the first mod(ny, np) images one row more, which requires the coarray extent to be the maximum slab size.

Two barriers per step involve all the images even though each image only needs its neighbours. When the images are many, sync images with the neighbour set restricts the waiting. The change is three lines: the set of neighbours is built once with pack, and the two sync all of the time loop become sync images (nbr). A set of size zero (one image) is allowed and does nothing:

The full program is 17_halo_sync_images.f90. With eight images it printed images=8 max= 0.512 sum= 99.811976, the same as the sync all version, with both libcaf_shmem and OpenCoarrays (four images). The neighbour form is correct here because the ordering needed is only between neighbours (a pull by image k + 1 of the rows of image k, which is neighbour synchronization) and the barrier is not needed for correctness; it is faster when load is unbalanced, since an image that finishes early waits only for two others. The cost is the care needed: the set must be symmetric, as image k’s set contains k + 1 exactly when that of image k + 1 contains k, and a mismatch deadlocks.

Compiler support, as found: gfortran 16.2 supports the language in this page with -fcoarray=lib (two defects noted above, and no working stop or failed-image handling in the shared-memory library); Flang 23 has experimental support in the front end and needs a PRIF runtime, which the toolchain here did not provide; Intel, NAG and HPE Cray document coarray support for their compilers, and the NAG and Cray manuals and compiler options were not available for this page.

Coarray chapters written before 2018 describe the language as it stood in Fortran 2008, and some assume that a coarray program is a niche. Four points of the modern landscape differ. Teams, events, collectives, atomics and failed images (Fortran 2018) and put with notify (Fortran 2023) are not part of the older description, and are covered in the next page. A coarray program no longer needs a proprietary compiler: gfortran has run multi-image programs since the OpenCoarrays integration, and GCC 16 ships its own shared-memory runtime (libcaf_shmem, selected with -fcoarray=lib); the -fcoarray= keywords seen in the compiler’s own help are none, single and lib. LLVM Flang is a newcomer with experimental coarray support. Finally, coarrays are one of four mainstream models in 2026, together with do concurrent, OpenMP and MPI (separate pages), and the choice is made on the machine: coarrays and MPI for distributed memory, do concurrent and OpenMP within a node, and a coarray program can be run on a single image for testing.

See also Best Practices and Style and Common Gotchas and Pitfalls.