National UTF-8 and Code Pages

A character set is a list of characters; a coded character set (a code page) assigns each one a numeric code; the code page has a CCSID, a number from 1 to 65536 (Enterprise COBOL Programming Guide, Unicode and the encoding of language characters). A Unicode character has a code point, and an encoding (UTF-8, UTF-16) says how code points become bytes.

Source: Enterprise COBOL 6.5 Programming Guide: Unicode and the encoding of language characters and Storage of character data

The ASCII characters have the same code in UTF-8 as in ASCII, and the letter A is the single byte X"41" there. In UTF-16 the same letter is the two bytes NX"0041", and ABC is NX"004100420043". Enterprise COBOL keeps national data in big-endian order: UTF-16LE data from another platform must be converted before the program treats it as national.

A collating sequence is the order of the characters of a code page. It decides whether "a" is less than "A", where the digits fall relative to the letters, and which character is HIGH-VALUE. The EBCDIC and ASCII orders disagree in a way that affects programs every day:

Source: Enterprise COBOL 6.5 Language Reference: EBCDIC and ASCII collating sequences and EBCDIC collating sequence

So a master file sorted on the mainframe by a key that mixes digits and letters is in a different order from the same file sorted on a PC, and a program that compares keys, or merges two files, gives different results on the two platforms. The data is the same; the sequence differs. The example sorts five shelf labels of the Circulation library twice: first in the native order of the compiler (ASCII here), then with the COLLATING SEQUENCE phrase naming an alphabet defined as EBCDIC in SPECIAL-NAMES. The table form of SORT is covered in Searching and Sorting Tables; the file form in SORT and MERGE.

The output is real. The digit comes first in ASCII and last in EBCDIC, and the lowercase labels come before the uppercase ones in EBCDIC only:

Source: GnuCOBOL Programmer’s Guide: Table SORT and Enterprise COBOL 6.5 Programming Guide: Choosing alternate collating sequences

When a file crosses the boundary the data must be translated, not only the order. On z/OS the translation is usually done by the transfer tool (for example a text-mode file transfer that converts between EBCDIC and ASCII). Enterprise COBOL 6.5 can also convert tagged z/OS UNIX source files automatically when the file’s code page does not match the CODEPAGE option of the compile job; that covers program text, not the data files a program reads. Inside a program the standard tool is the CODE-SET clause of the FD: it names an alphabet, and the runtime translates the record data between the program’s native character set and that alphabet as it reads or writes. GnuCOBOL accepts it for sequential and line sequential files.

The next program writes one Circulation-style record through a file described with CODE-SET IS EBC, then reads the same file back through a second description with no CODE-SET and shows the raw bytes:

The compiler labels CODE-SET unfinished in 3.2 (this example warns, and the warning is real), but the translation happens: A, B, C, the space and 1, 2, 3 become the EBCDIC bytes C1 C2 C3 40 F1 F2 F3.

The last byte is the Latin-1 letter X"E9" (é), a character outside the 7-bit set where the tables disagree. The default translation table of GnuCOBOL maps it to 3A; running the same source with -febcdic-table=ebcdic500_latin1, one of the tables in the config directory of the installation, gives 51, the EBCDIC 500 code of that letter:

The manual of 3.2 lists the values DEFAULT, RESTRICTED-GC, IBM and GCOS for -febcdic-table; the table name above was accepted and changed the output. A table that does not match the actual code page of the partner system corrupts the accented letters and leaves the plain ones intact, so the corruption goes unnoticed in tests that use only plain ASCII names. The related option -fsign=EBCDIC for the sign nibble of DISPLAY numbers is in USAGE and Numeric Representation.

Source: GnuCOBOL Programmer’s Guide: File/Sort Description (CODE-SET) and Enterprise COBOL 6.5 Language Reference: CODE-SET clause

National data stores each character as a UTF-16 code unit in big-endian order (CCSID 1200). It was added by COBOL 2002; in Enterprise COBOL it is the older of the two Unicode types. Its pieces are:

Alphabetic and alphanumeric items, DBCS items and integers are converted to national implicitly by MOVE, and national is converted to numeric by MOVE. Comparisons convert in the same way. The compiler does this using the CODEPAGE value; the program has to take care when the data is in another code page (see below). Assigning X"C1C2" to a national item moves the characters that the bytes represent in the CODEPAGE, not the bytes C1C2, which is why the manual recommends not using alphanumeric hexadecimal literals where a national literal is meant.

Source: Enterprise COBOL 6.5 Programming Guide: Using national data (Unicode) in COBOL, Defining national data items, Using national literals and NSYMBOL

UTF-16 is convenient because one position is one character, but the world exchanges UTF-8 (JSON, HTML, Db2 Unicode tables, MQ messages). Enterprise COBOL 6.3 added native UTF-8 items, and the 6.5 What’s New page lists further UTF-8 enhancements: moves from a UTF-8 sender to numeric items, comparisons with numeric operands, and ULENGTH accepting UTF-8 arguments. There are three kinds of UTF-8 item, all declared with the U picture symbol (with USAGE UTF-8 implied or explicit):

The rules around UTF-8 items:

Source: Enterprise COBOL 6.5 Programming Guide: Using UTF-8 data (Unicode) in COBOL, Defining UTF-8 data items, Using UTF-8 literals, COBOL statements and UTF-8 data items, Db2 considerations and What’s new in Enterprise COBOL for z/OS 6.5

CODEPAGE(ccsid) (abbreviation CP) names the EBCDIC code page for compile-time and run-time operations that depend on the encoding. The default is CODEPAGE(1140), the same as CCSID 37 plus the euro sign. It decides:

It does not change national items (always UTF-16 big-endian, CCSID 1200), UTF-8 items (always 1208), or the characters of the basic COBOL set: the compiler always reads the dollar sign as X"5B" and the quotation mark as X"7F", as in CCSID 1140. Programs that use DBCS names, DISPLAY-1 items or DBCS literals must compile with one of the mixed EBCDIC multibyte CCSIDs (for example 930, 939, 933, 935 or 937), and a program compiled with the TEST option must use the CCSID of the source. NATIONAL-OF, DISPLAY-OF, XML processing and a few other operations can override the option with an explicit CCSID, but an option set wrongly for the whole program is the more frequent cause of garbled accents on a mainframe: the source was saved in one code page and compiled with another, so every literal is translated wrongly while the data is fine. Compiler options are collected in Performance and Compiler Options.

Source: Enterprise COBOL 6.5 Programming Guide: CODEPAGE

GnuCOBOL has no code-page option: the source, the data and the screen use the bytes of the system, which is UTF-8 on a modern Linux or macOS terminal. Its support for national and UTF-8 data is partial and, in 3.2, labelled unfinished by the compiler itself. The Programmer’s Guide says that national support is "currently only partially implemented" and that USAGE NATIONAL is "syntactically recognized" but not supported, and cobc --list-intrinsics marks NATIONAL-OF, DISPLAY-OF and CHAR-NATIONAL as not implemented. This table shows what GnuCOBOL 3.2.0 does in practice, found by compiling and running the programs on this page:

The next program shows the two storage items on a real compile. The warnings are part of the evidence; the program is a probe, not a recommendation:

The national item holds 0041 0042 correctly but its blanks are two-byte 2020 pairs, not the UTF-16 space 0020, which is a difference from the IBM storage layout that a program exchanging national data would notice. The PIC U(4) item holds José in the five bytes 4A 6F 73 C3 A9 and then blanks: it takes the bytes of the source as they come, and nothing validates them.

Code that calls the IBM Unicode functions does not compile. This example is deliberately wrong for GnuCOBOL 3.2:

The real messages (the cobc output also prints the source line after each, left out here):

Source: GnuCOBOL Programmer’s Guide: NATIONAL-OF and GnuCOBOL manual

JSON generation and parsing in UTF-8 are in JSON GENERATE and PARSE; the Db2 coprocessor is in Embedded SQL and Db2; the file organizations that depend on key order are in Indexed Files and VSAM.

The book consulted for this section (Taulli, Modern Mainframe Development, 2022, written for Enterprise COBOL 6.3) presents text as PIC X data and does not cover national or UTF-8 data; nothing is copied from it. Three points follow from that: