Indexed Files and VSAM

A file is declared indexed by the SELECT entry in the FILE-CONTROL paragraph. The entry that every example on this page uses is the following.

Each clause has a fixed meaning.

The record itself is described in the FD by a copybook, here BOOKREC, so that every program that opens the file sees the same layout. The keys must be inside the record, and in a file with variable-length records the alternate key should lie within the minimum record length. On z/OS the SELECT clauses of the keys must match how the file was defined: the number of alternate keys, their position and their length must agree with the catalog entry.

A SUPPRESS WHEN phrase for the alternate key is described as a COBOL 2023 feature (the standard’s text is paywalled, so this page could not check it). It leaves records out of that key’s index when the key field holds a stated value, such as spaces, so that records without a value do not clutter the index. GnuCOBOL 3.2 accepts ALTERNATE RECORD KEY IS BOOK-ISBN WITH DUPLICATES SUPPRESS WHEN SPACES: in a test, a record whose key was spaces was not returned by a START on that key, and the program compiled without a warning even with -std=ibm. Enterprise COBOL 6.5 documents no SUPPRESS WHEN for files (the word only appears there in the JSON GENERATE and XML GENERATE statements and in the COPY statement), so code that must also run on z/OS should not use it.

Source: Enterprise COBOL for z/OS 6.5 Language Reference: RECORD KEY clause; ALTERNATE RECORD KEY clause; ACCESS MODE clause

A new indexed file is created by opening it for output and writing records. With ACCESS MODE IS SEQUENTIAL the records must arrive in ascending order of the prime key; with RANDOM or DYNAMIC they may arrive in any order, at the price of more index work. The program below loads the indexed book master from the sequential BOOKMAST of the Circulation scenario (4 books, ids 1 to 4, copybook BOOKREC). It checks every WRITE with INVALID KEY and counts the records that were accepted.

Source: Enterprise COBOL for z/OS 6.5 Language Reference: WRITE for indexed files

Two details of the run are worth knowing. The WRITE statement names the record (BOOK-RECORD), not the file, and the key value written is whatever the record holds at that moment, so a program that builds records in WORKING-STORAGE must MOVE them to the record area (or use WRITE ... FROM) first. And the run created two files in the current directory, BOOKKSDS for the data and the prime index and BOOKKSDS.1 for the alternate key, because the GnuCOBOL build used here stores indexed files with the Berkeley DB library (see GnuCOBOL indexed back ends). On z/OS nothing is created by the program: the cluster must already exist, defined with IDCAMS, and only its first load is done by COBOL (see Defining a KSDS with IDCAMS).

All the file statements take an INVALID KEY and a NOT INVALID KEY phrase when the file is indexed, and end with a scope terminator (END-READ, END-START, END-WRITE, END-REWRITE, END-DELETE). The phrase runs when the statement raises the invalid key condition (status 21, 22, 23 or 24); an AT END phrase on a sequential READ is used instead for the at-end condition (status 10). If a statement has neither the phrase nor an applicable declarative, the outcome depends on the FILE STATUS clause, as described on Error Handling and Declaratives.

The statements that are legal for each access mode and open mode are those of the IBM table for VSAM relative and indexed files, and GnuCOBOL follows the same table in the cases tried here: WRITE needs OUTPUT (or I-O for random and dynamic access), READ and START need INPUT or I-O, and REWRITE and DELETE need I-O.

Source: Enterprise COBOL for z/OS 6.5 Programming Guide: Coding input and output statements for VSAM files; Language Reference: START statement; READ statement; REWRITE statement; DELETE statement

These programs are deliberately wrong or stress an error path. Their messages are the ones GnuCOBOL 3.2.0 printed.

START in random access mode. START is accepted only for sequential and dynamic access. With ACCESS MODE IS RANDOM the compiler refuses it (the program, saved as d2.cob, is the book file opened for input with ACCESS MODE IS RANDOM, followed by that START):

START on a field that is not a key. The key named in START must be the prime key, an alternate key, or (on z/OS) a field that begins at the same position as one of them and is not longer than it. Naming BOOK-TITLE, which is not a key, in a program saved as d1.cob gives:

The READ BOOK-KSDS KEY IS BOOK-TITLE on line 19 raised no diagnostic of its own, but it is invalid in the same way: the language reference requires the key of a random READ to be a record key.

Loading out of order. In sequential access mode the prime keys must ascend. In a test with ACCESS MODE IS SEQUENTIAL and OPEN OUTPUT, a WRITE of book 5 followed by a WRITE of book 3 gave status 00 for the first and status 21 for the second. Status 21 is the sequence error of the invalid key class, and the statement that breaks the order is not stored. A loader fed from an unsorted file should either sort it first (see Sort and Merge) or use DYNAMIC access, and it should stop at the first status 21 instead of carrying on, because later records of the same run may also be refused.

An alternate key without WITH DUPLICATES. The program below, BAD1, defines BOOK-STATUS as an alternate key without the phrase and writes three books with statuses "A", "A" and "W":

The first WRITE is fine, the second sets status 22 although its prime key (6) is new: it is the alternate key that is duplicated, and the record is not stored. The third has a new status value and is accepted. A status flag is a typical field that needs WITH DUPLICATES. On z/OS the same program would additionally fail if the alternate index had been defined UNIQUEKEY while the program says WITH DUPLICATES, or the other way round, because the definitions in the program and in the catalog must agree.

REWRITE after changing the key. In sequential access mode, REWRITE replaces the record last read, so its prime key must still be the one that was read. In a test with dynamic access, a READ of book 1 followed by a MOVE 7 TO BOOK-ID and a REWRITE returned status 21, and the file was unchanged. The key is the identity of the record: to "change" it, WRITE a new record with the new key and DELETE the old one. The language reference lists this situation under status 21 (the prime record key was changed between the READ and the next REWRITE); a DELETE of a key that is not in the file returned status 23 in the same test.

Opening a file that does not exist. An OPEN INPUT of a file that was never created gives status 35. On z/OS the equivalent is a missing cluster, or one that has never contained a record.

Source: Language Reference: Common processing facilities (file status key, invalid key condition)

The VSAM (Virtual Storage Access Method) data set types are the physical side of the COBOL file organizations. The Programming Guide uses three of them: the entry-sequenced data set (ESDS) is a sequential file, the key-sequenced data set (KSDS) is an indexed file and the relative-record data set (RRDS) is a relative file. A fourth type, the linear data set (LDS), has no COBOL organization.

An ESDS can have alternate indexes in VSAM, but the Programming Guide says COBOL does not support them for it. The linear data set is outside what a COBOL READ can do: IBM states that, since it has no control information, it cannot be accessed as if it contained individual records. The files of the Circulation scenario that need a key, BOOKMAST and MEMBMAST, are KSDS clusters; the loan transaction file is a sequential data set (ESDS or QSAM); a slot-numbered file would be an RRDS.

Source: Programming Guide: VSAM files; z/OS 3.2 DFSMS Using Data Sets: Linear data sets; Key-sequenced data sets

Enterprise COBOL 6.5 adds a second kind of indexed file, VSAMDB. It is a VSAM KSDS defined with the DATABASE(JSON) attribute and accessed with record-level sharing, and it stores UTF-8 JSON text: one JSON document is one record, and the key is a JSON value, usually a string. The same data set can also be reached through the z/OS VSAM APIs and the EzNoSQL C, Python and Java APIs, which is what makes it a "NoSQL database on z/OS". COBOL uses the usual statements (OPEN, WRITE, START, READ, REWRITE, DELETE, CLOSE), with these differences:

The fragment above (z/OS; not run here) follows the shape of the two VSAMDB examples in the Programming Guide, with the Circulation names; the key would be produced with JSON GENERATE BOOK-JSON-KEY FROM WS-BOOK-ID NAME WS-BOOK-ID IS OMITTED, which yields the string "000002" including its quotes. PIC U DYNAMIC is a COBOL 6 feature that GnuCOBOL 3.2 does not have, so this code has no portable counterpart; its JSON statements are on JSON GENERATE and PARSE. The cluster is defined with the DATABASE(JSON) parameter of DEFINE CLUSTER, which also lets the definition name an explicit primary key field in the documents (KEYNAME), or leave the key implicit so that VSAM generates it and returns it in the key item after a WRITE. A VSAMDB data set needs z/OS RLS (and DFSMStvs for recoverable use); the details are in the Programming Guide chapter on VSAMDB and in the DFSMS manual on defining a VSAM RLS database. The GnuCOBOL 3.2 manuals do not mention VSAMDB.

Source: Enterprise COBOL 6.5 Programming Guide: Processing VSAMDB files; Defining VSAMDB files; Reading records from a VSAMDB file; Language Reference: START statement (PARTIAL phrase)

GnuCOBOL does not implement indexed files itself: it hands them to a file-management library chosen when the compiler runtime is built. The GnuCOBOL Programmer’s Guide says that the physical structure varies with the facility built in (Berkeley DB, VBISAM and others), and its description of file mapping options lists the indexed handlers CISAM, DISAM, VBISAM, BDB and LMDB. The manual section on indexed file packages is marked "in progress" in the 3.2 manual. The build used for every example on this page reports its handler:

so its indexed files are Berkeley DB files (the runtime library links libdb-5.3), and a file with one alternate key is stored as two files, BOOKKSDS and BOOKKSDS.1. Other builds (for example distribution packages) may use another handler, and a file written by one handler cannot be read by another. Which handler a given cobc has is shown by cobc --info; this page tested only Berkeley DB, and has not built GnuCOBOL with VBISAM or LMDB. With the Berkeley DB back end, the environment variable DB_HOME names the directory for the lock files and the data files and switches on record locking for READ, REWRITE and WRITE.

Where the data files live is decided at run time through file name mapping. With the default configuration, the name in ASSIGN TO "BOOKKSDS" is resolved in this order: the environment variable DD_BOOKKSDS, then dd_BOOKKSDS, then BOOKKSDS, then the literal name itself. A relative name without a path is looked for under the directory in COB_FILE_PATH, which defaults to the current directory. This is how the same program moves from a test directory to production, and the closest thing GnuCOBOL has to a JCL DD statement. The order of the lookup comes from the Programmer’s Guide and the default of COB_FILE_PATH from the manual’s runtime configuration chapter; this page did not run a test of either.

Source: GnuCOBOL Programmer’s Guide: ORGANIZATION INDEXED, File Location Strings, file mapping options; GnuCOBOL manual: Indexed file packages, SELECT ASSIGN TO, runtime configuration

An indexed file is the right tool when a program needs to find single records by a business key and also to walk them in key order, and when the volume is moderate. A sequential file is cheaper to read from start to end, as in the nightly batch of Sequential Files, and a relative file suits slot-numbered lookups. Today many installations keep the data in a database (see Embedded SQL and Db2); an indexed file remains the fast, simple choice for master files inside a batch system, and the reason a KSDS survives in so many shops. A new program should not choose it for data that other systems must query, and a design with more than one or two alternate keys is usually a sign that a relational table would serve better.