Chunk-oriented processing

This section documents the current Spring Batch line — 6.0.x, on Spring Framework 7 and Spring Boot 4.1.x, with a Java 17+ baseline — as published at the Spring Batch reference documentation. No specific patch version is pinned. Some surfaces (Spring Cloud Task and Spring Cloud Data Flow orchestration, the deployer-based partition handler, and JSR-352) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

Chunk-oriented processing is what Spring Batch is for: read items one at a time, transform each one, accumulate them until a chunk is full, then write the whole chunk inside a single transaction.

The loop

flowchart TB Start(["step starts"]) --> Tx["begin transaction"] Tx --> Read["ItemReader.read()
one item"] Read -->|null| Flush["write the partial chunk,
commit, end the step"] Read -->|item| Process["ItemProcessor.process(item)
transform or filter"] Process -->|null: filtered| Full Process -->|output| Buffer["add to the in-memory chunk"] Buffer --> Full{"chunk complete?
(commit interval reached)"} Full -->|no| Read Full -->|yes| Write["ItemWriter.write(chunk)
the whole list at once"] Write --> Update["update the StepExecution,
persist the ExecutionContext"] Update --> Commit["commit transaction"] Commit --> Tx Flush --> Done(["step ends"])

Three properties of that picture matter:

  • the reader and processor see one item at a time, so memory is bounded by the chunk, not by the input;

  • the writer sees the whole chunk, so it can issue one batched statement instead of N;

  • the commit interval is the chunk size is the transaction boundary — these are three names for one number. A chunk of 500 means one transaction per 500 items, one ExecutionContext save per 500 items, and a restart granularity of 500 items.

If anything in the chunk fails, the whole transaction rolls back — all 500 items — and the framework’s fault-tolerance machinery decides whether to retry, skip or fail (Fault tolerance: skip & retry).

Building a chunk step

The current idiom constructs the builder directly and passes the chunk size and the transaction manager:

@Bean
public Step loadTradesStep(JobRepository jobRepository,
                           PlatformTransactionManager transactionManager,
                           ItemReader<TradeCsv> reader,
                           ItemProcessor<TradeCsv, Trade> processor,
                           ItemWriter<Trade> writer) {
    return new StepBuilder("loadTradesStep", jobRepository)
            .<TradeCsv, Trade>chunk(500, transactionManager)
            .reader(reader)
            .processor(processor)
            .writer(writer)
            .build();
}

The two type parameters are the input type (what the reader returns) and the output type (what the processor produces and the writer consumes). They may differ — that type change is the processor’s main job. A step without a processor uses the same type twice.

The 6.0 ChunkOrientedStep

Spring Batch 6.0 introduces ChunkOrientedStep and its ChunkOrientedStepBuilder, which take the chunk size as a constructor argument and expose the new concurrency model (see Scaling & parallel processing):

@Bean
public Step loadTradesStep(JobRepository jobRepository,
                           PlatformTransactionManager transactionManager,
                           ItemReader<TradeCsv> reader,
                           ItemProcessor<TradeCsv, Trade> processor,
                           ItemWriter<Trade> writer) {
    return new ChunkOrientedStepBuilder<TradeCsv, Trade>("loadTradesStep", jobRepository, 500)
            .reader(reader)
            .processor(processor)
            .writer(writer)
            .transactionManager(transactionManager)
            .build();
}

The two forms are not equal in status. StepBuilder.chunk(int, PlatformTransactionManager) and chunk(CompletionPolicy, PlatformTransactionManager) — along with the SimpleStepBuilder and FaultTolerantStepBuilder they return — are annotated @Deprecated(since = "6.0", forRemoval = true). The non-deprecated entry point is StepBuilder.chunk(int), which returns a ChunkOrientedStepBuilder.

The .chunk(size, transactionManager) form is still what most existing code and most published examples use, and it is what the other pages in this section show, so it is worth recognising — but new code should prefer ChunkOrientedStepBuilder, and code being migrated should plan to move off the deprecated form.

Dynamic chunk sizes with a CompletionPolicy

A fixed chunk size is a fixed count. When "how much is a chunk" depends on something else — elapsed time, a change of business key, a memory budget — supply a CompletionPolicy instead:

@Bean
public Step timeBoundedStep(JobRepository jobRepository, PlatformTransactionManager tx,
                            ItemReader<Trade> reader, ItemWriter<Trade> writer) {
    CompositeCompletionPolicy policy = new CompositeCompletionPolicy();
    policy.setPolicies(new CompletionPolicy[] {
            new SimpleCompletionPolicy(1000),          // at most 1000 items ...
            new TimeoutTerminationPolicy(5_000L)       // ... or 5 seconds, whichever comes first
    });

    return new StepBuilder("timeBoundedStep", jobRepository)
            .<Trade, Trade>chunk(policy, tx)
            .reader(reader)
            .writer(writer)
            .build();
}

SimpleCompletionPolicy is what a plain integer chunk size creates internally. A custom policy implements CompletionPolicy and decides, per item, whether the chunk is finished — for instance closing the chunk whenever the account number changes, so that all items for one account commit together.

The underlying RepeatOperations abstraction that runs this loop is described in Repeat & retry internals.

Transaction attributes

The chunk transaction is a normal Spring transaction, and its attributes are configurable per step:

@Bean
public Step isolatedStep(JobRepository jobRepository, PlatformTransactionManager tx,
                         ItemReader<Trade> reader, ItemWriter<Trade> writer) {
    DefaultTransactionAttribute attribute = new DefaultTransactionAttribute();
    attribute.setPropagationBehavior(Propagation.REQUIRES_NEW.value());
    attribute.setIsolationLevel(Isolation.READ_COMMITTED.value());
    attribute.setTimeout(120);   // seconds; fail the chunk rather than hold locks forever

    return new StepBuilder("isolatedStep", jobRepository)
            .<Trade, Trade>chunk(200, tx)
            .reader(reader)
            .writer(writer)
            .transactionAttribute(attribute)
            .build();
}

  • Propagation — REQUIRED by default. REQUIRES_NEW isolates the chunk from any surrounding transaction; NOT_SUPPORTED runs the chunk without one, which only makes sense for a read-only step.

  • Isolation — the default is the database’s. Raising it increases contention; see SQL Transactions.

  • Timeout — an upper bound on one chunk. A long timeout plus a large chunk is the classic recipe for lock contention with the online system.

.noRollback(SomeException.class) declares exceptions that must not roll the chunk back — typically a business exception thrown by a writer that has already handled the failure itself:

return new StepBuilder("tolerantStep", jobRepository)
        .<Trade, Trade>chunk(200, tx)
        .reader(reader)
        .writer(writer)
        .faultTolerant()
        .noRollback(DuplicateTradeException.class)
        .build();

Registering an ItemStream

Readers and writers that hold restart state implement ItemStream, and the step must know about them so that open, update and close are called at the right moments. A reader or writer passed to .reader(…​) or .writer(…​) is registered automatically — but a delegate hidden inside a composite or a custom component is not, and must be registered explicitly:

@Bean
public Step compositeWriterStep(JobRepository jobRepository, PlatformTransactionManager tx,
                                ItemReader<Trade> reader,
                                FlatFileItemWriter<Trade> auditFileWriter,
                                JdbcBatchItemWriter<Trade> databaseWriter) {

    CompositeItemWriter<Trade> composite = new CompositeItemWriter<>();
    composite.setDelegates(List.of(databaseWriter, auditFileWriter));

    return new StepBuilder("compositeWriterStep", jobRepository)
            .<Trade, Trade>chunk(500, tx)
            .reader(reader)
            .writer(composite)
            .stream(auditFileWriter)   // the delegate needs open/update/close
            .build();
}

Forgetting the .stream(…​) registration is the usual cause of "the file was never created" or "restart started from the beginning of the file".

Further reading

For the full detail behind this page: