Pro edition

Stream

At a glance

The Stream module renders batches of documents durably and concurrently, with exactly-once local commit to single-host durable stores (cross-host exactly-once is the Enterprise Stream boundary). It splits the work into two cleanly separated responsibilities: a render engine that turns validated manifests into bytes (and nothing else), and a set of durable stores — committer, checkpoint, idempotency, dead-letter — that publish those bytes safely and let a run resume after a crash without re-publishing committed output.

Availability & licensing

This capability ships in NextPDF Pro (nextpdf/pro) and activates with a Pro-tier license envelope. A deployment without that entitlement does not load the capability’s classes. Compare editions and get a license.

There is no separate per-feature license flag. Concurrency (worker count), batch size, retry budget, and store backend (in-memory versus durable filesystem) are runtime parameters, not license switches.

Install

composer require nextpdf/pro:^3

The code lives under the NextPDF\Pro\Stream namespace.

Conceptual overview

Stream is organised around a frozen seam — NextPDF\Pro\Stream\Engine\RenderEngineInterface — that separates the throughput engine from the stream semantics:

The render engine owns concurrency and bounded memory. It renders a window of pre-validated, pre-deduped manifests through renderBatch() and returns one EngineRenderResult per manifest, in input order. Crucially the engine is side-effect-free with respect to final output: it returns rendered bytes plus their sha-256 digest, never writing to a final object key. That purity is what makes exactly-once delivery possible.
The stream collaborators own delivery. The committer, checkpoint store, idempotency (dedup) store, and dead-letter store decide where bytes land, how a run resumes, which work is a replay, and what happens to terminal failures.

A per-manifest render failure is reported as a per-item Failed (or Timeout) result; it never aborts the batch. The batch envelope always succeeds with per-item outcomes.

Key concepts

Render engines and executors

InProcessRenderEngine is the synchronous, single-process correctness baseline. It validates each manifest fail-closed through the shipped RenderManifestValidator before rendering it through the Core SingleDocumentRenderer, so a bad manifest becomes a per-item failure (error code SPEC-MANIFEST-INVALID) instead of reaching the renderer.
ConcurrentRenderEngine fans a batch out to a RenderUnitExecutorInterface and restores deterministic batch order by unit index. Output is byte-identical to a sequential render regardless of completion order; a missing, duplicate, or unknown completion is a hard failure, never a silent drop.
Executors are the concurrency seam. InlineRenderUnitExecutor is the deterministic baseline; ProcessPoolRenderUnitExecutor distributes a batch across up to N php worker subprocesses that render in parallel, then collects and integrity-checks their results.

Durable, side-effect-free commit

OutputCommitterInterface::commit() publishes rendered bytes to their final destination exactly once: atomically (no partial object is ever observed), idempotently (re-committing byte-identical content performs no write and returns a CommitReceipt with idempotentReuse = true — a fresh receipt, not the original), with no silent clobber (divergent bytes to an occupied key without overwrite raises a conflict), and integrity-checked (the committer recomputes the digest before writing). The LocalFilesystemCommitter implements this for the local filesystem.

Checkpoint recovery

A RunCheckpoint is a durable barrier recording how many items a run has committed plus a snapshot of keyed state. On recovery the processor fast-forwards past the committed offset and restores keyed state, so a crash mid-run resumes without re-publishing committed output. FilesystemCheckpointStore persists each barrier atomically.

Idempotency dedup, retry, and dead-letters

The idempotency store is the fast path that lets the processor short-circuit before rendering a replayed manifest; the committer’s digest comparison remains the durable exactly-once guarantee, so a lost dedup record at worst causes a wasted re-render that the committer dedups. RetryPolicy provides bounded, deterministic exponential backoff for transient (timeout) failures; a job that exhausts its budget is captured in a DeadLetterStoreInterface rather than lost. Each store ships an in-memory variant (single-run / test scope) and a durable filesystem variant.

Durability marker

Stores whose state survives a process restart implement the DurableCapability marker. A crash-safe run requires every collaborator to be durable so it fails fast rather than promising exactly-once semantics that an in-memory store cannot keep across a restart.

Code sample — Quick start

Render one manifest and commit its bytes exactly once. The engine returns bytes plus a digest; the committer publishes them.

<?php

declare(strict_types=1);

use NextPDF\Manifest\OutputObjectKey;
use NextPDF\Manifest\Render\SingleDocumentRenderer;
use NextPDF\Manifest\RenderManifestBuilder;
use NextPDF\Manifest\TemplateRef;
use NextPDF\Pro\Stream\Commit\LocalFilesystemCommitter;
use NextPDF\Pro\Stream\Engine\InProcessRenderEngine;

$outputRoot = __DIR__ . '/out';
\is_dir($outputRoot) || \mkdir($outputRoot, 0o775, true);

// The engine renders bytes only — it never writes the final object.
$engine = new InProcessRenderEngine(SingleDocumentRenderer::standalone());

$target = OutputObjectKey::file('out', 'invoices/1001.pdf');

$manifest = RenderManifestBuilder::create('invoice-1001')
    ->withInlineInput('<h1>Invoice 1001</h1><p>Amount due: 42.00</p>')
    ->withTemplate(TemplateRef::html())
    ->withOutputKey($target)
    ->build();

$result = $engine->renderBatch([$manifest])[0];

// A durable committer publishes the rendered bytes exactly once.
$committer = new LocalFilesystemCommitter($outputRoot);

if ($result->isRendered()) {
    $receipt = $committer->commit($result->jobId, $target, $result->bytes, $result->sha256);
    echo $receipt->target->toUri(), ' (', $receipt->bytesWritten, " bytes)\n";
}

Code sample — Production

Render a batch, route timeouts to the retry policy, and dead-letter terminal failures. Commit refuses to clobber divergent bytes, so a key collision is caught and captured rather than lost.

<?php

declare(strict_types=1);

use DateTimeImmutable;
use NextPDF\Manifest\OutputObjectKey;
use NextPDF\Manifest\Render\SingleDocumentRenderer;
use NextPDF\Manifest\RenderManifest;
use NextPDF\Manifest\RenderManifestBuilder;
use NextPDF\Manifest\TemplateRef;
use NextPDF\Pro\Stream\Commit\LocalFilesystemCommitter;
use NextPDF\Pro\Stream\Engine\EngineRenderStatus;
use NextPDF\Pro\Stream\Engine\InProcessRenderEngine;
use NextPDF\Pro\Stream\Exception\OutputCommitConflictException;
use NextPDF\Pro\Stream\Retry\DeadLetterRecord;
use NextPDF\Pro\Stream\Retry\InMemoryDeadLetterStore;
use NextPDF\Pro\Stream\Retry\RetryPolicy;

$outputRoot = __DIR__ . '/out';
\is_dir($outputRoot) || \mkdir($outputRoot, 0o775, true);

$engine     = new InProcessRenderEngine(SingleDocumentRenderer::standalone(), maxBatchSize: 64);
$committer  = new LocalFilesystemCommitter($outputRoot);
$deadLetter = new InMemoryDeadLetterStore();
$retry      = RetryPolicy::default(); // 3 attempts, 100ms base, 30s cap.

/**
 * Build one manifest and remember its output target for the commit stage.
 *
 * @return array{RenderManifest, OutputObjectKey}
 */
$makeJob = static function (string $jobId, string $html): array {
    $target   = OutputObjectKey::file('out', 'invoices/' . $jobId . '.pdf');
    $manifest = RenderManifestBuilder::create($jobId)
        ->withInlineInput($html)
        ->withTemplate(TemplateRef::html())
        ->withOutputKey($target)
        ->build();

    return [$manifest, $target];
};

/** @var array<non-empty-string, OutputObjectKey> $targets */
$targets   = [];
$manifests = [];
foreach (['inv-2001' => '<h1>2001</h1>', 'inv-2002' => '<h1>2002</h1>'] as $id => $html) {
    [$manifest, $target] = $makeJob($id, $html);
    $manifests[]  = $manifest;
    $targets[$id] = $target;
}

foreach ($engine->renderBatch($manifests) as $result) {
    // A timeout is transient — the policy decides whether to re-enqueue it.
    if ($result->status === EngineRenderStatus::Timeout && $retry->shouldRetry(1)) {
        // Re-enqueue on the caller's work queue after delayMsForAttempt(1) ms.
        continue;
    }

    if (!$result->isRendered()) {
        $deadLetter->add(new DeadLetterRecord(
            jobId: $result->jobId,
            idempotencyKeyValue: $result->jobId,
            attempts: $retry->maxAttempts,
            lastErrorCode: $result->errorCode ?? 'SPEC-RENDER-EXCEPTION',
            lastErrorMessage: $result->errorMessage ?? '',
            failedAt: new DateTimeImmutable(),
        ));

        continue;
    }

    try {
        // overwrite=false: identical bytes are an idempotent no-op; divergent
        // bytes to an occupied key raise SPEC-COMMIT-409 instead of clobbering.
        $receipt = $committer->commit(
            $result->jobId,
            $targets[$result->jobId],
            $result->bytes,
            $result->sha256,
        );
    } catch (OutputCommitConflictException $e) {
        $deadLetter->add(new DeadLetterRecord(
            jobId: $result->jobId,
            idempotencyKeyValue: $result->jobId,
            attempts: 1,
            lastErrorCode: $e->specCode(),
            lastErrorMessage: $e->getMessage(),
            failedAt: new DateTimeImmutable(),
        ));

        continue;
    }

    echo $receipt->idempotentReuse
        ? "reused {$receipt->target->toUri()}\n"
        : "committed {$receipt->target->toUri()}\n";
}

if ($deadLetter->count() > 0) {
    \fwrite(\STDERR, $deadLetter->count() . " job(s) dead-lettered\n");
}

When to use

High-volume batch rendering where throughput benefits from concurrent (process-pool) execution.
Long-running runs that must survive a crash and resume without double-publishing output.
Pipelines that must guarantee exactly-once delivery of each rendered document to its target.

For a single ad-hoc document, render directly with the Writer module; Stream’s value is in durable, resumable, concurrent batches.

Performance

Throughput scales with worker count in ProcessPoolRenderUnitExecutor (bounded by maxWorkers and maxBatchSize), while the engine keeps render output byte-identical to the sequential baseline. A wall-clock timeout caps each parallel batch so a hung worker cannot block forever. There is no published fixed throughput figure; it depends on document complexity and host parallelism. Measure with representative documents.

Security notes

Manifests are validated fail-closed before rendering. The committer rejects path traversal, null bytes, stream-wrapper schemes, symlinked targets, and NTFS alternate-data-stream (colon) vectors, and resolves every key under one configured root. Cross-process worker results are re-hashed and matched against the worker-reported digest so a garbled worker cannot corrupt output silently. This module logs no document content.

Enterprise boundary note

Stream’s durable stores here are filesystem-backed and single-host. Cross-host concurrent exactly-once to the same key, and durable dedup across runs, are the job of the Enterprise object-storage committers and stores; the document-job stream processor that drives these collaborators is an Enterprise concern. Pro provides the engine, the contracts, and the local durable implementations.

Core fallback / alternative

Without Pro, render documents one at a time with NextPDF Core’s writer; durable batch streaming, concurrent execution, and exactly-once commit are Pro additions. See /modules/writer/.

Publication boundary

This page documents externally observable behavior and the supported public API surface only. Internal namespace paths, helper classes, mechanism tables, runbook filenames, and ticket prefixes are out of scope.