Stream — Deep Reference

At a glance

This page documents the public contracts, classes, methods, and failure modes of the NextPDF\Pro\Stream subsystem beyond the overview page. Every type below is part of the documented Pro public surface.

Availability & licensing

This capability ships in NextPDF Pro (nextpdf/pro) and activates with a Pro-tier license envelope. A deployment without that entitlement does not load the capability’s classes. Compare editions and get a license.

No per-feature license flag applies; the code ships with the Pro edition. Worker count, batch size, retry budget, and store backend are runtime parameters.

Architecture: the frozen engine seam

NextPDF\Pro\Stream\Engine\RenderEngineInterface is the contract between the throughput engine and the document-job stream processor. The engine implements it (owning concurrency, worker-pool lifecycle, backpressure, bounded memory); the stream processor consumes it (owning keyed state, dedup, retry, checkpoint, and exactly-once commit). The engine returns bytes plus sha-256, never a committed location — that side-effect freedom is what lets the processor stage, commit, and checkpoint exactly once.

public function renderBatch(array $manifests, array $variablesByJobId = []): array; // list<EngineRenderResult>, input order
public function maxBatchSize(): int;   // int<1, max> backpressure hint
public function isAvailable(): bool;

$manifests is a list<RenderManifest> of size at most maxBatchSize(); $variablesByJobId maps job id to array<string, scalar> template variables. A per-manifest failure is a per-item Failed/Timeout result and never aborts the batch.

Engines

NextPDF\Pro\Stream\Engine\InProcessRenderEngine

Synchronous, single-process baseline. Validates each manifest fail-closed through RenderManifestValidator (16 MiB inline-payload cap, conformance/signature allow-lists, sha-256 content-hash format, BCP-47 locale syntax) before rendering through the Core SingleDocumentRenderer. A blocking validation error short-circuits to EngineRenderResult::failed(jobId, 'SPEC-MANIFEST-INVALID', ...); a render exception becomes 'SPEC-RENDER-EXCEPTION'. Constructor: __construct(SingleDocumentRenderer $renderer, int $maxBatchSize = 64, ?RenderManifestValidator $validator = null) — maxBatchSize < 1 throws InvalidArgumentException. isAvailable() is always true.

NextPDF\Pro\Stream\Engine\ConcurrentRenderEngine

final readonly, __construct(RenderUnitExecutorInterface $executor). Wraps each manifest in an indexed RenderUnit, runs them through the executor, and re-sorts completions by index so output is byte-identical to a sequential render. A completion index outside [0, count) throws RenderEngineException::unknownUnit(); a repeated index throws duplicateResult(); a missing index throws missingResult(). maxBatchSize() and isAvailable() delegate to the executor.

Executors

NextPDF\Pro\Stream\Engine\RenderUnitExecutorInterface

public function execute(array $units): iterable; // iterable<CompletedRenderUnit>, any order
public function maxBatchSize(): int;
public function isAvailable(): bool;

Implementations may yield completions in any order; ConcurrentRenderEngine restores order by index.

NextPDF\Pro\Stream\Engine\InlineRenderUnitExecutor

final readonly, __construct(RenderEngineInterface $inner). Renders each unit in order through the inner engine — the deterministic correctness reference a parallel executor must match byte-for-byte. No time, processes, threads, or randomness.

NextPDF\Pro\Stream\Engine\ProcessPoolRenderUnitExecutor

final readonly. Distributes a batch across up to maxWorkers php worker subprocesses (one chunk each) that render in parallel; output is byte-identical to the inline baseline. Constructor:

__construct(
    int $maxWorkers = 4,
    int $maxBatchSize = 64,
    ?string $phpBinary = null,
    ?string $workerScript = null,
    ?string $autoload = null,
    ?int $timeoutSeconds = 300, // null disables the wall-clock watchdog
)

Robustness contract:

• Deadlock-free, Windows-safe. Unit payloads and results travel through temp files, not pipes; the parent polls proc_get_status() and only drains a pipe to EOF after a worker has exited, so a worker cannot wedge the parent.
• Bounded wait. timeoutSeconds caps the whole parallel render; on expiry every still-running worker is terminated and a RenderEngineException is thrown.
• Resource hygiene. A finally closes pipes, makes a bounded terminate-and-reap attempt on surviving workers (graceful terminate → force-kill → reap; a child not observed to stop within the bounded grace is abandoned rather than risking an indefinite block), and unlinks every temp file on all paths.
• Trusted correlation. Each worker must return exactly its assigned index set (no missing, duplicate, or foreign index); each rendered result’s bytes are re-hashed and matched against the worker-reported sha-256, and any status other than rendered/failed hard-fails. A per-manifest render failure is a per-unit Failed result; only an infrastructural fault (non-zero exit, unreadable/garbled output, timeout) hard-fails the executor.

isAvailable() requires both the autoload file and the worker script to exist. A non-positive bound or negative timeout throws InvalidArgumentException.

Render units and results

NextPDF\Pro\Stream\Engine\RenderUnit

final readonly — int<0, max> $index, RenderManifest $manifest, array<string, scalar> $variables. Correlation is by index, never by job id (job ids are not guaranteed unique within a batch).

NextPDF\Pro\Stream\Engine\CompletedRenderUnit

final readonly — int $index (untrusted, validated by the engine), EngineRenderResult $result.

NextPDF\Pro\Stream\Engine\EngineRenderResult

final readonly. Fields: jobId, EngineRenderStatus $status, ?string $bytes, ?string $sha256, int $pageCount, ?string $errorCode, ?string $errorMessage, array<non-empty-string, float> $timings. Factories: rendered(jobId, bytes, sha256, pageCount, timings = []), failed(jobId, errorCode, errorMessage), timedOut(jobId, errorMessage) (code SPEC-ENGINE-TIMEOUT). isRendered() reports the status. A rendered result carries bytes and a digest, never a committed location.

NextPDF\Pro\Stream\Engine\EngineRenderStatus

String-backed enum: Rendered, Failed, Timeout. isRetryable() is true only for Timeout, so the caller classifies a timeout as transient without re-inspecting the error.

Commit

NextPDF\Pro\Stream\Commit\OutputCommitterInterface

public function commit(
    string $jobId,
    OutputObjectKey $target,
    string $bytes,
    string $sha256,
    bool $overwrite = false,
): CommitReceipt;

Exactly-once publication: atomic, idempotent (byte-identical re-commit performs no write and returns a CommitReceipt with idempotentReuse = true — a fresh receipt, not the original; its committedAt is the current clock), no silent clobber, and integrity-checked (the committer recomputes the digest). Failure modes: CommitIntegrityException (declared sha-256 does not match the bytes), OutputCommitConflictException (divergent bytes at an occupied key with overwrite = false), UnsupportedTargetException (unsupported target scheme).

NextPDF\Pro\Stream\Commit\LocalFilesystemCommitter

final readonly, implements OutputCommitterInterface, DurableCapability. __construct(string $rootDirectory, ?AtomicFileWriter $writer = null, ?ClockInterface $clock = null). Services only the file scheme; resolves every target under one configured root and writes through an atomic writer (O_EXCL temp → fsync → same-volume rename). The full critical section (including parent-directory creation) runs under an exclusive flock on a per-root lock file kept outside the output keyspace, and the commit is fail-closed if the lock cannot be opened or acquired. It refuses symlinked final components and any key containing a colon (NTFS alternate-data-stream vector). Cross-host concurrent exactly-once to the same key requires the durable Enterprise committer. A root that is or contains the system temp directory throws InvalidArgumentException.

NextPDF\Pro\Stream\Commit\CommitReceipt

final readonly — jobId, OutputObjectKey $target, sha256, int<0, max> $bytesWritten, bool $idempotentReuse, DateTimeImmutable $committedAt. toArray() / fromArray() are fully round-trippable (the target is structured, not a lossy URI); fromArray() is strict and throws InvalidArgumentException on missing or malformed fields.

Checkpoint

NextPDF\Pro\Stream\Checkpoint\CheckpointStoreInterface

load(string $runId): ?RunCheckpoint and save(RunCheckpoint $checkpoint): void (durable and atomic — a reader never sees a half-written checkpoint).

NextPDF\Pro\Stream\Checkpoint\RunCheckpoint

final readonly — runId, int<0, max> $committedOffset, array $keyedState, DateTimeImmutable $updatedAt; SCHEMA_VERSION = '1.0'. Factories start(runId, at) and advancedTo(committedOffset, keyedState, at). toArray()/toJson()/fromArray()/fromJson() serialise it; fromArray() requires a non-empty run id and a valid updated_at, rejects an incompatible (non-1.x) schema_version, and normalises keyed state by dropping any non-JSON-serialisable values at every depth so recovered state is always re-serialisable. On recovery the processor fast-forwards past committedOffset and restores keyed state; state mutated after the last barrier is recomputed forward, never an error, because durable exactly-once comes from the committer’s digest dedup.

NextPDF\Pro\Stream\Checkpoint\FilesystemCheckpointStore

final readonly, implements CheckpointStoreInterface, DurableCapability. One JSON file per run, written atomically. Run ids must match [A-Za-z0-9._-]+ and contain no ..; a non-existent directory throws InvalidArgumentException.

Idempotency dedup

NextPDF\Pro\Stream\Dedup\IdempotencyStoreInterface

isCommitted(IdempotencyKey $key): bool, markCommitted(IdempotencyKey $key, CommitReceipt $receipt): void, receiptFor(IdempotencyKey $key): ?CommitReceipt. The fast path that short-circuits before rendering a replay; the committer’s digest comparison remains the durable guarantee, so a lost record at worst wastes a re-render the committer dedups.

• InMemoryIdempotencyStore — single-run / test scope (lost on crash).
• FilesystemIdempotencyStore — DurableCapability; one atomic JSON file per committed key (the serialized receipt), named by a hash of the key value. Marks are idempotent; a concurrent re-mark races harmlessly on one file. A non-existent directory throws InvalidArgumentException.

Retry and dead-letters

NextPDF\Pro\Stream\Retry\RetryPolicy

final readonly — positive-int $maxAttempts, positive-int $baseDelayMs, positive-int $maxDelayMs. __construct(int $maxAttempts = 3, int $baseDelayMs = 100, int $maxDelayMs = 30000) with invariants maxAttempts >= 1 and 1 <= baseDelayMs <= maxDelayMs <= 7 days (else InvalidArgumentException). Factories default() and none() (single attempt). shouldRetry(int $attempt): bool. delayMsForAttempt(int $attempt): int<0, max> is deterministic exponential backoff baseDelayMs * 2^(attempt-1) capped at maxDelayMs (no built-in jitter; apply at the call site).

NextPDF\Pro\Stream\Retry\DeadLetterStoreInterface

add(DeadLetterRecord $record): void, all(): list<DeadLetterRecord>, count(): int<0, max>.

NextPDF\Pro\Stream\Retry\DeadLetterRecord

final readonly — jobId, idempotencyKeyValue, positive-int $attempts, lastErrorCode, lastErrorMessage, DateTimeImmutable $failedAt, optional ?string $runId, optional int<1, max> $sourceOffset. dedupKey() is runId:sourceOffset when both are known, else the idempotency key value. fromArray() parses failed_at strictly as ATOM (rejecting relative or non-ATOM expressions) so serialise/deserialise stays symmetric.

• InMemoryDeadLetterStore — single-run / test scope.
• FilesystemDeadLetterStore — DurableCapability; one atomic JSON file per record, named by a SHA-256 hash of the dedup key (….dlq.json), so re-adding the same item on resume is idempotent. all() reads records in deterministic (sorted) order and surfaces a corrupt record by throwing; count() is a cheap file count, not a validity check.

Keyed state

NextPDF\Pro\Stream\State\KeyedStateStoreInterface

has, get, put, remove, clear, plus snapshot(): array and restore(array $snapshot): void for the checkpoint boundary. Values must be JSON-serialisable. For the default render-and-commit workload no keyed state is used; it exists for aggregation/windowing extensions. InMemoryKeyedStateStore is the single-run implementation; losing it on recovery is a semantic no-op for the default workload because exactly-once comes from the committer’s digest dedup.

NextPDF\Pro\Stream\State\KeySelector

final readonly, __construct(string $tenantField = 'tenant_id', string $documentField = 'document_id'). keyFor(RenderManifest $manifest): non-empty-string derives the partition key from manifest metadata as rawurlencode(tenant):rawurlencode(document) (the encoding stops ("a:b","c") colliding with ("a","b:c")), falling back to the job id when either field is absent — so every manifest resolves to a stable, non-empty key.

Durability marker and exceptions

NextPDF\Pro\Stream\DurableCapability is a marker interface for any store/committer whose state survives a process restart. A crash-safe run requires every collaborator to implement it, so it fails fast instead of promising exactly-once that an in-memory store cannot keep.

All subsystem exceptions implement NextPDF\Pro\Stream\Exception\StreamException (extends Throwable), so a caller can catch (StreamException) uniformly:

• RenderEngineException (RuntimeException) — executor violated the batch contract (unknown, duplicate, or missing unit; worker fault; timeout).
• CommitIntegrityException (RuntimeException) — declared sha-256 does not match the payload; spec code SPEC-COMMIT-422.
• OutputCommitConflictException (RuntimeException) — divergent bytes at an occupied key with overwrite disabled; spec code SPEC-COMMIT-409 (exposed via specCode()).
• UnsupportedTargetException (InvalidArgumentException) — target scheme a committer cannot service.

Conformance

The engine validates manifests against the Core manifest model and produces deterministic bytes plus sha-256 digests; the committer enforces atomic, integrity-checked, exactly-once writes. The module performs no cryptographic operations beyond sha-256 content digests and defines no FIPS-specific behavior.

Edge cases & gotchas

• renderBatch() never aborts on a per-manifest failure; inspect each EngineRenderResult.
• ProcessPoolRenderUnitExecutor correlates strictly by index and re-hashes worker bytes; a buggy worker hard-fails rather than corrupting output.
• LocalFilesystemCommitter is single-host; cross-host exactly-once needs the durable Enterprise committer.
• Crash-safe runs must use the DurableCapability (filesystem) stores throughout, not the in-memory variants.

Publication boundary

This page documents externally observable behavior and the supported public API surface only. Internal namespace paths, helper classes, mechanism tables, runbook filenames, and ticket prefixes are out of scope.

See also

• Stream
• Writer — Deep Reference
• Output Pipeline — Deep Reference