Tech Lead |Software Architecture & Production Systems

What determinism actually gives you

It’s worth being precise about why determinism matters, because “AI is nondeterministic” is often treated as a technical footnote rather than a fundamental property with organizational implications.

Determinism gives you reproducibility. If a system produces a wrong answer, you can reproduce the exact conditions that led to it, trace the execution, and find the cause. Debugging is possible in any meaningful sense because you can make the problem happen again.

Determinism gives you testability. You can write a test that says: given this input, the output must be this. You can run that test a thousand times and trust that if it passes, the behavior is correct. The test is meaningful because the relationship between input and output is stable.

Determinism gives you auditability. When an enterprise process produces an outcome: an approval, a calculation, a classification, the determinism of the underlying system means you can reconstruct exactly how that outcome was reached. The audit trail is complete and verifiable.

Determinism gives you accountability. When something goes wrong in a deterministic system, you can trace it to a cause and a responsible component. The causality is clear.

AI systems don’t automatically provide any of these properties. That doesn’t make them worthless, they provide capabilities that deterministic systems can’t approach. But it means that enterprise teams deploying AI need to engineer for the properties that determinism used to provide automatically.

The gap in practice

I’ve been collecting examples of how the determinism gap manifests in real enterprise AI deployments. The failure modes cluster around a few patterns.

The irreproducible incident. A user reports that the AI system gave them wrong information. The engineering team tries to reproduce the issue. They send the same query. They get a different, and correct, answer. The incident can’t be reproduced, so it can’t be debugged. It’s marked as a one-off and closed. It happens again next month.

This is the debugging failure mode. Without logging that captures the complete context of every query, not just the input and output, but the retrieved documents, the exact prompt, the model version, the generation parameters, you can’t reconstruct the conditions that produced the failure. And without reproducibility, systematic improvement is impossible.

The passing test that fails in production. A RAG system is evaluated on a test set of two hundred queries. It performs well, 87% accuracy by the evaluators’ rubric. It’s shipped. In production, accuracy on the long tail of unexpected queries is significantly lower. The test set didn’t represent the actual query distribution.

This is the evaluation failure mode. Probabilistic systems require probabilistic evaluation, you can’t certify correctness by passing a fixed test suite, because the system’s behavior on unseen inputs is genuinely uncertain. Evaluation needs to be ongoing, not a one-time pre-launch gate.

The silent regression. The team upgrades to a newer version of the embedding model. Benchmarks improve. A few spot checks look good. The change ships. For a specific category of queries (ones that were at the boundary of the old model’s semantic understanding) retrieval quality silently degrades. Nobody notices for three weeks, when a pattern in user complaints becomes visible.

This is the regression detection failure mode. Deterministic systems can be regression-tested with high confidence: if the tests pass, the behavior is preserved. AI systems require statistical regression detection, comparing output distributions before and after changes, not just checking that specific test cases pass.

Engineering toward determinism where it matters

The right response to the determinism gap isn’t to pretend AI systems are deterministic, or to avoid using them in contexts that require reliability. It’s to engineer the properties that determinism used to provide automatically.

Reproducibility through comprehensive logging. If every query’s complete context: retrieved documents, full prompt, model version, generation parameters, raw output, is logged, you can reconstruct the conditions of any incident. This is the engineering substitute for deterministic reproducibility. It doesn’t make the system deterministic, but it makes failures debuggable.

Testability through distributional evaluation. Instead of a fixed test suite, build an evaluation framework that runs continuously on a representative sample of production traffic. Define acceptable performance ranges, not a pass/fail threshold but a distribution, and alert when the system’s output distribution drifts outside those ranges. This is the engineering substitute for deterministic testability.

Auditability through source tracing. In RAG systems, every output should be traceable to the retrieved sources that grounded it. This doesn’t replicate the determinism of traditional audit trails, but it provides the next best thing: a chain of evidence from output back to source. When a user asks “why did it say that,” you can answer with “it said that because it retrieved these documents and used this specific context.”

Accountability through system decomposition. When something goes wrong, the question “which component failed” requires that the components be separable in the logs. The retrieval step, the generation step, the post-processing step, each should have its own logged inputs and outputs. Monolithic logging that only captures the final output makes failure attribution impossible.

The mental model shift

The deeper issue is that the determinism gap requires a different mental model for how reliable enterprise AI systems are built and maintained.

Deterministic systems can be made correct. You define correct behavior, you test for it, you ship when the tests pass. The system is either correct or it isn’t.

Probabilistic systems have to be managed toward acceptable behavior. You define acceptable performance ranges, you measure continuously, you intervene when the system drifts outside those ranges. “Correct” is a distribution, not a state.

This is unfamiliar territory for most enterprise engineering teams. It requires different tooling, different processes, different organizational structures. The quality assurance process for a probabilistic system looks more like statistical process control than unit testing. The incident response process looks more like anomaly detection than error handling.

None of this is beyond the capability of competent engineering teams. It’s just different. And the teams that make the mental model shift explicitly, that acknowledge the determinism gap rather than engineering as if AI systems were deterministic — are the ones that build systems that actually hold up in production.

What Reliable Enterprise AI looks like, revised

I want to update the definition I offered in September, now that I’ve been thinking about the determinism gap specifically.

Reliable enterprise AI is not AI that never makes mistakes. It’s AI that makes mistakes in ways that are detectable, traceable, bounded, and improvable. The system’s behavior is understood well enough that failures can be caught before they cause harm, investigated when they occur, and corrected through systematic improvement rather than guesswork.

That’s achievable. It requires engineering discipline, evaluation infrastructure, comprehensive logging, and the organizational willingness to treat AI system quality as an ongoing process rather than a launch milestone.

It also requires acknowledging, explicitly and honestly, the difference between what these systems can and can’t guarantee. Probabilistic systems cannot guarantee any specific output. They can be engineered so that their outputs fall within acceptable ranges, reliably, for the use cases they were designed for. That’s a different guarantee than deterministic systems provide, but it’s a real one, and it’s enough for most enterprise use cases if it’s built and maintained honestly.

The determinism gap is real. Engineering around it is the work.

That’s a wrap on 2024. In January I’m going deeper on the implementation side: what reliable RAG looks like in code, not just in architecture diagrams. The framework has been useful for thinking. It’s time to stress-test it against building.