Make AI and mission-critical system behaviour reproducible, gated, and defensible.
The BQS Reliability Platform applies the discipline we built for market systems to any system whose failure is expensive: pin the inputs, measure the behaviour, gate the regressions, and seal the evidence so a reviewer can check it rather than trust it.
Engagement-led · scoped per system · no black-box claims
Most AI reliability is asserted, not measured.
- Model and system behaviour changes between releases, and nobody can say exactly how.
- Evaluations are run once, by hand, and never reproduced.
- Failure modes are discovered by users rather than by the pipeline.
- Reviewers ask how a system was validated and receive a narrative, not evidence.
- Governance policy exists on paper but is not enforced anywhere in the build.
Five controls that turn behaviour into evidence.
Deterministic validation
Pin inputs, seeds, versions, and environment so an evaluation is a repeatable measurement rather than a one-off observation. Re-run it and get the same answer, or a recorded reason why not.
Behavioural regression gates
Express acceptable behaviour as thresholds — accuracy, drift, refusal rate, latency, output-schema conformance — and enforce them in CI. A change that breaches the gate fails before release.
Evidence packaging
Every validation run seals an artifact bundle: inputs, configuration, environment metadata, metric outputs, and gate results, hashed together so a reviewer can verify it was not edited after the fact.
Governance controls
Map internal policy and external obligations to concrete, checkable controls, and record which controls a given release satisfied. Governance becomes a build artifact instead of a document.
Runtime monitoring hooks
Detect divergence between validated behaviour and production behaviour: distribution shift, tail-latency drift, invariant violations, and silent degradation that aggregate dashboards hide.
Teams that have to answer for a system's behaviour.
A pipeline that blocks behavioural regressions instead of surfacing them post-release.
Evidence that a specific release satisfied specific controls, reproducible on demand.
An audit trail describing how system behaviour was verified, not asserted.
An independent way to validate reliability claims about a system being acquired.
A sealed bundle and a memo a reviewer can check.
Below is a redacted sample of the two artifacts an engagement produces: a findings excerpt with severities and a bundle manifest with hashes and a re-run command.
The primary model evaluation depends on unpinned dependency versions and an unseeded sampler, so two runs of the same commit produce different headline metrics.
Recommended · Pin dependency set and seeds, then re-baseline behaviour under a deterministic validation harness before any gate is agreed.
Aggregate accuracy is tracked at merge, but latency tails and refusal-rate shifts are unmonitored, which is where the observed production incidents originated.
Recommended · Introduce p99 latency and refusal-rate thresholds as CI gates with a recorded baseline per release.
Internal governance policy is documented but not linked to any checkable artifact, so a reviewer cannot confirm which release satisfied which control.
Recommended · Emit a per-release evidence bundle whose gate results map one-to-one onto the documented controls.
Client identifiers, workload names, and absolute figures redacted
| File | SHA3-256 |
|---|---|
| bench.jsonl | 9f2c1a7b4e0d63a8c5f1b92e7d40a6c3b8e51f79d2a4c6083b1e7f5c9a2d4e60 |
| metrics.prom | 4a8e0c92d7f31b6584a2c0e9fb7d5136c8492ae70df13b6a5c208e9147fd3b02 |
| environment.json | c73b1f9e4a086d52b7c1e930fa48d65b2c907e3145af8b60d29c15e7b3048fa9 |
| gates.json | 17d4b6e93c0a58f2b1e7d940c635a82f4b0e19d7c26a3f584b9e0c17d3a6f2b8 |
| manifest.yaml | e50a29c7b14d63f8a920c5e17b4d8f36a1c09e7d52b3f6480c19a7e2d5b306f4 |
# re-run and verify git clone <harness-repo> && cd harness ./scripts/run_validation.sh --config configs/sample.yaml --seed 42 sha3sum -a 256 -c out/manifest.yaml
Hashes shown are placeholders from a sample run, not a client artifact
Assessment first. Platform only where it earns its place.
One system in scope. Failure surfaces mapped, measurable signals identified, evidence gaps written up.
Deterministic validation harness stood up; baseline behaviour recorded and pinned.
Thresholds agreed and enforced in your pipeline; breaching changes blocked at merge.
Sealed bundles produced per run and mapped to your control and audit requirements.
- Scoped to one system so results are attributable.
- Findings written down, including what we recommend against.
- Artifacts you can forward to risk, audit, or a buyer.
- No claim we cannot reproduce on request.