Ryan Metz, founder of Roll Data, writes on why enterprise AI must validate decisions, not datasets.
Synthetic data enters most enterprise conversations through an appealing proposition: reproduce the useful patterns in sensitive data without reproducing the people or records behind it.
It also encourages the wrong technical question. Teams ask whether synthetic data resembles production data, then compare distributions, correlations, and similarity scores.
But enterprise work does not happen inside a distribution. It happens through relationships, rules, sequences, and decisions. A synthetic claims dataset can reproduce the frequency of every field while placing events in an impossible order. A financial dataset can look realistic row by row while violating an accounting identity. A customer dataset can preserve demographic correlations while removing the rare combinations most associated with fraud or service failure.
The dataset is statistically convincing and operationally false.
That distinction defines our position: synthetic data should be evaluated by whether it preserves the structure and decision behaviour of the work, not simply the appearance of the source data.
Fidelity has more than one layer
Most generators learn a probability distribution from source data using methods such as copulas, Bayesian networks, variational autoencoders, generative adversarial networks, or diffusion models.
An enterprise schema contains primary and foreign keys, permitted state transitions, temporal dependencies, policy thresholds, and domain rules. These are not merely statistical tendencies. An account cannot transact before it exists. A claim cannot precede the insured event. Assets must equal liabilities plus equity. If generation breaks those constraints, greater statistical realism may only make the error harder to see.
Enterprise validation therefore needs three layers. Distributional fidelity tests whether statistical patterns survive. Structural integrity tests whether relationships, sequences, and deterministic rules still hold. Decision fidelity tests whether a model or System of Work reaches materially equivalent outcomes when operating on synthetic and real data. Train-on-synthetic, test-on-real evaluation is useful here, but the final measure must reflect the actual task: fraud detected, claims routed correctly, or credit decisions remaining within policy.
Privacy complicates the design further. Building Roll Data has made that tension concrete: a generator can optimise privacy until it destroys task utility, or optimise utility until records sit unusually close to the source. Its public benchmark maps downstream LightGBM performance against P5 nearest-neighbour distance ratio across 23 datasets and four privacy settings. The broader lesson is that synthesis is a multi-objective optimisation problem, not a single quality score or reusable safety certificate.
The architecture should combine learning with rules
This leads to a specific technical stance. Enterprise synthetic data should be produced through a hybrid architecture: a probabilistic generator to learn variation, a deterministic constraint layer to preserve how the business operates, and a task-level evaluation layer to prove that the output remains useful and safe.
The constraint layer may draw on schema definitions, ontologies, policy rules, and valid workflow states. The evaluation layer should test privacy leakage, coverage of rare conditions, and downstream decision performance. Each release should remain tied to its source, generator version, configuration, intended use, and validation evidence. A new use case requires a new decision about fitness; “synthetic” is not a reusable safety certificate.
This is where synthetic data becomes infrastructure for Systems of Work. It gives an enterprise a controlled environment in which intelligent systems can encounter routine cases, rare exceptions, and deliberately adversarial conditions before their authority expands. Failures become new scenarios. Human corrections become repeatable evaluations. Over time, the organisation builds an owned library of the situations, boundaries and judgements its systems must understand.
Systems of Record preserve the reality the enterprise has experienced. Synthetic data allows Systems of Work to rehearse the realities the enterprise must be prepared to face.
About ToolShed
ToolShed is ConceptVines’ thought leadership series examining what it really takes to build, govern and scale AI inside the enterprise.
As AI moves from experimentation into core business systems, the hardest questions are no longer about what the technology can do. They are about where intelligence should sit, how decisions should be made, what enterprises should build versus buy, how autonomy should be governed, and where lasting value will accrue as models and capabilities continue to evolve.
ToolShed brings together perspectives from leaders across ConceptVines to explore these questions from different vantage points of AI architecture, engineering, enterprise strategy, economics, governance, security, data, transformation, and the changing nature of work.
Not predictions about where AI might go, but considered points of view on the decisions enterprises need to make as they build what comes next.
