The Compliance Officer's Guide to Not Being Fooled by a Vendor Demo

AI Procurement & Governance  ·  June 2026  ·  8 min read

Synthetic data, cherry-picked metrics, and back-tested performance figures are the three most common ways AI vendors present a compelling case that does not survive contact with your actual environment. Here is how to ask the questions that cut through the noise — and what regulators now require you to ask.

Why Vendor Demos Are a Compliance Risk, Not Just a Procurement Challenge

Procurement teams have always known that a vendor demo shows a product at its best. What is newer — and materially more consequential — is that regulators now treat the failure to scrutinise AI vendor claims as a compliance failing in its own right, not merely a commercial miscalculation.

In September 2024, the US Department of Justice updated its Evaluation of Corporate Compliance Programs (ECCP) guidance — the document prosecutors use when assessing whether a company's compliance programme is adequate during a criminal enforcement investigation. For the first time, the ECCP explicitly requires prosecutors to assess how organisations "proactively identify, assess, mitigate, and manage the risks associated with their use of emerging technologies, including AI," and how companies "curb any potential negative or unintended consequences from the use of technologies, both in their commercial business and in their compliance programme." In plain terms: if an AI tool you procured caused harm because you failed to adequately scrutinise the vendor's claims, that failure is now part of your compliance exposure.

Separately, under the EU AI Act (Regulation 2024/1689), deployers of high-risk AI systems carry independent legal accountability that cannot be transferred to a vendor. Article 13 requires that high-risk AI systems be designed to ensure sufficient transparency for deployers to correctly interpret outputs. Article 26 places direct obligations on deployers to verify that systems are used as intended and to monitor for performance changes after deployment. Crucially, as legal guidance published in early 2026 has made clear, deployers of third-party AI tools cannot inherit provider compliance — each party in the AI value chain is independently accountable for their specific deployment context.

The compliance officer sitting in a vendor demo is, therefore, not just a technical evaluator. They are the organisation's first line of defence against a procurement decision that could later be judged inadequate by a regulator.

The Three Red Flags in Every AI Demo

Three specific presentation techniques consistently appear in AI vendor demos and consistently obscure the information a compliance professional most needs to make a sound evaluation.

1. Synthetic data as proof of performance. Synthetic data — artificially generated datasets designed to mimic real data — has legitimate uses in AI development, particularly where real data is scarce, sensitive, or subject to privacy constraints. However, it is also routinely used in vendor demos to show performance results that were never validated on the kind of messy, incomplete, context-specific data your organisation actually holds. When a vendor presents accuracy rates, false positive rates, or bias metrics, the first question is always: what data was used to generate these results? Synthetic data that was generated by the same team that built the model can produce self-reinforcing performance figures that bear no relation to how the system will perform in your environment. California AB 2013, effective January 2026, now requires generative AI developers to publish high-level training data summaries disclosing whether synthetic data was used — a regulatory signal that synthetic data provenance is no longer a technical footnote but a material disclosure item.

2. Cherry-picked metrics. AI systems can be made to look accurate, fair, or robust depending entirely on which metric is chosen and which population it is measured against. A fraud detection model might report 98% accuracy overall while performing significantly worse on specific customer segments that happen to be protected classes under applicable equality or non-discrimination law. A model might report strong precision while concealing a high rate of false negatives that would matter enormously in your use case. The NIST AI Risk Management Framework (AI RMF 1.0) specifically notes, in its Measure function, that "the risks or trustworthiness characteristics that will not — or cannot — be measured are properly documented." A vendor who presents a single headline metric without documenting what was not measured, or why, is not meeting the standard that regulators expect deployers to apply in their own due diligence.

3. Back-tested performance. Back-testing — running a model against historical data to show how it would have performed in the past — is a standard evaluation technique, but it is also one of the most routinely abused. A model that was designed, tuned, and selected after reviewing historical outcomes will almost always perform well on that same historical data. This tells you very little about how the model will perform on future data, in a changed regulatory environment, or when the population it is applied to shifts in composition. The gap between back-tested performance and live performance is a known and documented failure mode in AI deployment, and it is specifically what the NIST AI RMF's Measure function asks organisations to assess through independent validation and out-of-sample testing.

What Good Vendor Due Diligence Actually Requires

The questions below are not optional extras for technically sophisticated teams. They are the baseline standard implied by the regulatory frameworks now in force across the EU, the US, and increasingly other jurisdictions through ISO/IEC 42001, the first internationally certifiable standard for AI management systems.

The Accountability Question Sits with You, Not the Vendor

Perhaps the most important shift in the current regulatory environment is the clarification that deployer accountability does not diminish because a vendor made impressive claims in a sales process. The DOJ's updated ECCP guidance asks prosecutors to assess how companies "ensure technology is used for its intended purposes" and "how accountability over the use of AI is enforced." It does not ask whether the vendor's brochure was convincing.

Similarly, the EU AI Act places the responsibility for verifying that a high-risk AI system is suitable for its intended purpose squarely on the deployer. A vendor demo, however polished, does not constitute that verification. What does constitute it is documented due diligence: written questions, written answers, independent validation where warranted, and a record of the evaluation process that can be produced to a regulator if required.

The compliance officer's role in an AI procurement process is therefore not to be impressed by what a vendor can demonstrate. It is to create a documented record of what the vendor can prove — and to understand, with precision, the difference between the two.