A single benchmark score can tell us whether an AI model answered a controlled set of questions correctly. It cannot tell us whether a complete application will help a person complete a messy task, recover from an ambiguous instruction, resist manipulation, or behave consistently when its tools and data change. As generative AI moves from demonstrations into everyday software, evaluation has to move with it.
The U.S. National Institute of Standards and Technology is testing a broader approach through its Assessing Risks and Impacts of AI program, known as ARIA. Its pilot combined model tests, adversarial red teaming, and field testing with people. The important idea is not that one government test can declare an application safe. It is that useful evidence must connect technical performance to the context in which people actually use the system.
Benchmark Scores Answer Narrow Questions
Benchmarks are valuable because they make repeatable comparisons possible. A developer can run the same question set across models, measure a code task, or check whether an image detector identifies a known manipulation. Controlled tests are especially useful during development, when teams need quick feedback after changing a model, prompt, retrieval system, or safety control.
The limitation is that a benchmark simplifies the world. Its instructions are fixed, the expected output is defined in advance, and the data may not resemble a user’s actual environment. A model can also become indirectly familiar with a public test through training data or repeated optimization. A high score may therefore reflect skill on the test format without proving that the surrounding product is dependable.
This resembles the problem discussed in our guide to evaluating technology beyond marketing claims. A number has meaning only when the method, use case, comparison, and limitations are visible.
ARIA Evaluates Applications, Not Just Base Models
NIST’s ARIA 0.1 pilot evaluated seven AI applications submitted by five organizations. The pilot used three scenarios: avoiding television spoilers, planning meals, and helping a user navigate a fictional environment called Pathfinder. These were not intended to represent every use of AI. They were controlled settings for exploring how different layers of evaluation work together.
An application includes more than a model. It may add a system prompt, a retrieval database, content filters, memory, tools, and an interface. Two products built on the same foundation model can produce different outcomes because those components shape what the user sees and what the system can do. Conversely, a strong base model can be weakened by poor instructions, stale data, or a confusing interface.
That system view is particularly important for the AI agents that can take actions across software. An evaluation of the model’s prose does not measure whether a tool call used the correct account, whether a retry duplicated an action, or whether the user understood what would happen before approving it.
Three Levels Reveal Different Failures
The ARIA pilot used model testing, red teaming, and field testing. Model testing examines controlled inputs and outputs. It can measure whether responses satisfy a defined criterion and can be automated across many cases. This layer is useful for coverage, but it may miss strategies that users adopt over a longer interaction.
Red teaming deliberately searches for weaknesses. Testers vary instructions, exploit ambiguity, introduce conflicting information, or try to bypass safeguards. The goal is not to produce one dramatic failure screenshot. A useful red-team program records the conditions that triggered the behavior, checks whether it can be reproduced, and connects it to a realistic harm or operational consequence.
Field testing puts the application in the hands of people performing a task. Testers may misunderstand the system, trust it too much, ignore useful warnings, or discover a workflow that designers did not anticipate. Questionnaires and interaction records can reveal whether the application is genuinely usable and whether users can recognize its limits.
These layers are complementary. A controlled test can isolate a technical behavior. Red teaming explores unexpected paths. Field testing shows how the system and the person adapt to each other. Passing one layer does not cancel a failure in another.
Measurement Trees Connect Claims to Evidence
The NIST pilot used measurement trees to break a broad quality such as validity into claims that can be supported by observable evidence. This helps prevent vague statements such as “the assistant is reliable” from becoming the end of the analysis. A team instead has to define reliable for a particular task, identify what could go wrong, choose measurements, and explain the thresholds used for a decision.
For a meal-planning application, for example, relevance, instruction following, consistency, and the handling of constraints may all matter. The importance of each measure depends on the context. A harmless preference is different from a medically significant dietary restriction. An ordinary consumer evaluation should not imply that a general AI application provides professional medical advice.
Measurement trees also expose gaps. If a product claim has no testable evidence beneath it, reviewers can see that the claim remains an assumption. If one metric dominates the tree simply because it is easy to calculate, the team can ask whether it is measuring what users actually need.
Real-World Testing Needs Guardrails
Field evaluation does not mean releasing an unfinished system without controls. Testers need informed instructions, defined data handling, a way to report problems, and limits on actions that could affect other people or live systems. Sensitive scenarios may require synthetic records, isolated accounts, or a simulation rather than production access.
Privacy is part of the measurement design. Conversation logs can contain personal information, confidential documents, or details inferred by the system. Teams should collect only what the evaluation requires, restrict access, set a retention period, and separate research data from ordinary product analytics.
Provenance also matters when an evaluation uses generated or edited media. As our article on Content Credentials explains, a signed history can help identify where an asset came from, but it does not establish that the claim inside the asset is true. Evaluators still need to inspect the content and task outcome.
What Product Buyers Should Ask
A vendor’s evaluation should identify the exact application version, model, tools, data sources, and settings that were tested. Results from a base model do not automatically transfer to a customized product. Buyers should ask whether testing included their language, workflow, user population, and foreseeable failure conditions.
Useful reporting includes the distribution of outcomes, not only an average. A system that works most of the time but fails badly on one subgroup or rare instruction may require a different deployment boundary. Reviewers should also look for human-correction rates, refusal quality, recovery after a tool failure, and whether users can tell when the system is uncertain.
No evaluation remains current forever. Models, prompts, retrieval indexes, dependencies, and policies change. Teams need regression tests and a record of which evidence supports each release. A major update should trigger targeted field and adversarial testing rather than inheriting the old product’s claims.
Limits of the Current Evidence
ARIA 0.1 was a pilot with seven applications and three constructed scenarios. NIST presents it as a method-development exercise, not a universal certification or a ranking of the entire AI market. Its lessons need to be adapted for domains with different risks, users, and regulations.
Human testing also introduces variability. People bring different expectations and skills, while a short study may not reveal habits that emerge after months of use. Qualitative feedback can explain why something failed, but it should be analyzed systematically rather than selected to support a preferred story.
What to Watch Next
NIST’s broader Generative AI Evaluation Program now includes text, image, and code challenges, while the AI Risk Management Framework connects evaluation to ongoing governance. Watch for shared scenario libraries, clearer reporting formats, stronger tests for agentic systems, and methods that compare field results without exposing private user data.
The direction is healthy: model scores remain useful, but they become one instrument inside a larger evaluation. The most credible AI products will state what was tested, where the evidence applies, what failed, and which decisions still require a person.
Leave a Reply