NVIDIA’s evaluator gives agent skills a three-part review
NVIDIA’s SkillEvaluator reached its initial public release on August 5, 2026. The open-source tool separates skill review into three tiers: deterministic validation for quality and safety issues, semantic checks for overlapping guidance, and live evaluation that compares agent work with and without a skill. On August 20, NVIDIA researchers also posted a paper on continuous evaluation of skills, describing paired trials and a measured “skill lift.” The code and the research make a useful point for businesses adopting reusable instructions: a skill should be reviewed as an executable part of the agent workflow, not treated as harmless documentation.
This does not mean a tool can certify a skill as safe or universally useful. Static checks only find issues covered by their rules. A live comparison depends on the tasks, agent, environment, and evaluator. A positive result in one setup does not prove the same skill will improve every team’s work. The value is a more inspectable decision before rollout, with evidence that can be reviewed and repeated.
Separate safety checks from usefulness checks
Start with the first question: what does the skill contain and ask the agent to do? Review its scripts, external destinations, requested permissions, and assumptions. A scanner can help identify structural or security concerns, but a clean report is not a substitute for reading the source and understanding the skill’s access. Do not give a new skill broad credentials merely because a scan completed without a warning. Quarantine unfamiliar files, inspect what they can run, and test them without production data.
The overlap tier addresses a different problem. Several skills may repeat instructions or compete to guide the same task. That can make it harder to know which guidance the agent will use and can waste context. Similarity findings should prompt a human review rather than automatic deletion: two skills may use similar language but apply to distinct roles, systems, or approval boundaries.
The live tier asks whether the skill improves a defined task. Run matched tasks with and without the skill using the same model, prompt, permissions, and environment. Include cases where the skill should activate and negative cases where it should stay out of the way. Compare correctness, task completion, security behavior, and wasted work. A skill that improves one dimension but weakens another needs a deliberate tradeoff, not a single headline score.
Build a small evaluation before installing broadly
A small business can begin with one non-sensitive workflow and a short set of representative examples. For a customer-support drafting skill, include a straightforward request, a missing-information case, a request outside the skill’s scope, and a case that should be handed to a person. Use fictional data and no real customer accounts. Have a reviewer score whether the result is correct, whether the instructions activated only when appropriate, and whether the agent stayed within its permissions.
Keep the skill version, task set, model, and evaluation setup together. If the skill changes, rerun the same examples before adopting the update. Save the failed examples as well as the successful ones; otherwise the evaluation can drift toward showing only where the skill works. If you use an automated judge, inspect a sample of its decisions and keep a human responsible for high-impact tasks.
Make the go or no-go decision specific
Before rollout, decide what evidence would justify keeping the skill. It might need to improve a target task without increasing errors, triggering on unrelated work, or expanding access. If the available tests cannot measure those conditions, narrow the skill or defer broader use until you can. A benchmark is most useful when the criteria are tied to a real business consequence, not when a dashboard produces a score without context.
The practical next step is to choose one skill that matters to a repeated workflow and run a controlled, fictional-data evaluation. Review safety and overlap separately from live usefulness, then document the model, skill version, task examples, and reviewer decision. Treat the result as evidence for that exact setup, and repeat the check when the workflow or skill changes.