Greetings, traveler!
Your team adds a SwiftUI performance skill to its coding agent. The instructions tell the agent to inspect view updates, investigate identity problems, and measure the result before declaring an optimization successful.
A week later, the agent produces a convincing explanation for a slow screen. It replaces a few computed properties, adds Equatable, and reports that rendering should be faster. The app builds. The explanation sounds reasonable. But nobody has measured the screen, and the change may even have introduced stale UI.
How would you know whether the skill helped?
You could ask the team if the agent feels more useful. That feedback matters, but it cannot tell you which changes led to better outcomes. A repeatable evaluation gives you a way to compare the agent’s work under defined conditions, inspect failures, and decide whether a new configuration deserves wider use.
This article builds an evaluation around a SwiftUI performance skill. The same approach works for a model change, a revised system prompt, a new tool, or a different development workflow.
What an agent evaluation measures
An agent evaluation is a set of tasks, execution conditions, and assessment rules designed to answer a specific question about an agent’s behavior.
For a coding agent, one task might begin with a particular repository revision, a description of a slow screen, and permission to inspect and modify the project. The run ends when the agent submits a change or reaches a defined limit. The assessment then checks the resulting code, tests, performance evidence, and report.
This is broader than checking the agent’s final message. A correct explanation is useful, but it does not establish that the code works. A passing build shows that the project compiled, but says little about whether scrolling improved or selection still works.
Before designing the suite, make the decision explicit. For example:
Does the SwiftUI performance skill improve the rate at which this agent delivers correct, measured performance changes compared with the same agent without that skill?
That question defines the comparison. The model, tools, starting code, task description, and limits should remain as similar as practical. The skill is the intended difference.
An evaluation is also different from a regression test for the application. The application’s tests check its behavior. The evaluation uses those tests, along with other evidence, to assess the work produced by the agent.
Choose tasks that represent the work
A useful suite begins with real engineering problems. Select tasks from previous investigations, bug reports, or changes that the team is comfortable reproducing in a safe copy of the repository. Remove secrets and sensitive user data from any fixtures, logs, or traces.
For the SwiftUI performance skill, a small initial suite could cover:
| Scenario | Agent’s assignment | Evidence needed for acceptance |
|---|---|---|
| Excessive updates | Investigate a dashboard whose unrelated state changes trigger costly work in a child view. | Relevant update evidence, preserved behavior, and a measured reduction in the identified work. |
| Unstable identity | Fix a list where refreshed data causes rows to lose state or animate incorrectly. | Stable identity for the actual data model, plus checks for reordering, insertion, and retained row state. |
| Slow scrolling | Investigate hitches in a populated list on a specified device and build configuration. | Comparable traces or measurements and functional checks for images, selection, and pagination. |
| Slow launch | Investigate work delaying the first usable screen. | A defined launch interval, repeated measurements, and verification that deferred work still completes correctly. |
| No optimization warranted | Review a screen with no demonstrated performance problem. | A defensible decision to avoid speculative code changes, with the evidence and its limits explained. |
If every task requires a code change, an agent can improve its score by always changing something. Some investigations should end with a request for better data, a diagnosis outside the SwiftUI layer, or no change at all.
The launch task also tests the boundary of the skill. A SwiftUI performance skill may help investigate work on the first screen, but launch time can depend on networking, persistence, SDK initialization, or other subsystems. The agent should follow evidence rather than assume the cause belongs to its area of specialization.
Write each task as you would give it to an engineer. Provide enough information to reproduce the problem, but avoid describing the expected implementation. An evaluation that says “use EquatableView to prevent updates” mostly tests whether the agent can follow that instruction.
For each case, record the repository revision, setup procedure, inputs, expected behavior, constraints, and acceptance checks. Keep private evaluation checks and reference traces outside the agent’s accessible workspace where practical. The agent should know the product requirements; it does not need the exact scoring script or a ready-made answer.
A small set is a starting point, not proof of general performance. Add cases as you encounter failures across different screens, data sizes, and architectural styles.
Define success before running the agent
Define success for each kind of assignment before running the agent. A task that calls for a performance fix needs evidence of improvement and no unacceptable regressions. A diagnostic task can succeed with an evidence-based explanation even if the fix is outside the agent’s permitted scope. For a screen without a demonstrated problem, a justified decision to leave the code unchanged can be the correct outcome.
When a performance fix is requested, acceptance checks can include:
- The project builds with the specified scheme and configuration.
- Relevant functional tests pass, including tests held outside the agent’s editable workspace when feasible.
- The required user behavior remains intact under the scenario’s acceptance checks.
- The performance claim is supported by an appropriate measurement against the original revision.
- The agent accurately reports what it changed, what it checked, and what it could not verify.
A test failure, a missing measurement, and a performance regression are different outcomes. For a task requiring a measured fix, a test failure or confirmed regression can fail the run; inadequate measurement may leave the performance outcome unverified. Record each outcome separately. A single pass/fail value is useful for comparing runs, but the underlying reasons tell you what to improve.
For each task that requires improvement, define a measurable target before comparing configurations. A scrolling task might require the chosen hitch metric to improve by a specified amount without functional regressions. Set the threshold using a pilot that establishes measurement noise and a difference meaningful to the team. If noise is larger than the expected improvement, collect better measurements before treating a small difference as a win.
Not every important property can be reduced to an automated assertion. An engineer may need to review whether the change narrows an observation dependency, introduces unreasonable complexity, or simply moves expensive work elsewhere. Use a written rubric and ask reviewers to record evidence for their decisions. Where possible, mask the configuration during review so that the presence of a skill does not influence the assessment.
Establish a baseline and compare like with like
The baseline is the configuration you already use. For the skill experiment, run the same agent on the same tasks without the skill. Then run the candidate with the skill enabled.
Two comparisons answer different questions. If the skill is explicitly loaded in every candidate run, the comparison tests how its instructions affect the work when available in context. If the agent must discover and select the skill itself, the result also reflects activation: a failure to load a relevant skill is part of that configuration’s outcome. Keep both sets of results separate, and verify loading from runtime records when the agent exposes them. If it does not, record activation as unknown rather than inferred from the answer.
Keep a record of the conditions that can affect the result:
- Model and agent versions, system instructions, and tool availability.
- Skill version and whether it was actually loaded or invoked.
- Starting commit, dependencies, Xcode and OS versions, device or simulator, and build configuration.
- Task prompt, time and token limits, network conditions, and permitted interventions.
- Commands, exit statuses, logs, test reports, measurement artifacts, and final diff.
Use an independent clean workspace for every run. Reset application data and fixtures as required by the scenario. An agent that starts with another run’s cache, modified source, or trace files is no longer solving the same problem.
Agent behavior varies between runs, so repeat each task for both configurations where budget permits. Pair runs by task, randomize execution order, and use the same limits. For hardware measurements, alternate baseline and candidate measurements to reduce the effect of thermal state or other time-related drift. A random seed can be useful metadata when supported, but it does not guarantee an identical run.
Separate environment failures from agent failures. If the build service is unavailable or the simulator cannot boot, mark the run as an infrastructure error. If an independent measurement remains too noisy to establish the outcome, classify that criterion as inconclusive and explain whether it affects the overall result. Define the retry policy in advance and apply it consistently. A task that times out because the agent spends its entire budget investigating is normally an agent outcome, not a reason to quietly discard the result.
Record more than one number
A basic outcome metric is the success rate per valid run:
run success rate = successful runs / valid attempted runsA valid attempted run includes failures and timeouts attributable to the agent. Report infrastructure errors and inconclusive outcomes separately, together with the number of attempted and valid runs. State in advance whether an inconclusive run is included in the denominator; report its count either way. Always state the denominator.
This metric weights each run equally. If some tasks have more repetitions, they contribute more to the aggregate. Use the same number of valid runs per task or report each task’s success rate and an equally weighted average across tasks. A different metric, the fraction of tasks with at least one successful run, answers a different question and rises as the number of attempts increases.
For illustration, imagine six tasks run three times per configuration. The baseline passes 9 of 18 valid runs; the skill passes 12 of 18. These are invented numbers, not an experiment. They suggest a possible improvement in this small suite, but three additional passes are too little evidence for a broad conclusion on their own. Inspect the task-level results: the skill may help with list identity while making launch investigations slower or less reliable.
Several other measurements help explain the result:
| Measure | What to record | How to interpret it |
|---|---|---|
| Functional correctness | Build and test results, plus review findings | A faster screen with broken selection is a failed change. |
| Performance | Baseline and candidate values, distributions, device and method | A claimed improvement needs measurements from comparable conditions. |
| Completion time | Wall-clock time, including retries and review | A quick answer that leaves substantial work for the engineer may not save time. |
| Cost | Model and tool costs, as well as measurement and infrastructure cost when available | Compare total cost per accepted task, not just cost per run. |
| Human intervention | Number, type, and time spent on interventions | A clarification, a correction, and a manual fix are different kinds of help. |
| Change quality | Scope, maintainability, and reviewer findings against a rubric | Smaller diffs can help review, but size alone does not establish quality. |
| Reporting accuracy | Claims checked against artifacts | A report should not imply a test or profile was run when it was not. |
Report both successful and unsuccessful runs when calculating cost and time. If the candidate fails more often, the cost of retries and manual recovery belongs in the comparison. For small suites, show individual results and uncertainty instead of presenting a percentage with misleading precision. Averages can hide a task that consistently fails; medians and per-task distributions are often more informative for time and performance.
Subjective feedback still has a place. Ask reviewers whether the output was easy to understand or whether the agent’s investigation was useful. Keep that feedback separate from factual checks, and describe the rubric so another reviewer can interpret it.
Measure the application, not the explanation
Consider the scrolling scenario. Both configurations receive the same repository revision and a reproducible dataset. The task asks them to investigate hitches when scrolling through a populated screen, preserve existing behavior, and report the evidence behind any change.
A reasonable setup might use an optimized build on a specified physical device, a fixed dataset, and the same repeatable interaction for each recording. Before the agent changes code, collect several baseline measurements. Afterward, run the same interaction against the candidate revision and collect the same measurements again. An independent evaluator should retain enough setup information and artifacts to repeat the important measurements rather than rely only on the agent’s report. Keep the device, OS, build settings, and measurement procedure consistent. Account for warm-up, thermal conditions, background activity, and the normal variability of the scenario.
Instruments can help identify long or frequent SwiftUI updates, time spent on the main thread, and hitches. Available instruments and features depend on the Xcode and OS versions; record them and choose a supported measurement method. The appropriate metric depends on the symptom. A reduction in view body evaluations can support a diagnosis, but the user-facing outcome still needs its own measurement. Equally, a smoother trace in one recording is not enough to conclude that a change consistently improves scrolling.
Check the behavior affected by the implementation. If the agent changes row identity, exercise insertions, deletions, reordering, navigation, and any row-local state. If it introduces caching, verify invalidation when the underlying data changes. If it moves work off the main actor, examine isolation and cancellation as well as performance.
When the environment cannot produce a trustworthy measurement, mark the performance criterion as unverified. The agent may still submit a useful diagnosis or a proposed change, but the evaluation should not count an unmeasured improvement as a confirmed success.
A practical evaluation in an existing project
The following example is a template for one task, not a recorded experiment. Adapt the scheme, test targets, device, and paths to your application. It assumes a reproducible list screen and a test environment that can load the same dataset on each run.
Give both configurations the same assignment:
Investigate the intermittent scrolling hitch on ActivityList.
Use the Performance fixture with 2,000 activities and follow the
scrolling procedure in the project’s test documentation.
Preserve selection, navigation, pagination, and row state.
Report your diagnosis, any code changes, checks performed, and
measurements or limitations. Do not claim an improvement without
before-and-after evidence.
Before running either configuration, an evaluator checks that the fixture and interaction reproduce a measurable issue. The baseline uses the agent without the skill; the candidate uses the same agent with the SwiftUI performance skill explicitly loaded. Each starts from the same commit in a separate clean checkout. Record the task as a case-specific contract:
case_id: activity-list-scroll-01
starting_commit: <commit-sha>
scheme: MyApp
configuration: Release
fixture: Performance-2000
interaction: docs/performance/activity-list-scroll.md
required_behavior:
- selection and navigation work
- pagination and row state are preserved
acceptance:
build_and_functional_tests: pass
performance: compare repeated measurements against baseline
reporting: claims match the retained evidenceThe example does not specify a numeric performance threshold. Establish one from a pilot on the chosen device before the comparison and record the metric, units, number of repetitions, and acceptable noise. If the issue cannot be reproduced reliably, revise the setup or mark the performance outcome inconclusive.
For an illustrative local build and simulator test, replace the placeholders with an existing workspace, scheme, and installed destination. These commands check compilation and behavior; they do not measure scrolling performance:
xcodebuild build \
-workspace MyApp.xcworkspace \
-scheme MyApp \
-configuration Release \
-destination 'generic/platform=iOS Simulator' \
CODE_SIGNING_ALLOWED=NO
xcodebuild test \
-workspace MyApp.xcworkspace \
-scheme MyApp \
-destination 'platform=iOS Simulator,id=<simulator-udid>' \
-resultBundlePath /tmp/activity-list-tests.xcresultChoose a new result bundle path for each run. A real project may require package resolution, a specific test plan, additional build settings, signing, or a different destination. Run performance measurements separately with an optimized build on the selected device and the same documented interaction. Preserve traces and let an evaluator inspect or repeat them independently.
A compact run record can sit alongside the logs outside the agent’s editable checkout:
case_id: activity-list-scroll-01
variant: skill_loaded # alternative: no_skill
attempt: 1
starting_commit: <commit-sha>
agent_and_model: <versions>
skill_revision: <commit-sha-or-none>
activation: confirmed # or not_loaded / unknown
build: pass
functional_tests: pass
performance: inconclusive
reason: measurement_variation_exceeded_pilot_threshold
artifacts:
- build.log
- tests.xcresult
- performance-trace.trace
- change.diff
human_interventions: 0
elapsed_seconds: 840
estimated_cost_usd: 1.20All values are illustrative. The example records an inconclusive performance result despite passing tests: it cannot establish that scrolling improved. The evaluator may still score the diagnosis and reporting independently. Compare repeated runs task by task before summarizing success, time, cost, and human effort.
Suppose the development runs reveal that the agent repeatedly reports an improvement after checking only the build. A narrowly scoped skill revision might add:
For a performance claim, identify the metric and retain comparable
before-and-after measurements. If measurement is unavailable or
inconclusive, report that limitation and leave the claim unverified.
Run the revised skill against both the failure case and cases it previously handled well. Select a candidate using the validation set, then assess it on held-out cases. Do not promote the revision solely because it passed the case that prompted the change.
Turn failures into changes to the skill
After the first comparison, inspect the failed runs before editing instructions. Suppose several runs produce reasonable code changes but skip measurement. The observed problem is specific: the agent makes a performance claim without collecting evidence. A targeted revision could instruct it to establish a baseline, repeat the same measurement after the change, and state explicitly when measurement is unavailable.
That instruction should remain conditional on the tools and environment. Requiring an Instruments trace for every task is impractical when the agent cannot access the required device or automation. The skill can instead tell the agent to describe the manual measurement needed and avoid claiming an improvement.
A concrete revision cycle looks like this:
- Group failures by behavior and inspect the traces or diffs behind them.
- Form a hypothesis about which instruction could address one failure pattern.
- Change a small part of the skill and keep the previous version available.
- Rerun development cases, including cases the old version already passed.
- Compare outcomes, cost, and unintended changes before promoting the candidate.
A longer skill is not necessarily a better one. Additional instructions consume context and can conflict with a task or distract the agent from evidence. If a revision improves one scenario by forcing a particular technique, check whether it also pushes that technique onto cases where it does not apply.
Keep a separate set of unseen cases
Repeatedly editing a skill against the same few tasks can produce a skill that knows the benchmark rather than one that generalizes to the team’s work. Split cases by role:
- Development set: inspect freely and use to diagnose failures and propose revisions.
- Validation set: compare candidate versions and choose among them.
- Held-out test set: reserve for a final assessment after selecting the candidate.
If several tasks share the same screen, root cause, or code pattern, keep the whole group in the same split when possible. Otherwise, an apparently unseen case may be nearly identical to a case used during improvement. With a small suite, strict splitting leaves little data in each group. Treat early results as exploratory and add new cases over time.
The held-out set loses its independence once its results repeatedly influence decisions. Refresh it with new cases and document when that happens. A final evaluation does not guarantee success on future work; it provides a more credible estimate than the cases used to write the skill.
Where an LLM judge can help
Some qualities are difficult to check with scripts. A language model can help classify a report, flag unsupported claims, or apply a rubric to a proposed change. It can also help reviewers find runs that need closer inspection.
Its judgment is still another model output. It may favor fluent explanations, overlook a subtle behavioral regression, or infer that a command succeeded from an agent’s description. A judge should receive the actual evidence it needs: the diff, relevant requirements, test results, and measurement summaries. Ask it to identify the artifact supporting each finding and allow an “insufficient evidence” result.
Calibrate the rubric against a sample reviewed by engineers, including difficult disagreements and deliberately incomplete reports. Check both false positives and missed defects. For high-impact changes, keep a human reviewer responsible for the decision. An LLM judge can supplement deterministic tests and performance measurements; it cannot replace evidence that those checks were actually performed.
Automating instruction improvement
Once you have a repeatable evaluation, an agent can propose changes to a skill itself. The agent receives failure examples and evaluation feedback, drafts a candidate, and runs the development suite. A separate process compares the candidate with the previous version. The candidate is retained only if it meets the chosen criteria and does not introduce unacceptable regressions.
That loop can be useful, but the word “self-improving” describes the workflow rather than a guarantee. The system is optimizing against the feedback it receives. If the evaluation rewards confident explanations or an easy-to-game proxy, the resulting skill can become better at satisfying that metric while doing worse engineering work.
DSPy provides a framework for building and optimizing language-model programs. GEPA is an optimizer available in that ecosystem: it uses feedback about program behavior to propose and evaluate revised instructions. Rich feedback can explain a failure more effectively than a score alone. In a coding-agent setting, connecting such an optimizer to a skill requires an adapter that runs tasks, gathers artifacts, computes feedback, and installs each candidate in the correct place. GEPA does not automatically understand your repository, establish performance measurements, or guarantee that an optimized instruction will generalize.
Use separate data for proposing and selecting candidates, and keep a final test set outside the optimization loop. Candidate selection itself can overfit a validation set after enough iterations. Set budgets for model calls, execution time, and tool costs; coding-agent evaluations that build and profile an app can be expensive.
Skill maintenance is a related but separate concern. Systems such as Hermes can create and revise skills during use, and its Curator feature addresses the accumulation of generated skills. Maintaining a skill catalog does not establish that agents solve tasks better; that still requires evaluation. Usage information can help identify stale or overlapping instructions, but frequency alone does not establish quality: a rarely used incident-response skill may still be valuable. Review proposed merges, deletions, and instruction changes against the cases they are meant to support.
For a team skill, keep a versioned history, inspect the diff, run regression cases, and define who can approve a new version. Treat automatically generated instructions and their supporting files as code with a trust boundary: they can influence tool use and may contain mistakes or unsafe directions.
From a local experiment to a team practice
You do not need a large evaluation platform to start. A small set of carefully documented tasks, clean workspaces, a results spreadsheet, and stored artifacts can answer the first comparison. Automate setup and scoring where repetition makes manual work unreliable or expensive.
A practical adoption process might begin with a handful of recurring investigations. Run the baseline and candidate, examine each diff and measurement, and publish the limitations alongside the result. If the candidate helps, try it on new tasks before making it the team’s default. If it fails, preserve the failed run as a regression case only after removing sensitive data and deciding whether it belongs in the development or held-out set.
The outcome may be that the skill improves reporting but does not improve code, or that it helps on large lists while increasing cost on simpler tasks. Those results can guide where the team uses the skill and where it needs more evidence.
It might be interesting:
- Agentic Engineering in Teams: Building Practices People Can Trust
- Cognitive Debt: Keeping Your Codebase Understandable in the Age of AI
- Agent Security: Giving AI Useful Access Without Giving It the Keys
- Long-Running Agents: How to Keep Autonomous Work Reliable
- Multi-Agent Development: When Parallel Work Helps and How to Keep It Under Control
