[yileicn]

Summary

Agent-testing plugin: a new Template case type with turn files, plus a round of fixes to run state, cost reporting and AI case authoring.

Template cases and turn files

• A Template case calls one of the entry agent's templates directly, the way code such as a photo check calls it outside any conversation, so a template can be tested without routing, function calls or dialog history.
• Turns can attach files, which both Template and Agent cases pass on to the model, so vision templates can be tested with real photos. The Anthropic provider now sends URL-only images as URL image sources instead of dropping them.
• Saving a Template case requires entryAgentId and templateName, checks that the agent has that template (exact, case-sensitive match, as the runtime looks it up), and rejects history.

Run state

• Cancel sets CancelRequested with a single-field update on Pending or Running runs instead of replacing the run document, which could revert a finished run to Running or lose case counts. It returns 409 when the run already finished.
• UpdateRunAsync only replaces while the stored CancelRequested matches, so the executor's whole-document writes never undo a cancel; the executor also honours a cancel sent while the run was Pending.
• A case PUT without enabled keeps the current value instead of enabling an unreviewed draft.
• Abandoned runs are failed when read, not swept on startup. The startup sweep failed every Running run, including runs another host was still executing (several pods, or a local host sharing the database). Runs now carry UpdateDate, set on every write and by a 30s executor heartbeat while cases run; listing, reading or deleting a run fails it when it is Running with no heartbeat for 3 minutes. The heartbeat is independent of case completions, since PrepareAsync and the judge's provider call are not cut off by the case timeout. Failing is an atomic update conditioned on the run still being Running with the UpdateDate that was read, and sets only Status, Error and CompletedAt, so a stale read never overwrites newer progress or a finished status. Pending runs lost with a host stay Pending until triggered again.
• A cancel carried over by the executor's per-case write stops the next case; it was checked before that write and missed.
• Runs record PlannedCount (cases x models) when they start Running, so progress reads as finished / planned.

Cost reporting

• The queued run executor now gets ILlmProviderService, so the pricing snapshot is filled and the run page no longer shows unit cost as unknown when model pricing is configured.
• llmJudge spend is no longer billed to the model under test: the judge resolves its provider from the case's scope, so its tokens landed in the case's TotalTokens/Cost. It is now recorded as JudgeTokens/JudgeCost on each case result and summed per model in the performance summary.
• Each run records the judge model and its unit costs as JudgePricing, taken from the suite it executed with, since the suite's judge setting can change afterwards.
• The latency, token and cost summary is recomputed after every case instead of only at the end, so the run page shows spend while a run goes and a run whose host died keeps the figures for the cases that finished.

AI case authoring

• Mocks and tool assertions naming a function the entry agent lacks are kept with a warning instead of dropped: mocks match for every agent in the conversation, so a routed-to agent's function is valid.
• When the model rewrites turns without their files, the files are restored by position if the turn count is unchanged; otherwise a warning reports the loss.
• The author can declare templateName, so it can draft and repair a Template case.

UI

Matching UI changes: SciSharp/BotSharp-UI#494.

Test plan

• Unit tests: BotSharp.Core.UnitTests AgentTesting (369 passed)
• Cancel race, conditional replace, heartbeat and conditional abandon checked against a real MongoDB
• Anthropic URL image source against the live Claude API
• Run a Template case with image files end to end
• Run a suite with an llmJudge assertion; check JudgeTokens/JudgeCost on results and the summary

🤖 Generated with Claude Code

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[Copilot]

🟡 Changes recommended

Abandonment checks can overwrite newer run progress or misclassify still-executing cases as failed, potentially allowing live runs to be deleted.

7 open issues

• Persist the run's effective case timeout · New
• Prevent abandonment while case stages are still executing · New
• Prevent stale run snapshots from overwriting newer progress · New
• Match template names using the runtime's case-sensitive comparison · New
• Include templateName in case authoring and diff handling · New
• Propagate cancellation after each case update · New
• Support URL-only attachments in Anthropic vision cases · New

What changed in this PR

Extends the agent-testing plugin with direct Template cases and file attachments, alongside improvements to run management, cost reporting, and AI-assisted case authoring.

Changes:

• Adds Template case validation and execution, with attached files supported in both Template and Agent turns.
• Revises cancellation and abandoned-run handling, preserves omitted case-enabled values, and separates judge usage from model-under-test costs.
• Preserves routed-agent mocks and assertions with warnings, restores omitted turn files, and adds regression tests.

File	Description
CaseAuthoringTests.cs	Tests warning-based retention and turn-file restoration.
AgentTestRunExecutorTests.cs	Tests pending cancellation and separate judge totals.
AgentTestJudgeTests.cs	Updates the driver stub for files and Template calls.
AgentTestControllerTests.cs	Tests abandoned-run detection and deletion.
AgentTestCaseRunnerTests.cs	Tests Template execution, attachments, and judge accounting.
LlmCaseAuthor.cs	Restores files and retains unfamiliar tool names with warnings.
IAgentConversationDriver.cs	Adds file parameters and direct Template execution.
CaseValidation.cs	Validates required Template fields and rejects history.
BotSharpAgentConversationDriver.cs	Forwards attachments and invokes templates through InstructService.
AgentTestRunQueue.cs	Removes startup reconciliation and supplies pricing services.
AgentTestRunExecutor.cs	Honors pending cancellation and summarizes judge pricing and spend.
AgentTestCaseRunner.cs	Dispatches Template calls and separates judge usage.
AgentTestRepository.cs	Adds targeted cancellation writes and progress timestamps.
AgentTestRun.cs	Adds judge pricing, judge totals, and update timestamps.
AgentTestDtos.cs	Adds template names and nullable enabled updates.
AgentTestCaseResult.cs	Records judge tokens and cost separately.
AgentTestCase.cs	Defines Template cases and turn attachments.
AgentTestController.cs	Validates templates and updates cancellation and abandonment handling.

_{Review effort: Balanced}