[jasonkuhrt]

Motivation

When a task fails, vp run cancels the whole run. Tasks that are still running are killed with exit code 137, no new task starts, and none of them gets a cache entry. In a recursive lint or test over many packages, the first failure hides every other package's result, and the killed tasks run again from nothing next time. #822 asks for an option to keep going.

This PR is the first step of #822. Tasks that depend on a failed task don't run and aren't listed, the same way fast-fail already treats the tasks it never started. Listing them as skipped, as #822 proposes, needs decisions about the reporter and the summary format, so it's left for a follow-up.

Part of #822.

flowchart TB
  subgraph today["Today: a failure cancels the run"]
    direction TB
    g1["db#generate<br/>failed"]:::failed --> a1["api#lint<br/>not run"]:::notrun
    g1 --> w1["web#lint<br/>not run"]:::notrun
    u1["utils#lint<br/>killed, exit 137, not cached"]:::failed
  end
  subgraph cont["With --continue"]
    direction TB
    g2["db#generate<br/>failed"]:::failed --> a2["api#lint<br/>not run"]:::notrun
    g2 --> w2["web#lint<br/>not run"]:::notrun
    u2["utils#lint<br/>passed, cached"]:::passed
  end
  classDef failed fill:#FCEBEB,stroke:#A32D2D,color:#791F1F
  classDef passed fill:#EAF3DE,stroke:#3B6D11,color:#27500A
  classDef notrun fill:#F1EFE8,stroke:#5F5E5A,color:#444441

Loading

Both runs exit with a non-zero status. Today utils#lint is reported as failed (137) and runs again next time. With --continue it is cached, and api#lint and web#lint aren't listed, as with fast-fail today.

Change

vp run gets a --continue flag (RunFlags::continue_on_failure). Without it, nothing changes.

With it, a failed task doesn't cancel fast_fail_token. Running tasks aren't killed, new tasks keep being scheduled, and tasks that succeed are cached through the existing path. Ctrl-C still cancels everything.

In the scheduler, execute_leaf, execute_node and execute_expanded_graph now return whether they failed. A failed node doesn't release its dependents, so their dependency counts never reach zero and they never start, directly or through other tasks. A node stops at its first failed && item. A task whose script runs vp run fails if any task in that nested graph fails, so its own dependents don't start either.

The exit status needs no change: once every task that can run has finished, the reporter already exits with the failed task's exit code for one failure, and 1 for several.

New exit_codes snapshot cases cover a task that finishes and is cached beside a failure and replays from the cache on the next run, dependents of a failure and of a failure in a nested vp run, a && chain that stops at its failure, and the exit code with two failures. docs/cancellation.md describes the flag in its Fast-fail section.

[limzykenneth]

I'm not sure why this behavior is guarded behind a --continue flag rather than being the default? Could it not have been that in the case where a task in the graph failed, nothing gets cached but succeeded tasks keep their succeeded status while tasks not depending on failed tasks continue to completion?

If the concern is that even when a task failed, its output may be the one contributing to other failures, simply not caching the entire graph when there is a failure, while reporting accurate exit codes for individual task seems to be a sensible default that provides the maximum amount of information.

[wan9chi]

I still prefer fail-fast by default simply because it gives faster feedback. Most of the time we just want to know anything (not everything) that goes wrong.

I also added a --no-bail alias to --continue just in case ai guesses --no-bail first. (pnpm uses --no-bail).

@fengmk2 @cpojer Do fail-fast by default and --no-bail aliasing --continue look good to you?

[jasonkuhrt]

Written by an AI agent (Claude) on behalf of @jasonkuhrt.

@wan9chi fail-fast by default and --no-bail as an alias both look right to me.

Fast feedback and the cache don't have to be traded against each other. Today vp run makes them a trade in two places.

A failure is reported only when the run ends. In this PR's own snapshot, failing exits 42 while the other tasks keep running. Its line, vtt barrier ../../.barrier sync 2 --exit=42, carries no mark. The run ends on 3/4 successful, with a pointer to --last-details. No log mode prints anything when a task exits non-zero (interleaved; labeled and grouped are the same). Under fail-fast, the early exit stands in for that line. Under --continue, nothing does.

The small fix is a line printed the moment a task fails, in every log mode, e.g. ✗ pkg-c#failing (exit code: 42). I can add it to this PR.

The full fix is a run event log. vp run appends one JSON line per event to a file next to last-summary.json as the run goes, and --json streams the same lines to stdout. A finished task's event carries its exit code, its cache status and the path of a file holding its own output:

{
  "event": "task_finished",
  "task": "pkg-c#failing",
  "exit_code": 42,
  "cache": "miss",
  "log": "<cache>/logs/pkg-c#failing.log"
}

An agent or a CI step that tails the file can start on a failure the moment it lands and read only that task's output, while the rest of the run finishes and is cached. Task output never shares stdout with the JSON, as the rule for JSON output asks. Fail-fast can stay the default for people at a terminal, and automation passes --continue without waiting longer for its first failure. The same records give this PR its missing piece: skipped tasks become task_skipped events naming the failure they waited on. They also give the JSON output that #791 left for later. Bazel's build event protocol works this way: --build_event_json_file streams events, and a test's event points at its test.log. I wrote it up in #923.

Fail-fast kills the tasks in flight (exit 137, not cached), so the next run starts them over. Make and Cargo fail fast without that (make: *** Waiting for unfinished jobs...., warning: build failed, waiting for other jobs to finish...). They start nothing new, and the jobs already running finish and keep their results. With the failure line, that mode reports a failure just as early; the exit comes later, by up to the remaining time of the longest running task. Would you consider it for the default?

[fengmk2]

Do fail-fast by default and --no-bail aliasing --continue look good to you?

LGTM.

Besides enabling it through a command-line flag, can it also be enabled in the task config?