Skip to content

Commit f2ad92f

Browse files
feat(heavy): let long jobs yield, go round by worktree, and backfill
A long job held the solo lock until it ended: a 200 run balance batch, a nightly, or a three profile splash run kept every other agent waiting, and one worktree's back-to-back jobs could hog the queue. A waiting Timing job also made every later Shared job wait behind it, even when the Shared job fit the budget beside the running CPU job. - Long jobs yield. The guard gives its job HEAVY_TURN_DIR, and the job calls yieldTurn (tools/heavy/turn.ts) at each boundary. When another job waits, the guard lets go of its file, queues again as a new arrival, and grants the job the WORKERS of its next turn; when nobody waits it goes on at once. The balance bot plays chunks of about 25 runs (rounded to whole rounds of its workers, HEAVY_CHUNK_RUNS to set or 0 to turn off), a wave pass goes by snapshot slot, the nightly yields between its jobs, splash.ts between profiles, and perf-check between scenarios. Runs stay seeded by index, campaign chunks merge in seed order and wave runs are sorted, so the report matches an unchunked batch. A requeued batch is not counted or refused again and its second wait never gives up. - The queue goes round by worktree. A guard records when its worktree's turn ended (held-<key>.json); a worktree with a turn in the last 5 minutes goes behind other worktrees' waiting jobs, then arrival order. - Backfill. A Shared job that fits may start ahead of a Timing or CPU waiter that only waits for a running CPU job, for up to 2 minutes of that waiter's wait, never when the waiter waits for Shared jobs to drain. - Leases. A solo turn is a 10 minute lease (HEAVY_LEASE_MINUTES) in the lock file. Past it the job keeps running, but its guard says so on stderr and waiting jobs name it; a kept turn at a boundary renews it. decision.test.ts covers the queue order, round robin, handover, leases, backfill, and a model of the queue over time: no job waits more than 15 minutes behind an hour of chunked runs (an unchunked hour breaks it), and in a typical mix screenshot jobs wait 6 s on average and 70 s at most (160 s without backfill). A scratch run with HEAVY_STATE_DIR showed two Shared jobs from two directories start at a fake CPU job's next chunk boundary, a backfilled shot start beside it, and the lease warnings.
1 parent 168be99 commit f2ad92f

11 files changed

Lines changed: 1663 additions & 70 deletions

File tree

‎AGENTS.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,10 @@ Before you finish a change, run `typecheck`, `lint`, `test`, and `build`. All fo
4141

4242
- Run every heavy job through the guard: `bun run balance`, `bun run perf:check`, or `bun run heavy -- <command...>` for anything else heavy, such as an audit or a shots script. Never call `tools/balance/simulate.ts` or a heavy audit directly.
4343
- The guard aims at about 70% of the machine: a 1 minute load near 0.70 x cores (8.4 on 12, `HEAVY_TARGET_LOAD` to change it). A job starts only when the load, the jobs started in the last minute, and its own cost fit under that target.
44-
- **Timing** jobs (`perf:check` and the timing audits, `splash.ts`, and any command it does not know) run alone. **CPU** jobs (the balance bot, nightly, wave snapshots, determinism) run one at a time beside audits and size their `WORKERS` to the budget, up to 8. **Shared** jobs (the shots scripts, `taps.ts`, and the other screenshot and functional audits) run up to 4 at once, each costing about one core per browser context. Jobs queue fairly by arrival. `--timing`, `--cpu`, or `--shared` before the command overrides the class.
44+
- **Timing** jobs (`perf:check` and the timing audits, `splash.ts`, and any command it does not know) run alone. **CPU** jobs (the balance bot, nightly, wave snapshots, determinism) run one at a time beside audits and size their `WORKERS` to the budget, up to 8. **Shared** jobs (the shots scripts, `taps.ts`, and the other screenshot and functional audits) run up to 4 at once, each costing about one core per browser context. `--timing`, `--cpu`, or `--shared` before the command overrides the class.
45+
- **Long jobs yield.** The balance bot plays its runs in chunks of about 25 (a wave pass by snapshot slot), the nightly yields between its jobs, `splash.ts` between profiles, and `perf:check` between scenarios. At each boundary the guard hands over to any waiting job and queues again, so nobody waits long behind one batch. Results are the same as one unchunked run. A new long script should do the same with `yieldTurn` from `tools/heavy/turn.ts`.
46+
- **The queue goes round by worktree.** A worktree that just had a turn goes behind other worktrees' waiting jobs; otherwise jobs go by arrival. A Shared job may start ahead of a Timing or CPU job that only waits for a running CPU job (backfill).
47+
- **A solo turn is a 10 minute lease** (`HEAVY_LEASE_MINUTES`). A job past it keeps running, but the guard says so loudly and waiting jobs name it: split such a job so it yields.
4548
- `bun run test` sizes vitest's workers to what is left of the budget (at least 2) and counts itself, so test runs from several agents share the target. CI is unaffected.
4649
- The guard refuses a sixth balance batch per branch without `--allow-more` (`bun run heavy -- --reset-budget` starts the count again). How it works is in [`tools/heavy/README.md`](./tools/heavy/README.md).
4750
- Cap tuning at about three batches per change. Use 100 runs to sort ideas and 200 only to confirm the final numbers. Results within a few points of a target are fine: list misses as follow-ups instead of running more batches.

‎tools/audit/perf-check.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import { connect } from 'node:net'
33
import { loadavg } from 'node:os'
44
import { resolve } from 'node:path'
55

6+
import { yieldTurnAsync } from '../heavy/turn'
67
import { launch, wait } from './lib'
78
import { type FrameRow, calibrationMs, measureScenario } from './measure'
89

@@ -104,7 +105,11 @@ const measureAll = async (base: string) => {
104105
const browser = await launch()
105106
const calibration = await calibrationMs(browser)
106107
const rows: Array<FrameRow> = []
107-
for (const name of SCENARIO_NAMES) {
108+
for (const [index, name] of SCENARIO_NAMES.entries()) {
109+
// NOTE: under the heavy guard, other jobs may go between scenarios.
110+
if (index > 0) {
111+
await yieldTurnAsync(`the ${SCENARIO_NAMES[index - 1]} scenario`, 1)
112+
}
108113
for (const rate of RATES) {
109114
const trials: Array<FrameRow> = []
110115
for (let trial = 0; trial < TRIALS; trial++) {

‎tools/audit/splash.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,12 @@
99
// tasks, and with PROFILE=1 the functions that took the most CPU time before
1010
// the splash hid, and with NETWORK=1 every request the page and its workers
1111
// made, with when it started and finished. ONLY=<profile> runs one profile.
12+
// Under the heavy guard it yields the guard's turn between profiles, so
13+
// other jobs go in between instead of waiting for all three.
1214
// BASE=http://localhost:4173/ bun splash.ts
1315
import type { BrowserContext } from 'playwright-core'
1416

17+
import { yieldTurnAsync } from '../heavy/turn'
1518
import { GAME, launch, wait } from './lib'
1619

1720
const BASE = process.env.BASE ?? GAME
@@ -210,9 +213,13 @@ type Visit = Readonly<{
210213
}>
211214

212215
const browser = await launch()
213-
for (const profile of PROFILES.filter(
216+
const chosen = PROFILES.filter(
214217
({ name }) => ONLY === undefined || ONLY === name,
215-
)) {
218+
)
219+
for (const [index, profile] of chosen.entries()) {
220+
if (index > 0) {
221+
await yieldTurnAsync(`the ${chosen[index - 1]!.name} profile`, 1)
222+
}
216223
const context = await browser.newContext({
217224
viewport: { width: 844, height: 390 },
218225
deviceScaleFactor: 2,

‎tools/balance/nightly.ts‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
// targets and the stored baseline, and writes
66
// tools/balance/reports/<YYYY-MM-DD>.md. `--update-baseline` rewrites
77
// tools/balance/baseline.json, which is also written when it is missing.
8+
// Each job yields the guard's turn between its chunks, and the suite
9+
// yields between jobs, so other agents' jobs go in between.
810
// `--extend-baseline` adds only the metrics the baseline lacks. `--runs n`
911
// and `--wave-runs n` shrink the jobs for a quick try.
1012

@@ -18,6 +20,7 @@ import {
1820
FileSystem,
1921
Option,
2022
Path,
23+
Ref,
2124
Runtime,
2225
Schema,
2326
String,
@@ -28,6 +31,7 @@ import { ChildProcess, ChildProcessSpawner } from 'effect/process'
2831
import * as NodeRuntime from '@effect/platform-node/NodeRuntime'
2932
import * as NodeServices from '@effect/platform-node/NodeServices'
3033

34+
import { yieldTurn } from '../heavy/turn'
3135
import { balanceModel } from './model'
3236
import {
3337
Baseline,
@@ -86,6 +90,7 @@ const SummaryJson = Schema.fromJsonString(Summary)
8690
const runJob = (
8791
job: Job,
8892
outPath: string,
93+
maybeWorkers: Option.Option<number>,
8994
): Effect.Effect<
9095
Summary,
9196
JobFailed | BadSummary | PlatformError,
@@ -101,7 +106,10 @@ const runJob = (
101106
process.execPath,
102107
[SIMULATE, ...job.args, '--json', outPath],
103108
{
104-
env: job.env,
109+
env: Option.match(maybeWorkers, {
110+
onNone: () => job.env,
111+
onSome: workers => ({ ...job.env, WORKERS: `${workers}` }),
112+
}),
105113
extendEnv: true,
106114
stdin: 'inherit',
107115
stdout: 'inherit',
@@ -212,8 +220,23 @@ const program = Effect.gen(function* () {
212220
const date = localDate(startedAt)
213221
const scratch = yield* fs.makeTempDirectoryScoped({ prefix: 'nightly-' })
214222
const jobs = nightlyJobs(invocation.sizes)
215-
const summaries = yield* Effect.forEach(jobs, job =>
216-
runJob(job, path.join(scratch, `${job.id}.json`)),
223+
// NOTE: between jobs the suite yields the guard's turn, and runs the next
224+
// job with the WORKERS the guard granted.
225+
const grantedWorkers = yield* Ref.make(Option.none<number>())
226+
const summaries = yield* Effect.forEach(jobs, (job, index) =>
227+
Effect.gen(function* () {
228+
if (index > 0) {
229+
const maybeGranted = yield* yieldTurn(`the job before ${job.id}`)
230+
if (Option.isSome(maybeGranted)) {
231+
yield* Ref.set(grantedWorkers, maybeGranted)
232+
}
233+
}
234+
return yield* runJob(
235+
job,
236+
path.join(scratch, `${job.id}.json`),
237+
yield* Ref.get(grantedWorkers),
238+
)
239+
}),
217240
)
218241
const played = collect(jobs, summaries)
219242
const model = balanceModel()

‎tools/balance/simulate.ts‎

Lines changed: 100 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,11 @@
1111
// bun run balance -- --wave all --runs 100 (every campaign wave)
1212
// Flags: --runs n, --first-seed n, --wave n|all, --slots n (snapshot slots,
1313
// 10 by default), and --json path (also write the summary the nightly suite
14-
// reads). It drives the game's own init model and update with the bot in
14+
// reads). Under the guard it plays the runs in chunks of about 25 (by
15+
// slot groups for a wave pass) and yields the guard's turn between them, so
16+
// waiting jobs go in between; the report is the same as one unchunked
17+
// batch, since every run is seeded by its index. HEAVY_CHUNK_RUNS=n sets
18+
// the chunk, and 0 plays it whole. It drives the game's own init model and update with the bot in
1519
// tools/balance/bot.ts and prints survival per wave, run length, the money
1620
// curve, and which zombie kinds cost the bot the most hearts. Nothing here
1721
// ships with the game.
@@ -46,6 +50,8 @@ type NewKindLog = import('./bot').NewKindLog
4650
type BotOptions = import('./bot').BotOptions
4751
const { FINAL_WAVE } = await import('../../src/waves')
4852
const summary = await import('./summary')
53+
const turn = await import('../heavy/turn')
54+
const { nextChunk } = await import('../heavy/decision')
4955

5056
// TUNE=path/to/tune.ts loads a module whose default export edits the
5157
// tables in place before the runs, so a balance idea can be tried without
@@ -106,6 +112,7 @@ const firstSeed = Number(flagValue('--first-seed') ?? positional[1] ?? 1)
106112
const slots = Number(flagValue('--slots') ?? 10)
107113
const seedAt = (index: number) => seedFrom(firstSeed, index)
108114
const WORKERS = Math.max(1, Number(process.env.WORKERS ?? 1))
115+
const IS_CHUNKED = turn.isUnderGuard()
109116
const started = performance.now()
110117
const STYLE = SWAPS_BACK ? 'SwapsBack' : 'Default'
111118

@@ -125,9 +132,13 @@ const waveList = (flag: string): Array<number> => {
125132
// A worker plays one share of the work and writes its results as JSON, so
126133
// WORKERS=n gives the same report as one process, only sooner. A campaign
127134
// worker plays a slice of the seeds, from:to. A wave worker plays the
128-
// snapshot slots whose index modulo the worker count is its own, index:count.
135+
// snapshot slots whose index modulo the worker count is its own, index:count,
136+
// among the slots of its chunk, WORKER_SLOTS=from:to.
129137
if (process.env.WORKER_SLICE !== undefined) {
130138
const [from, to] = process.env.WORKER_SLICE.split(':').map(Number)
139+
const [slotFrom, slotTo] = (process.env.WORKER_SLOTS ?? `0:${slots}`)
140+
.split(':')
141+
.map(Number)
131142
const output =
132143
WAVE_FLAG === undefined
133144
? Array.from({ length: to! - from! }, (_, offset) =>
@@ -136,19 +147,28 @@ if (process.env.WORKER_SLICE !== undefined) {
136147
: playWavePass(
137148
{ waves: waveList(WAVE_FLAG), runs, slots, firstSeed },
138149
OPTIONS,
139-
slot => slot % to! === from!,
150+
slot => slot >= slotFrom! && slot < slotTo! && slot % to! === from!,
140151
)
141152
writeFileSync(process.env.WORKER_OUT!, JSON.stringify(output))
142153
process.exit(0)
143154
}
144155

145156
const SELF = fileURLToPath(import.meta.url)
146157

147-
const runWorker = (slice: string, index: number): Promise<unknown> =>
158+
const runWorker = (
159+
slice: string,
160+
index: number,
161+
slotRange?: string,
162+
): Promise<unknown> =>
148163
new Promise((resolve, reject) => {
149164
const out = `/tmp/simulate-${process.pid}-${index}.json`
150165
const child = spawn(process.execPath, [SELF, ...argv], {
151-
env: { ...process.env, WORKER_SLICE: slice, WORKER_OUT: out },
166+
env: {
167+
...process.env,
168+
WORKER_SLICE: slice,
169+
WORKER_OUT: out,
170+
...(slotRange === undefined ? {} : { WORKER_SLOTS: slotRange }),
171+
},
152172
stdio: 'inherit',
153173
})
154174
child.on('error', reject)
@@ -159,35 +179,97 @@ const runWorker = (slice: string, index: number): Promise<unknown> =>
159179
)
160180
})
161181

162-
const playAll = async (): Promise<Array<RunResult>> => {
163-
if (WORKERS === 1) {
164-
return Array.from({ length: runs }, (_, index) =>
165-
runOnce(seedAt(index), OPTIONS),
182+
// NOTE: the guard may give a later chunk more or fewer workers.
183+
let workers = WORKERS
184+
185+
const playRange = async (
186+
first: number,
187+
end: number,
188+
): Promise<Array<RunResult>> => {
189+
if (workers === 1) {
190+
return Array.from({ length: end - first }, (_, offset) =>
191+
runOnce(seedAt(first + offset), OPTIONS),
166192
)
167193
}
168-
const per = Math.ceil(runs / WORKERS)
169-
const slices = Array.from({ length: WORKERS }, (_, index) => [
170-
index * per,
171-
Math.min(runs, (index + 1) * per),
194+
const per = Math.ceil((end - first) / workers)
195+
const slices = Array.from({ length: workers }, (_, index) => [
196+
first + index * per,
197+
Math.min(end, first + (index + 1) * per),
172198
]).filter(([from, to]) => to! > from!)
173199
const outputs = await Promise.all(
174200
slices.map(([from, to], index) => runWorker(`${from}:${to}`, index)),
175201
)
176202
return outputs.flat() as Array<RunResult>
177203
}
178204

179-
const playWaves = async (waves: Array<number>): Promise<Array<WaveRun>> => {
205+
// NOTE: plays the runs in chunks, in seed order, yielding the guard's turn
206+
// between chunks. Run i is always seed i, so the results do not depend on
207+
// where the chunks fall.
208+
const playAll = async (): Promise<Array<RunResult>> => {
209+
const results: Array<RunResult> = []
210+
for (let done = 0; done < runs;) {
211+
if (done > 0) {
212+
workers = await turn.yieldTurnAsync(`${done} of ${runs} runs`, workers)
213+
}
214+
const [from, to] = nextChunk(
215+
done,
216+
runs,
217+
IS_CHUNKED ? turn.chunkRuns(workers) : 0,
218+
)
219+
results.push(...(await playRange(from, to)))
220+
done = to
221+
}
222+
return results
223+
}
224+
225+
const playSlots = async (
226+
waves: Array<number>,
227+
slotFrom: number,
228+
slotTo: number,
229+
): Promise<Array<WaveRun>> => {
180230
const pass = { waves, runs, slots, firstSeed }
181-
const workers = Math.min(WORKERS, slots)
182-
if (workers === 1) return playWavePass(pass, OPTIONS)
231+
const count = Math.min(workers, slotTo - slotFrom)
232+
if (count === 1) {
233+
return playWavePass(
234+
pass,
235+
OPTIONS,
236+
slot => slot >= slotFrom && slot < slotTo,
237+
)
238+
}
183239
const outputs = await Promise.all(
184-
Array.from({ length: workers }, (_, index) =>
185-
runWorker(`${index}:${workers}`, index),
240+
Array.from({ length: count }, (_, index) =>
241+
runWorker(`${index}:${count}`, index, `${slotFrom}:${slotTo}`),
186242
),
187243
)
188244
return outputs.flat() as Array<WaveRun>
189245
}
190246

247+
// NOTE: a wave pass chunks by snapshot slot, one slot a worker, so each
248+
// slot's snapshots are found once. The runs are sorted by wave and run, so
249+
// the order never depends on the chunks or the workers.
250+
const playWaves = async (waves: Array<number>): Promise<Array<WaveRun>> => {
251+
const waveRuns: Array<WaveRun> = []
252+
for (let done = 0; done < slots;) {
253+
if (done > 0) {
254+
workers = await turn.yieldTurnAsync(
255+
`${done} of ${slots} snapshot slots`,
256+
workers,
257+
)
258+
}
259+
const [from, to] = nextChunk(
260+
done,
261+
slots,
262+
IS_CHUNKED && turn.chunkRuns(workers) > 0 ? workers : 0,
263+
)
264+
waveRuns.push(...(await playSlots(waves, from, to)))
265+
done = to
266+
}
267+
return waveRuns.sort(
268+
(first, second) =>
269+
first.wave - second.wave || first.waveSeed - second.waveSeed,
270+
)
271+
}
272+
191273
const writeJson = (value: unknown) => {
192274
if (JSON_OUT !== undefined) {
193275
writeFileSync(JSON_OUT, `${JSON.stringify(value, null, 2)}\n`)

0 commit comments

Comments
 (0)