Skip to content

feat: add scheduler backend (incomplete) - #1282

Draft
kvlonge wants to merge 49 commits into
mainfrom
feat/orchestration-backend
Draft

kvlonge wants to merge 49 commits into
mainfrom
feat/orchestration-backend

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Why

Add a native control plane; Python stays in locations/workers. Backend is incomplete; UI is separate.

Changes

  • SQLite/PostgreSQL storage, immutable snapshots, fenced scheduling, atomic claims and audited scoped administration.
  • API/scheduler/watchdog roles, outbound checks, scoped reload/health and last-good publication.
  • Local/Docker workers pin frozen inputs/images, cancel statements and retain claims through physical-stop recovery. Linux uses claim locks/pidfds; Windows/macOS recovery remains incomplete.
  • Groups retain capacity/failure policies; pinned reruns preserve settings, dependencies and typed backfill windows. Calendar callbacks/refresh remain location-owned.
  • Optional server webhook uses durable acknowledgements, bounded credential-safe retries and at-least-once replay after endpoint changes.
  • Built-in platform sensors atomically publish deduplicated runs, outcomes and cursors, with fenced recovery and operator overrides. Enum-only public facade retains remaining_lines=0; native metadata/crossing policies remain enforced.
  • Shared pinned Testcontainers PostgreSQL harness serves local/CI gates; missing infrastructure fails, SQLite stays covered.
  • Remaining: custom/other sensor conditions, human authorization/SSO, warehouse sync, dev/SSE/clients and operational acceptance.

Verification

  • make check-ci passes, 0 Fensu faults. Native orchestration: 46 passed, 0 ignored, both databases and container cleanup verified.
  • Compiler/publication: 111 passed initially; both mixed-trigger assertion failures corrected and all 4 publication cases pass. Includes real local/Docker workers, calendars, sensors and recovery.
  • Independent review/follow-up complete; DuckShop sensor cases included.
  • Broad make test: 10,753 passed before 4 existing CLI/signal failures and a pytest reporter crash. Same 4 failures reproduced on rebuilt base; no gate weakened.
  • Exact-head CI monitored. Draft stays unmerged; private setup/evidence excluded.

Preserve in-progress storage, scheduling, claims, credentials and native API/CLI work. The orchestration backend remains incomplete and is not ready for delivery.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Add claim-scoped execution inputs and native snapshot drift validation.

Incomplete orchestration backend checkpoint; worker launch and the remaining control-plane integration are not implemented yet. Targeted SQLite/PostgreSQL tests pass; full static readiness is pending.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Verify pinned inputs before start and expose retryable run reports and cancellation heartbeats over the authenticated API. Disable redirects for credentialed requests and cover the lifecycle against SQLite and PostgreSQL. Keep the orchestration PR explicitly incomplete pending actual launchers and the remaining backend scope.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Separate lease expiry, replica connections, terminal run fixtures and fault cases so each integration binary imports only the support it uses. Preserve SQLite and PostgreSQL transaction coverage.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Observe heartbeat loss in fenced, bounded transactions while retaining physical write claims. Recover through existing worker heartbeats and continue full batches without an idle delay.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Own worker heartbeats on a dedicated native runtime and connection pool. Propagate cancellation and revoked credentials without cancelling work on transient API failures; stop and join with the execution owner.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Move bounded client construction, credential redaction, URL validation and JSON response handling to the API owner so worker and location flows can share them. Keep independent heartbeat pools and worker-facing error categories.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Add privileged native administration with JSON results, bounded location credentials and atomic security audit. Detect lost result delivery without claiming committed mutations rolled back.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Keep the public Python API limited to authoring records and native delegation. Resolve selectors and validate schedule data in Rust before publication. Preserve stable shorthand trigger names across cron edits.

Add the explicitly approved narrow Python facade boundary entries without increasing movable Python or native crossing budgets.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Collect exported jobs and nested factory containers natively, deduplicating aliases and terminating cyclic containers. Explicit empty declarations skip user module execution. Remove failed imports from the module cache so publication retries cannot reuse partial definitions.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Verify retained compiler baselines before worker startup and join native heartbeat work on interpreter-side close. Preserve claim identity and report retry keys without treating resource teardown as physical warehouse completion.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Restrict forward and reverse cron expansion and pure tick planning to the scheduler. Preserve scheduling bodies and all existing fault, DST and simulation coverage.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Expose owner-only cancellation with stable retry identity. Commit lifecycle facts, projections and security audit together while retaining active physical claims until termination.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Resolve manual requests against the current published job inside one write transaction. Preserve the accepted snapshot and execution inputs across retries after redeployment, and commit queue publication with the trusted owner audit.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Compile in the project interpreter and retain native snapshots through publication. Keep JSON results separate from progress and distinguish uncertain acknowledgements from failed result delivery.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Preserve the current backend implementation as a remote-review checkpoint. The pull request remains draft and this is not release-ready.

Group admission and the four complexity fixes passed targeted real-stack verification. Pinned reruns pass the SQLite/PostgreSQL transaction regression; installed worker rerun coverage and final aggregate gates remain in progress. Six existing Fensu interface findings remain unresolved.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Keep successful and cancelled nodes out of failure-only reruns, retain waits through omitted completed jobs, and prove the complete local and Docker worker paths.

The backend remains WIP. SQLite/PostgreSQL API, run and Recall regressions pass; the publication suite passes 42 cases. The exact static target still reports six known unused-public-entry findings.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Exercise the inherited seed and scalar function together, attach the required not-null audit and assert concrete SQL unit-test results.

The failing DuckShop compile checks now pass without relaxing any Rules or changing the core benchmark checksum.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Retain inherited claim locks through scheduler death and verify physical absence before releasing claims. Acquire ownership before fenced launch publication so a delayed stale scheduler cannot spawn after recovery.

WIP: local restart recovery is Linux-only. The backend remains incomplete and the static gate is red on six existing test/API-boundary findings; no checks are bypassed or relaxed.
Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Remove test-only production convenience exports and retain the real SQLite/PostgreSQL contracts in owner-local test harnesses. Exercise CLI worker lease setup through the public scheduler loop. Shared fixtures are declared once under their owners.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Freeze inclusive typed cursor windows with run requests, derive execution overrides without mutating pinned job settings, and preserve the window through group children and reruns. Verify real local and Docker warehouse writes and the SQLite/PostgreSQL publication contracts.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66
Queue sensor runs, commit cursors and publish outcomes atomically under scheduler fencing. Preserve deduplication, recovery and operator overrides across deployments.

Place the public PlatformEventKind enum in types.py with an enum-only rule-3 facade entry, retaining remaining_lines=0 and native implementation ownership.

Cover mixed schedule/sensor publication and real worker execution without relaxing existing regression assertions.

Amp-Thread-ID: https://ampcode.com/threads/T-01a18715-4e00-708e-a87d-6bd109748d66

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant