Skip to content

test(perf): regression check and competitor benchmark - #1517

Open
rjmreis wants to merge 2 commits into
callstack:mainfrom
rjmreis:perf/benchmarks
Open

rjmreis wants to merge 2 commits into
callstack:mainfrom
rjmreis:perf/benchmarks

Conversation

@rjmreis

@rjmreis rjmreis commented Oct 30, 2026

Copy link
Copy Markdown

Why

To iterate on Re.Pack safely (including with AI-assisted changes), we need to know when a change makes bundle size, build time or memory worse. Then we can use the same tooling to find performance wins.

What's in it

Two commits:

1. Regression check: tests/perf

Bundles apps/tester-app in production mode (rspack + webpack, iOS + Android) for the branch and its base, and compares them.

  • Size (exact): main bundle raw/gzip, all chunks, source maps, assets, module/chunk counts, emitted files. Fails on a >1% increase.
  • Build time / peak memory (sampled): median of interleaved runs, each in a fresh process. Fails only on a large change where head and base samples don't overlap.
  • Detects output that differs between runs of the same checkout (currently webpack). Those metrics are compared like timings, and a PR that introduces nondeterminism gets flagged.
pnpm bench --base-ref origin/main                       # compare with main
pnpm bench --base-ref origin/main --fixtures rspack/ios --runs 3   # quicker

CI: .github/workflows/perf.yml is on-demand only and independent of the regular CI checks. It runs when a PR gets the perf label, or from the Actions tab (optionally with a PR number). It writes a report to the job summary and is not a required check. The existing workflows are untouched.

2. Competitor benchmark: tests/perf/src/competitors.mjs

Two neutral apps that both bundlers build unchanged:

App Competitor Re.Pack setup
apps/bench-expo (Expo Router) Metro via Expo CLI (+ optional experimental tree shaking) repack-expo
apps/bench-bare (plain RN) Metro via React Native CLI (@react-native/metro-config defaults) repack-init template config

Each app is measured at several sizes (screens generated by generate-screens.mjs). The benchmark covers cold and warm release builds (wall time, CPU, peak memory), bundle and Hermes bytecode size, and dev server startup and edit-to-bundle latency.

node tests/perf/src/competitors.mjs --scales 0,1000,3000 --runs 3

Early findings (leads, to confirm)

Measured before rebasing onto RN 0.88 / Expo 58, on an M4 Pro, iOS only:

  • The Expo integration is the slow path. On the bare app Re.Pack beats Metro: 1.2–2.0× faster cold builds and ~3× faster dev server start. On the Expo app it's slower, keeping ~1.6 cores busy versus ~3.7 on the bare app.
  • Parallel loader memory grows with app size. The bare app peaks at 7.7 GB at 3,000 screens (Metro: 4.6 GB).
  • Edit → updated dev bundle is 1.25–2.5× faster than Metro on both apps.
  • Bundle size: 12–16% smaller on the Expo app, about the same as Metro on the bare app.
  • webpack builds are nondeterministic with thread-loader + babel-swc-loader: the same RN files get different @babel/runtime helpers depending on the worker.

More detail and pitfalls are in agent_context/perf-regressions-oct2026/design.md.

Notes for review

  • pnpm-lock.yaml also bumps @babel/types 7.29.7 → 7.29.8 in 5 existing entries. pnpm does this on any workspace change (it happens even when adding only tests/perf, which has no dependencies), so it isn't something these apps need.
  • No changeset: no published package changes.
  • Verified locally with lint and full typecheck (19 tasks) on the current main, plus smoke runs of both benchmarks. That was on Node 24.20; main now asks for ≥ 24.22.1.
  • perf.yml hasn't run on GitHub yet; adding the perf label to this PR will be its first real run.
  • Not included yet: One (Vite native mode). A feasibility spike is next.

🤖 Generated with Claude Code

rjmreis and others added 2 commits October 30, 2026 12:41
Adds `tests/perf`, which bundles apps/tester-app in production mode
(rspack and webpack, ios and android) for the current checkout and a base
checkout, and compares the results:

- output size (main bundle raw/gzip, all chunks, source maps, assets),
  module and chunk counts, and the list of emitted files;
- build time (median of interleaved runs, each in a fresh process);
- peak memory (process maxRSS plus the sampled process tree).

Size metrics fail on a >1% increase. Time and memory use loose thresholds
and only fail when head and base samples don't overlap. Metrics that vary
between runs of one checkout (currently webpack output) are detected and
compared like timings.

Run locally with `pnpm bench --base-ref origin/main`. In CI,
`.github/workflows/perf.yml` runs only on demand (the `perf` PR label or
a manual dispatch) and is independent of the regular CI checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds two neutral apps that both Re.Pack and Metro build without changes:

- apps/bench-expo: Expo Router app (Metro via Expo CLI, optionally with
  Expo's experimental tree shaking, vs Re.Pack with repack-expo);
- apps/bench-bare: plain React Native app (Metro via the React Native CLI
  with @react-native/metro-config defaults, vs Re.Pack with the
  repack-init template config).

`tests/perf/src/competitors.mjs` bundles each app at several sizes
(screens added by `generate-screens.mjs`) and measures cold and warm
release builds (wall time, CPU time, peak memory), bundle and Hermes
bytecode size, and dev server startup and edit-to-bundle latency. It is
not a regression gate: it is for comparing bundlers and finding
performance wins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 30, 2026

Copy link
Copy Markdown

@rjmreis is attempting to deploy a commit to the Callstack Team on Vercel.

A member of the Team first needs to authorize it.

@changeset-bot

changeset-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: bb80f61

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant