Skip to content

feat: exact KNN mesh magnification areas + fix partial-block neighbour search (arc phase 4b) #665

Closed

Description

@Jammy2211

Overview

Source & Cluster arc phase 4b. Phase 4a (#664) made every rectangular and Delaunay-family mesh's areas_for_magnification exact. KNearestNeighbor and KNNBarycentric still borrow the Delaunay dual areas, which aren't the integrals of their interpolants. Their identity-lens magnification is off by 1.6–4.4% (unit test) and up to 6% lensed. While measuring this I found #609, a neighbour-search bug in the last partial block, which corrupts KNN interpolation whenever the node count isn't a multiple of 128. This task fixes both.

Plan

  • Fix fix: Preserve KNN neighbors in a partial final point block #609: pad the KNN node array to whole blocks so the last block holds the right points with the right indices. Add a brute-force oracle regression test at N=130.
  • KNN and KNNBarycentric areas_for_magnification = the integral of each node's interpolation weight over a uniform 256² grid masked to the convex hull (same domain as the Delaunay dual areas; host-side, like them).
  • Tighten the KNN tolerances in the PyAutoArray unit test and the autolens_workspace_test imaging/mesh_magnification.py script to the new measurements.
  • DelaunayNN is left as is (within 0.5%) and recorded as the remaining approximation.
  • Tier: judge — merge mode: human /prm.
Detailed implementation plan

Affected Repositories

  • PyAutoArray (primary)
  • autolens_workspace_test

Branch Survey

Repository Current Branch Dirty?
./PyAutoArray main clean
./autolens_workspace_test main clean

Suggested branch: feature/mesh-magnification-knn-areas

Implementation Steps

  1. autoarray/inversion/mesh/interpolator/knn.py::get_interpolation_weights: jnp.pad the points to n_blocks * B rows and slice the padded array. The clamped dynamic_slice currently loads [N-B, N) while labelling the rows start + arange(B), so tail nodes are never seen and earlier nodes come back with wrong indices.
  2. Add the KNN quadrature areas: an areas_for_magnification override for the KNN interpolators' mesh geometry. Evaluate the interpolator's own _mappings_sizes_weights on cell centres of a 256² grid over the vertex bounding box, keep only centres inside the convex hull (scipy.spatial.Delaunay.find_simplex), and bincount the weights × cell area per node.
  3. Tests: a fix: Preserve KNN neighbors in a partial final point block #609 oracle test (N=130, point_block=128, eager and jit, both KNN variants); tighter tolerances in test_areas_for_magnification.py; area sum = hull area.
  4. autolens_workspace_test scripts/imaging/mesh_magnification.py: tighten the KNN tolerances to the new measurements.

Prototype (with #609 patched, 24×24 identity grid, 144 nodes)

mesh Delaunay dual areas exact quadrature (n=200/400/800)
KNearestNeighbor −3.99e-2 −7e-6 / +1.6e-5 / +1.5e-5
KNNBarycentric −1.38e-2 +1.4e-3 / +1.8e-3 / +1.6e-3

Key Files

  • autoarray/inversion/mesh/interpolator/knn.py
  • autoarray/inversion/mesh/mesh_geometry/delaunay.py
  • test_autoarray/inversion/pixelization/mesh_geometry/test_areas_for_magnification.py
  • autolens_workspace_test/scripts/imaging/mesh_magnification.py

Closes #609 on merge as well.

Original Prompt

Click to expand starting prompt

Mesh magnification: exact-quadrature areas for KNN meshes (phase 4b)

Type: feature
Target: autoarray
Repos:

  • PyAutoArray
  • autolens_workspace_test
    Themes:
  • pixelization
  • cluster
    Difficulty: medium
    Autonomy: supervised
    Priority: high
    Status: formalised
    Consequence: judge
    Review-minutes: 25
    Epic: cluster-strong-lensing
    Phase: 4b
    Parent: draft/test/workspaces/mesh_magnification_correctness.md
    Arc: draft/feature/autolens/source_cluster_arc.md
    Filed: 2026-10-28

Human rulings (2026-10-28, verbatim answers to the 4a flagged decisions)

"1 I authroi,e 2 perform proper fix 3 add to 4b fix 4 file this yes"

  • 2 = adaptive rectangular areas: perform the proper (exact hat-quadrature) fix.
  • 3 = KNearestNeighbor / KNNBarycentric: add to this 4b fix.
  • 4 = zeroed-ring guard-row mismatch filed separately as PyAutoArray#660 (not this phase).

NOTE: this renumbers the old parent's "4b" (library flux-ratio magnification promotion) — that and the
source_science recovery sweep become 4c/4d when issued.

Scope change (2026-10-29)

Human 2026-10-29 chose "Fix adaptive rect first": the adaptive rectangular exact hat quadrature
(work item 1 below) was folded INTO 4a (PyAutoArray#659, adaptive_rectangular_hat_areas_from)
before its PRs opened. Measured after: lensed BilinearAdaptImage +14.43% -> -0.57%, RTUAdaptImage
+8.56% -> +0.08%; identity BilinearAdaptDensity +11.23% -> -0.34%; tolerances tightened to 1%
(RTUAdaptDensity lensed stays -1.82%, unchanged by the areas). Open item from it: on a strongly
distorted toy (radial a=0.3, BilinearAdaptImage) the bbox-domain hat areas sit +6.5% / +13% vs the
traced-region quadrature (domain mismatch, separable interpolant extends into untraced bbox corners);
the lensed simulator check is unaffected. 4b is now items 2-4 (KNN / KNNBarycentric) only.

Measured bias at 4a (PyAutoArray#659; tolerances in 4a document these)

mesh unit μ_id−1 integ. lensed integ. identity
RectangularBilinearAdaptDensity +3.1e-2 −0.21% +11.23%
RectangularBilinearAdaptImage +2.4e-1 +14.43% +1.31%
RectangularRTUAdaptImage +8.0e-2 +8.56% +1.05%
RectangularRTUAdaptDensity +2.5e-3 −1.82% −0.24%
KNearestNeighbor +1.6e-2 +6.10% −5.11%
KNNBarycentric −4.4e-2 −1.71% −3.57%

Work

  1. DONE IN 4a (see scope change). Adaptive rectangular (mesh_geometry/rectangular.py): replace node-midpoint CDF-transformed cell areas
    with the exact per-axis quadrature of each node's bilinear hat through the CDF,
    q(i) = ∫ hat_i(U(x)) dx, outer-product per axis. A 4a scratch prototype
    (/tmp/claude-1000/-home-jammy-Code-PyAutoLabs/2aa32082-3125-4533-a9ef-6b0e392688b6/scratchpad/measure.py,
    copy it into the task before /tmp is swept) took unit errors to ~1e-3 (Bilinear density
    2.8e-2→2e-6, RTU image −6.5e-2→2e-3); one strongly distorted toy stayed +6.5% (suspected domain
    mismatch: data region smaller than its bbox — confirm). Must stay JAX-jit/grad safe; check plotting
    (edges_transformed consumers) keeps drawing cell edges, i.e. split "areas" from "edges" if needed;
    measure cost.
  2. KNN / KNNBarycentric: areas = exact quadrature of their interpolants (∫ of the per-node Wendland /
    barycentric-KNN weights), e.g. via the mapping-matrix column sums over a fine quadrature grid, or a
    loud documented approximation if exact is infeasible — measure.
  3. Tighten the 4a tolerances (unit test test_areas_for_magnification.py and autolens_workspace_test
    scripts/imaging/mesh_magnification.py) to the new measurements (~1.5× measured, floor 1%).
  4. RectangularUniform could optionally adopt the exact per-axis 9/8, 7/8 edge weights (review note);
    negligible for interior sources.

Activity

  1. Jammy2211 commented on Oct 29, 2026

    @Jammy2211
    CollaboratorAuthor

    Scope change 2026-10-29 (human ruling): #609 is split out of this task. Its fix breaks the KNN + AdaptSplit gradient check (jax_grad/knn.py); details and the patch are in #609. The exact KNN areas don't depend on it. They integrate whatever neighbours the interpolator selects, and measure KNN −0.23% / −0.12% and KNNBarycentric −0.40% / −0.14% (lensed / identity) against simulator truth either way. This task now ships the areas only.

  2. Jammy2211 commented on Oct 29, 2026

    @Jammy2211
    CollaboratorAuthor

    Shipped and merged 2026-10-29:

    • PyAutoArray#666 (e71feafd): KNearestNeighbor and KNNBarycentric areas_for_magnification are now the integral of each vertex's basis function over the hull (MeshGeometryKNearestNeighbor).
    • autolens_workspace_test#349 (5a41b981): KNN tolerances in imaging/mesh_magnification.py tightened to 1%.

    Against simulator truth: KNN −0.23% / −0.12%, KNNBarycentric −0.40% / −0.14% (lensed / identity). #609 stays open separately. PyAutoArray change pending release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions