feat(aarch64): add GDB debugging for Linux KVM by cshung · Pull Request #2016 · hyperlight-dev/hyperlight

cshung · 2026-11-03T00:34:05Z

Summary

ARM64 Linux/KVM guests support GDB register and memory access, two hardware breakpoints, BRK software breakpoints, and hardware single-step.

KVM reports debug stops through KVM_EXIT_DEBUG. Guest address translation uses the existing page-table reader. Debugger code writes synchronize the instruction cache. Debug state survives vCPU reset, and register access respects the active stack pointer.

Depends on #1907. This branch includes its shared ARM64 debugger support.

KVM-only comparison.

Validation

Raspberry Pi 4: 16 targeted native unit tests and all six GDB integration tests pass.
Real GDB sessions verify entry stepping, register writes, hardware and repeated software breakpoints, code writes, self-loop and RET stepping, detach, and snapshot debugging.
Windows ARM64: just test and just test-rust-gdb-debugging pass.
just fmt-apply, just guests, just clippy, and native ARM64 KVM Clippy with GDB enabled pass.

Pi validation uses a separate, uncommitted ARMv8.0 allocator fix and guests built with RUSTFLAGS="-C target-feature=-lse". Pi compatibility changes are outside this commit.

Adds GDB stub support for guests running on the ARM64 WHP backend. * Split the x86_64 debug code out of `hyperlight_vm/x86_64.rs` into a shared `hyperlight_vm/debug.rs`. * Rename `gdb/x86_64_target.rs` to `gdb/target.rs` and add per-architecture register handling in `gdb/arch/`. * Give `DebuggableVm` default methods so backends without hardware breakpoints or single step report `Unsupported`. * Implement `DebuggableVm` for the aarch64 WHP VM, using the batched register helpers. * Flush the host instruction cache after guest memory writes so software breakpoints take effect. * Document ARM64 debugging in `docs/how-to-debug-a-hyperlight-guest.md` and `CHANGELOG.md`. On Windows ARM64 (WHP): * `just build`, `just clippy`, `just guests`, `just test` and `just fmt-check` pass. --------- Signed-off-by: cshung <3410332+cshung@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Changes * Run the full disable path when the debugger detaches after a crash, so software breakpoints and pending steps are cleaned up. * Cancel pending software steps for any cancellation source, not only debugger interrupts. * Reject `map_region` requests that overlap the hidden cache sync page at GPA 0x1000, and add a test. * Restore the `≡` and `×` characters in `hyperlight_vm/x86_64.rs` comments. ## Verification On Windows ARM64 (WHP): * `just clippy`, `just fmt-check` and the `hypervisor::` and `map_region` lib tests with the `gdb` feature pass. --------- Signed-off-by: cshung <3410332+cshung@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Guest synchronous exception vectors forward debug stops through HVC. Hardware single-step supports branches, returns, and self-loops. Two hardware breakpoint slots and BRK software breakpoints are available. Interrupted register context and breakpoint configuration survive cache synchronization and vCPU reset. Asynchronous vectors retain their normal exception paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

Copilot

🟡 Changes recommended

Unresolved critical flaws allow guest modification of executable trampoline code and debugger memory corruption, while breakpoint cleanup and snapshot-ordering issues can leave sandboxes unrecoverable.

5 open issues

What changed in this PR

Adds ARM64 guest debugging on Linux/KVM, alongside the shared debugger infrastructure and Windows/WHP support from #1907.

Changes:

Supports ARM64 register and memory access, software and hardware breakpoints, and hardware single-step.
Adds instruction-cache synchronization, debug-state preservation across reset, and snapshot safeguards.
Adds debugger tests and documents platform capabilities.

File	Description
`src/hyperlight_host/src/sandbox/uninitialized_evolve.rs`	Initializes ARM64 debug vectors during VM setup.
`src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs`	Exposes backing ranges for cache synchronization.
`src/hyperlight_host/src/sandbox/initialized.rs`	Guards snapshots and restores against active software breakpoints.
`src/hyperlight_host/src/mem/mgr.rs`	Exposes checked host memory ranges to the debugger.
`src/hyperlight_host/src/mem/exe.rs`	Carries debug-vector addresses in load metadata.
`src/hyperlight_host/src/mem/elf.rs`	Finds and relocates the debug-vector symbol.
`src/hyperlight_host/src/hypervisor/virtual_machine/whp/x86_64.rs`	Adapts address translation to the shared interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/whp/aarch64.rs`	Implements WHP debugging, cache synchronization, and tests.
`src/hyperlight_host/src/hypervisor/virtual_machine/mshv/x86_64.rs`	Adapts address translation to the shared interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs`	Adds ARM64 debug exits and initialization errors.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs`	Adapts address translation to the shared interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/aarch64.rs`	Implements KVM debugging and active-stack-pointer handling.
`src/hyperlight_host/src/hypervisor/regs/aarch64/whp_reg.rs`	Defines WHP debug register identifiers.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Uses shared debugger handling and breakpoint storage.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs`	Integrates ARM64 stops, stepping, and mapping safeguards.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/debug.rs`	Provides shared requests and software-breakpoint lifecycle handling.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Connects ARM64 VM setup and execution to GDB.
`src/hyperlight_host/src/hypervisor/gdb/target.rs`	Adds ARM64 register conversion to the shared target.
`src/hyperlight_host/src/hypervisor/gdb/mod.rs`	Adds page-table translation and cache-flush support.
`src/hyperlight_host/src/hypervisor/gdb/event_loop.rs`	Reports ARM64 initial stops to GDB.
`src/hyperlight_host/src/hypervisor/gdb/arch/x86_64.rs`	Houses x86 breakpoint constants and stop classification.
`src/hyperlight_host/src/hypervisor/gdb/arch/aarch64.rs`	Defines ARM64 software breakpoints and alignment checks.
`src/hyperlight_host/src/hypervisor/gdb/arch.rs`	Selects architecture-specific debugger helpers.
`src/hyperlight_host/examples/guest-debugging/main.rs`	Adds ARM64 breakpoint, stepping, and code-write integration tests.
`src/hyperlight_host/build.rs`	Enables GDB for supported ARM64 platforms.
`src/hyperlight_guest_bin/src/arch/aarch64/exception/entry.rs`	Exports debug vectors and forwards synchronous exceptions.
`src/hyperlight_common/src/arch/aarch64/exn.rs`	Defines the shared debug-exception HVC immediate.
`docs/how-to-debug-a-hyperlight-guest.md`	Documents ARM64 capabilities and guest-symbol requirements.
`CHANGELOG.md`	Records Linux/KVM and Windows/WHP ARM64 debugging support.

_{Review effort: Balanced}

+                page_size::get() - (gpa as usize & (page_size::get() - 1)),
+            );
+            mem_access.read(&mut data[..read_len], gpa)?;
+            data = &mut data[read_len..];
+            gva += read_len as u64;
+        }
+
+        Ok(())
+    }
+
+    fn write_addrs(
+        &mut self,
+        mut gva: u64,
+        mut data: &[u8],
+        mem_access: &DebugMemoryView<'_>,
+    ) -> std::result::Result<(), ProcessDebugRequestError> {
+        tracing::debug!("Write addr: {:X} len: {:X}", gva, data.len());
+
+        while !data.is_empty() {
+            let gpa = self.vm.translate_gva(gva, mem_access)?;
+            let write_len = std::cmp::min(
+                data.len(),
+                page_size::get() - (gpa as usize & (page_size::get() - 1)),


+            .copy_from_slice(&DEBUG_CACHE_SYNC_CODE, 0)
+            .map_err(|e| CreateVmError::InitializeDebug(e.to_string()))?;
+        let (host_memory, guest_memory) = memory.build();
+        let region = guest_memory.mapping_at(DEBUG_CACHE_SYNC_GPA, MemoryRegionType::Scratch);


+        #[cfg(all(gdb, target_arch = "aarch64"))]
+        if self.vm.has_sw_breakpoints() {


+                                Err(error) => {
+                                    tracing::error!("Failed to clean up after detach: {error}");
+                                    DebugResponse::ErrorOccurred
+                                }


+        #[cfg(all(gdb, target_arch = "aarch64"))]
+        if self.vm.has_sw_breakpoints() {
+            return Err(crate::new_error!(
+                "Cannot restore an ARM64 sandbox while software breakpoints are installed"
+            ));


ARM64 KVM debugging uses KVM_SET_GUEST_DEBUG for hardware single-step and two hardware breakpoints. Software breakpoints use BRK. Guest virtual addresses use the existing page-table reader. Debugger code writes synchronize the host instruction cache. Register access respects the active stack pointer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

ARM64 KVM debug jobs run the GDB suite before the general Rust tests. Install GDB when the runner lacks it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

hyperlight-gh-bot · 2026-11-03T03:09:53Z

Benchmark Results

Measured commit: 55da94f34856
Baseline commit: fa13116cd998

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`373.20 µs` (➖ 1.00x faster)
`byte_chunks`		`748.93 ns` (➖ 1.01x faster)
`vec_bytes`		`600.21 ns` (➖ 1.02x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`143.88 ns` (➖ 1.02x slower)
`262144`	`534.15 ns` (➖ 1.10x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.17 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.38 ns` (➖ 1.09x slower)	`7.71 ns` (➖ 1.00x slower)	`7.76 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`105.59 µs` (➖ 1.06x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`1.95 µs` (➖ 1.07x faster)	`2.06 µs` (➖ 1.05x slower)
`262144`	`6.97 µs` (➖ 1.00x faster)	`7.05 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.08 µs` (➖ 1.07x slower)	`1.01 µs` (➖ 1.02x faster)
`65536`	`6.22 µs` (➖ 1.11x faster)	`6.28 µs` (➖ 1.04x faster)
`262144`	`28.05 µs` (➖ 1.04x slower)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`675.18 µs` (➖ 1.02x slower)
`byte_chunks`		`667.67 ns` (➖ 1.01x slower)
`vec_bytes`		`516.34 ns` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`155.05 ns` (➖ 1.09x faster)
`262144`	`493.37 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.52 ms` (➖ 1.03x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`6.78 ns` (➖ 1.00x slower)	`6.78 ns` (➖ 1.00x slower)	`6.77 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`49.92 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.13 µs` (➖ 1.01x slower)	`2.09 µs` (➖ 1.02x faster)
`262144`	`7.48 µs` (➖ 1.01x faster)	`7.47 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`702.17 ns` (➖ 1.01x slower)	`726.98 ns` (➖ 1.01x slower)
`65536`	`7.16 µs` (➖ 1.02x slower)	`7.01 µs` (➖ 1.02x faster)
`262144`	`29.64 µs` (➖ 1.02x slower)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`270.05 µs` (➖ 1.28x faster)
`byte_chunks`		`957.21 ns` (➖ 1.01x faster)
`vec_bytes`		`720.37 ns` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`192.86 ns` (➖ 1.00x faster)
`262144`	`731.70 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`67.60 ms` (➖ 1.03x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`9.53 ns` (➖ 1.00x slower)	`10.11 ns` (➖ 1.06x slower)	`9.62 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`105.64 µs` (➖ 1.04x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.24 µs` (➖ 1.01x slower)	`2.40 µs` (➖ 1.01x faster)
`262144`	`9.75 µs` (➖ 1.01x faster)	`11.45 µs` (➖ 1.16x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.26 µs` (➖ 1.01x faster)	`1.34 µs` (➖ 1.04x slower)
`65536`	`8.14 µs` (➖ 1.29x faster)	`8.35 µs` (➖ 1.20x faster)
`262144`	`36.16 µs` (➖ 1.07x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`654.14 µs` (➖ 1.18x faster)
`byte_chunks`		`938.11 ns` (➖ 1.04x slower)
`vec_bytes`		`643.65 ns` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`264.86 ns` (➖ 1.06x slower)
`262144`	`671.71 ns` (➖ 1.04x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`93.12 ms` (➖ 1.11x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.71 ns` (➖ 1.07x faster)	`8.73 ns` (➖ 1.00x slower)	`8.72 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`58.39 µs` (➖ 1.05x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.29 µs` (➖ 1.01x faster)	`2.31 µs` (➖ 1.00x slower)
`262144`	`8.00 µs` (➖ 1.01x slower)	`8.06 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`893.71 ns` (➖ 1.04x faster)	`914.34 ns` (➖ 1.01x faster)
`65536`	`7.53 µs` (➖ 1.00x faster)	`7.54 µs` (➖ 1.00x faster)
`262144`	`38.87 µs` (➖ 1.07x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`1.90 ms` (➖ 1.01x faster)
`byte_chunks`		`1.17 µs` (➖ 1.02x faster)
`vec_bytes`		`776.83 ns` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`300.53 ns` (➖ 1.31x slower)
`262144`	`818.57 ns` (➖ 1.04x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`11.22 ms` (➖ 1.04x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`9.99 ns` (➖ 1.00x slower)	`9.99 ns` (➖ 1.00x faster)	`9.95 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`763.84 µs` (➖ 1.05x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.33 µs` (➖ 1.10x faster)	`2.35 µs` (➖ 1.01x slower)
`262144`	`9.37 µs` (➖ 1.11x faster)	`8.93 µs` (➖ 1.04x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.27 µs` (➖ 1.02x faster)	`1.31 µs` (➖ 1.03x slower)
`65536`	`8.88 µs` (➖ 1.00x faster)	`9.02 µs` (➖ 1.02x slower)
`262144`	`39.72 µs` (➖ 1.15x faster)

_{Reported by cargo ci bench-report --candidate run:41563146933 --config-file bench_report.toml.}

cshung and others added 3 commits November 2, 2026 08:09

cshung added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. ready-for-review PR is ready for (re-)review labels Nov 3, 2026

Copilot AI balanced review requested due to automatic review settings November 3, 2026 00:34

cshung requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners November 3, 2026 00:34

cshung marked this pull request as draft November 3, 2026 00:36

cshung removed the ready-for-review PR is ready for (re-)review label Nov 3, 2026

Copilot started reviewing on behalf of cshung November 3, 2026 00:37

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

cshung and others added 2 commits November 2, 2026 17:36

ci(aarch64): run KVM GDB tests

55da94f

ARM64 KVM debug jobs run the GDB suite before the general Rust tests. Install GDB when the runner lacks it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

This comment has been minimized.

Sign in to view

cshung force-pushed the cshung/kvm-aarch64-gdb branch from a60be9b to 55da94f Compare November 3, 2026 01:58

		#[cfg(all(gdb, target_arch = "aarch64"))]
		if self.vm.has_sw_breakpoints() {

Conversation

cshung commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

This comment has been minimized.

hyperlight-gh-bot Bot commented Nov 3, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cshung commented Nov 3, 2026 •

edited

Loading