Runtime test suite

A Python harness that compiles, obfuscates, runs, and compares output — verifying that every pass/preset/combo preserves semantics, optionally across architectures and under -O2.

Scripts live under utils/:

• obf_runtime_tests.py — main runner (a thin shim over the runner package).
• obf_cross_test_wsl.py — same runner, defaulting to the aarch64,arm32,riscv64 cross targets.
• obf_report_html.py — standalone HTML report generator.

Prerequisites

• A built LLVM tree with opt, clang (and llc for cross-arch).
• Python 3.10+.
• Optional: Graphviz (dot) for offline HTML reports.
• Optional (cross-arch): Linux + qemu-user and the cross-gcc toolchains.

Basic run

python utils/obf_runtime_tests.py \
    --build-dir <path-to-llvm-build> \
    --config Release \
    --seeds 1,2,3 \
    --inputs 24

Useful flags

Flag	Description
--list	List all cases and categories, then exit (no build dir needed).
--filter <pattern>	Run only tests whose name contains the pattern.
--category <name>	One of pass, feature, adec, budget, meta, cpp, vm, strenc, constenc, edge, options, matrix, exhaustive.
--seeds 1,2,3	Run each test with multiple seeds.
--inputs N	Randomised input pairs per test (default 24).
--quick	Reduce to 8 inputs for fast iteration.
--o2-gate	After obfuscation, run the IR through -O2 and verify semantics survive.
--targets host,…	Cross-arch matrix (default host).
--extended	Option sweeps, alias tests, and pairwise pass-interaction matrix.
--exhaustive-combos	Every k-subset of passes (2 ≤ k ≤ --combo-max-size, default 3).
--obf-report	Emit obfuscation map + CFG artifacts per run (--obf-report-out <dir>).
--work <dir> / --keep	Stable work dir; preserve artifacts on success.
--json-report <path>	Write a JSON summary of results.

Debugging a regression

Pin one seed (--seeds 1), set a stable --work path, and add --keep to preserve intermediate artifacts for inspection.

Cross-architecture

The runner can build each obfuscated function for extra architectures and verify under qemu-user:

Name	Triple	Emulator
host	local	direct exec
aarch64	aarch64-linux-gnu	qemu-aarch64
arm32	arm-linux-gnueabi	qemu-arm
riscv64	riscv64-unknown-linux-gnu	qemu-riscv64

python utils/obf_cross_test_wsl.py --build-dir <path-to-llvm-build>

Adding tests

The runner is a package under utils/runner/, with test cases in utils/cases/, IR feature gates in utils/gates/, and sample programs in utils/programs/. Adding a case is a small Python file following the existing pattern; a gate asserts an expected IR feature appeared. See the repository's docs/TESTS.md for the full "adding things" walkthrough.