Code virtualisation — vm

The strongest transformation xollvm offers. vm compiles the entire function body into a private bytecode stream stored in a read-only global, then replaces the body with a minimal interpreter — a fetch → decode → execute → dispatch loop. No original basic blocks or instruction patterns survive in the emitted IR.

flowchart LR
    F["original function<br/>(18 blocks)"] -->|virtualise| W["thin wrapper block<br/>(vm.entry)"]
    W -->|tail call| E["__vm_engine<br/>(shared interpreter)"]
    E -->|reads| BC[("@fn.vm.bytecode<br/>(encrypted stream)")]

A reverse engineer sees only the interpreter loop and an opaque byte array. Recovering the original logic requires understanding the ISA, the per-function opcode permutation, the AES bytecode key, and — when hardened — the anti-debug and register-encryption layers.

Quick start

The easiest way in is a preset; explicit knobs override it.

// Minimal — virtualise with default layers (AES + register-index XOR)
__attribute__((annotate("obf: vm")))
int minimal(int x) { return x + 1; }

// Strongest single tier — full stack + a private metamorphic engine per function
__attribute__((annotate("obf: vm(preset=max)")))
int protected_fn(int key, int data) { return key ^ data; }

// Hand-picked hardening
__attribute__((annotate("obf: vm(hardened=1,regEncrypt=1,antiDebug=1,adDispatchThreshold=3000)")))
int licensed(int key, int data) { return key ^ data; }

preset=	Gives you
light	Structural virtualisation only.
medium	Today's defaults (bit-identical to a bare vm).
high	medium + structural hardening + threaded & IP-keyed dispatch.
max	Strongest tier — everything, plus a private metamorphic engine per function, handlerVariants=4, handlerDecoys=2.

Architecture at a glance

• Shared engine. All 56 opcode handlers live in one shared __vm_engine() per module — handler code is not duplicated per function. Only the thin wrapper and per-function globals are unique. (A build can instead use an engine pool, per-function, or metamorphic engines.)
• Per-function globals. Three private constants per virtualised function: @<fn>.vm.bytecode (encrypted stream), @<fn>.vm.ophandlers (permuted handler table), @<fn>.vm.callees (callee table).
• Four typed register files — i32, i64, double, ptr (up to 255 slots each; slot 0 is the zero/null sentinel). The interpreter's vm.ip and vm.salt are volatile allocas, so a later -O2 cannot simplify the dispatch loop away.

See the full parameter table and interaction rules below; the deep dives are:

ISA & bytecode

56 opcodes, variable-width encoding, per-function opcode permutation, compilation pipeline.

Hardening layers

Register-index XOR, AES bytecode, register-value XOR, structural hardening, anti-debug — and the structural features (nested VM, threaded/keyed dispatch, superOps, randISA, engine diversity).

Devirtualization resistance

What a lifter/-O3 recovers at each tier, and the preset that actually holds up.

Options

Key	Default	Range	Meaning
preset	—	light/medium/high/max	Canned knob bundle (above).
minBlocks	1	1–∞	Skip if the function has fewer than N blocks.
maxBlocks	400	0–∞	Skip if more than N blocks (0 = no limit).
obfRegIdx	1	0/1	XOR every register-index byte with a compile-time salt.
encBytecode	1	0/1	AES-128-CTR-encrypt the bytecode; a .init_array ctor decrypts at load.
lazyDecrypt	0	0/1	Decrypt per-fetch so bytecode stays ciphertext at rest (needs encBytecode).
constInStream	0	0/1	Hide constants inside the encrypted bytecode instead of plaintext stores.
hardened	0	0/1	MBA + opaque predicates on handlers; enables anti-debug traps.
regEncrypt	0	0/1	XOR-encrypt virtual register values at rest.
antiDebug	1	0/1	Anti-debug timing traps (effective only with hardened=1).
bindAntiDebug	0	0/1	Fold debugger detection into the AES key — wrong key under a debugger.
nestedVM	0	0/1	Virtualise hot arithmetic handlers against a second interpreter (depth-2).
threadedDispatch	0	0/1	Inline dispatch into every handler — no single central loop.
keyedDispatch	0	0/1	Key each opcode byte by IP; defeats static byte→handler maps.
superOps	0	0/1	Fuse mul+add, shl+add, icmp+select, and+icmp==0 into super-opcodes.
randISA	0	0/1	Per-build permutation of operand-field encodings; no cross-build signature.
enginePoolSize	1	1–∞	Spread functions across N structurally-distinct engines.
perFnEngine	0	0/1	Give this function its own dedicated engine.
metamorphicEngines	0	0/1	Diversify each engine's handler bodies (needs a pool or perFnEngine).
adDispatchThreshold	5000	—	RDTSC delta (cycles) for the dispatch-level timing gate.
adHandlerThreshold	5000	—	RDTSC delta (cycles) for handler-level spot checks.

useAES was removed

AES-128-CTR is now the only bytecode cipher. The old useAES knob is a no-op accepted for backward compatibility.

Eligibility

vm silently skips functions it can't virtualise and records a skip reason in the report:

Condition	Skip reason
EH pads / invoke	EH/invoke
callbr	callbr
indirectbr	indirectbr already
naked attribute	naked
block count < minBlocks	too few blocks
block count > maxBlocks	too many blocks

Interaction with other passes

Conflicts with flattening

Both restructure the entire CFG. Combining them on one function is rejected by the pipeline.

• Pre-passes help. Run mba/bcf/substitution before vm — the virtual ISA then encodes already-obfuscated logic, so the bytecode is harder to lift.
• shield / adec compose fine and protect the wrapper.
• strenc shares the same __obf_aes_ctr_decrypt runtime stub as vm's bytecode encryption.