ISA & bytecode

The virtual machine executes a private bytecode ISA. This page covers the instruction set, the per-function opcode permutation that hides it, and the two-pass compilation that produces the stream.

The ISA

• 56 logical opcodes (OP_COUNT = 0x38).
• Variable-width encoding; multi-byte immediates are little-endian.
• Recent additions: OP_LOADI64 = 0x33 (used by constInStream) and four fusion opcodes emitted by superOps: OP_MULADD = 0x34, OP_SHLADD = 0x35, OP_CMPSEL = 0x36, OP_ANDCMPZ = 0x37.

Handlers fall into six groups (arithmetic/logic, comparisons, casts, memory, control-flow/branch, call/return). Control never leaves the __vm_engine dispatch loop until an OP_RET_* returns the result to the caller.

flowchart LR
    FETCH["fetch<br/>BC[ip]"] --> DECODE["decode<br/>OpMap.decode(byte)"]
    DECODE --> EXEC["execute<br/>handler"]
    EXEC --> DISP{"OP_RET_* ?"}
    DISP -->|no| FETCH
    DISP -->|yes| RET[return to caller]

Register model

Four separate typed register files, each up to 255 slots (slot 0 = zero/null sentinel):

File	alloca type	Holds
vm.regs	[N × i32]	32-bit integers
vm.regs64	[N × i64]	64-bit integers
vm.fregs	[N × double]	floats (f32 widened to f64)
vm.pregs	[N × ptr]	pointers

The interpreter's instruction pointer vm.ip and the vm.salt are volatile allocas — the volatile is deliberate, stopping a later -O2 from folding the fetch/decode loop.

Per-function opcode permutation

The single biggest anti-signature feature. Each virtualised function gets a unique logical↔physical opcode bijection — a Fisher-Yates shuffle over all 56 opcodes seeded from the per-function RNG, stored in @<fn>.vm.ophandlers.

• The physical byte in the stream is not the logical VMOp value.
• The handler table is indexed by physical byte, so every function has a completely different dispatch table.
• The emitter writes OpMap.encode(logical_op); the interpreter recovers with OpMap.decode(physical_byte).

Result: opcode-signature pattern search across functions fails — the same logical operation is a different byte, dispatched through a different table, in every function.

Per-build ISA randomisation

randISA=1 additionally permutes the operand-field encodings per build, so two builds of the same source share no bytecode signature at all. See Hardening → structural features.

Compilation pipeline

How a function becomes bytecode:

1. Eligibility — isVMEligible() rejects EH/invoke, callbr, indirectbr, naked, and out-of-range block counts (see eligibility).
2. PHI demotion — the ISA has no PHI; all PHIs are demoted to alloca/load/store triples in a dedicated entry block, each getting a preg slot.
3. Pass 1 — slot assignment (Reverse Post-Order walk): assign register slots (args → entry allocas → SSA results), record each block's byte offset (BlockIP), build constant-materialisation lists. RPO is stable under a fixed seed, so slot assignment is deterministic.
4. Pass 2 — emission (second RPO walk): emit opcode bytes into BC[]. Register-index bytes are XOR'd with salt & 0xFF when obfRegIdx=1; forward branches are zero-filled placeholders patched after the walk; opcodes go through OpMap.encode when permutation is on.
5. IR construction — build the bytecode/callee/handler globals, select and (once) populate this function's engine, replace the body with the wrapper (buildVMEntry), and emit the AES-decrypt constructor. Hardening builders run last when hardened=1.

Inspecting emitted IR

# find the bytecode global for a function named 'secret'
opt -passes=obfuscation app.ll -S -o app.obf.ll -obf-seed=1
grep -n 'secret.*vm.bytecode' app.obf.ll

# look at the shared interpreter
grep -n '__vm_engine' app.obf.ll

See Debugging virtualised functions and, for the full opcode table and extension points, docs/VM.md in the repository.