Skip to content

Hardening layers

Five independent layers stack on top of the base bytecode interpreter, each behind its own knob and each adding runtime cost independently. Below them sit the structural features that change the shape of the interpreter itself.

flowchart TD
    L5["Layer 5 · Anti-debug timing gates (antiDebug)"] --> L4
    L4["Layer 4 · Structural hardening (hardened)"] --> L3
    L3["Layer 3 · Register-value XOR (regEncrypt)"] --> L2
    L2["Layer 2 · AES-128-CTR bytecode (encBytecode)"] --> L1
    L1["Layer 1 · Register-index XOR (obfRegIdx)"] --> CORE["bytecode interpreter core"]

Layer 1 — Register-index XOR (obfRegIdx, default on)

Every register-index byte in the stream is XOR'd with salt & 0xFF at compile time; each handler re-XORs the loaded byte with the same volatile salt before indexing:

real_slot = bytecode_slot ^ (vm.salt & 0xFF)

The volatile load blocks constant-folding — a static analyst sees an index depending on an opaque value. Disable with obfRegIdx=0.

Layer 2 — AES-128-CTR bytecode encryption (encBytecode, default on)

A per-function 128-bit AES key (from the RNG hierarchy) encrypts @<fn>.vm.bytecode. A .init_array constructor calls __obf_aes_ctr_decrypt(...) — the same runtime stub strenc uses — to decrypt in place before main().

  • lazyDecrypt=1 (needs encBytecode): bytecode stays ciphertext at rest; each fetched byte is decrypted on demand from a recomputed AES-CTR keystream block. The engine gains a lazyctx parameter (round key + nonce + keystream cache). A memory dump never sees contiguous plaintext bytecode. The per-byte recompute is branchless (fetch helpers stay straight-line).
  • constInStream=1 (needs encBytecode): integer/FP constants are spliced into the encrypted stream as OP_LOADI* prologue instead of plaintext stores, so they live inside the AES stream, not in cleartext IR. Pointer constants stay in the wrapper (they carry relocations, aren't secret).

Layer 3 — Register-value XOR (regEncrypt, default off)

A per-slot XOR key table per register file. Every read decrypts, every write encrypts:

stored = actual ^ key[slot]
actual = stored ^ key[slot]

Defeats memory-dump attacks that read the register file at runtime. Opt-in because of the per-access overhead.

Layer 4 — Structural hardening (hardened=1)

A secondary round of obfuscation on the wrapper and (on first build) __vm_engine itself:

  • hardenWrapper — split the wrapper, insert junk, opaque predicates around the tail call.
  • mbaHardenWrapper — MBA on wrapper arithmetic.
  • flattenWrapper — switch-dispatch flattening of the wrapper.
  • hardenVMEngine — MBA + opaque predicates inside the shared engine handlers.
  • buildIntegrityHashCtor — an FNV-1a hash of the decrypted bytecode checked against a compile-time value; on mismatch the salt is corrupted → silent incorrect execution.
  • buildCalleeXorCtor — XOR-masks callee-table entries, un-masked at dispatch.

Layer 5 — Anti-debug timing gates (antiDebug, needs hardened=1)

  • Dispatch-level gate: every adDispatchInterval fetches (default 64), check whether the RDTSC delta exceeds adDispatchThreshold cycles (default 5000). If exceeded (single-step / breakpoint), the salt is XOR'd with a poison key → all later register-index deobfuscation yields wrong slots → silent corruption.
  • Handler spot-checks: adHandlerProb% of handlers (default 10%) get an inline RDTSC check against adHandlerThreshold. To avoid scheduling-noise false positives, a trap poisons only after kDebounce consecutive slow executions (a latch stops the self-inverse XOR cancelling).

Salt corruption yields silently wrong results, not a crash — harder to diagnose under a debugger.

Timing traps can flake; prefer key-binding

RDTSC handler traps historically caused rare timing flakes. bindAntiDebug=1 (needs hardened) is the sturdier option: a priority-100 .init_array ctor — running before the AES-decrypt ctor — reads IsDebuggerPresent / CheckRemoteDebuggerPresent / NtQueryInformationProcess(ProcessDebugPort) and XORs a mask into the AES round key. Under a debugger the key is wrong, the bytecode decodes to garbage, and the program dies before a single opcode runs. When bindAntiDebug is on, the handler-level RDTSC traps are skipped entirely.

Structural features

These change the interpreter's shape — dispatch, ISA encoding, engine count — rather than layering encryption on a fixed interpreter. All default off (unless noted); each is byte-identical to the previous build when its knob is off, and they compose with each other and the layers above.

Dispatch shape

Knob Effect
threadedDispatch Inline dispatch into every handler — no single central dispatch loop to anchor analysis on.
keyedDispatch Key each opcode byte by the instruction pointer, defeating a static byte→handler map.
superOps Fuse mul+add, shl+add, icmp+select, and+icmp==0/!=0 chains into single super-opcodes.
randISA Per-build permutation of operand-field encodings — no cross-build signature.

Nested virtualisation (nestedVM)

The compute step of several hot opcodes (BINOP/BINOP64/ICMP/ICMP64/FCMP/CAST/BINOP_F) is outlined into a pure helper __vm_h_<op> that is itself virtualised — depth-2 virtualisation. Recursion is made impossible by using two distinct engines: the outer function targets __vm_engine.nest (handlers call the helper); each helper is virtualised against the plain __vm_engine (handlers compute inline).

Engine diversity

Knob Effect
enginePoolSize=N Spread functions across N structurally-distinct engines.
perFnEngine Give a function its own dedicated engine.
metamorphicEngines Diversify each engine's handler bodies (needs a pool or perFnEngine).
handlerVariants / handlerDecoys Multiple handler bodies per opcode + live decoy handlers (Handler Polymorphism II; preset=max sets handlerVariants=4, handlerDecoys=2).

By default a single shared engine per module keeps handler code from being duplicated. Engine diversity trades size for the loss of a single canonical engine to attack.