01 / Challenge Overview

The challenge provided two files: a stripped 64-bit ELF executable and a binary file named program.bin. There were no symbols and no hints, other than the description implying it was a custom architecture.

02 / Initial Triage

As always, I started with basic static and dynamic analysis to understand the lay of the land.

$ BASH 
$ file challenge
challenge: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, stripped
$ strings challenge | head -20
/lib64/ld-linux-x86-64.so.2
libc.so.6
fopen
fread
malloc
...

The presence of fopen and fread strongly suggested the executable was reading the program.bin file.

03 / Identifying the VM Dispatcher

Loading the binary into Ghidra, I navigated to the main function. The setup was typical for a VM challenge: an initialization block allocating memory for a "virtual stack" and "registers", followed by a massive switch/case statement inside a while loop.

This central loop is the VM Dispatcher. It fetches the next byte from the loaded program.bin buffer, advances the Instruction Pointer (IP), and executes the corresponding case.

04 / Reconstructing the Instruction Set

By analyzing the operations within each switch case, I mapped out the custom instruction set. The VM was stack-based, meaning most operations occurred on the top of the virtual stack.

  • 0x01 (PUSH): Read the next 4 bytes, push onto the stack.
  • 0x02 (POP): Pop the top value off the stack.
  • 0x03 (ADD): Pop two values, add them, push the result.
  • 0x04 (XOR): Pop two values, XOR them, push the result.
  • 0x0A (CMP): Compare top two values, set a virtual flag register.
  • 0x0B (JMP): Jump to address if the flag is set.

05 / Writing the Disassembler

Analyzing raw hex bytecode is tedious. To understand the logic in program.bin, I wrote a quick Python disassembler based on my instruction set mapping.

$ PYTHON 
# vm_disasm.py — Custom VM bytecode disassembler
OPCODES = {
    0x01: ('PUSH', 4),
    0x02: ('POP',  0),
    0x03: ('ADD',  0),
    0x04: ('XOR',  0),
    0x0A: ('CMP',  0),
    0x0B: ('JMP',  4),
}

def disassemble(bytecode):
    pc = 0
    while pc < len(bytecode):
        op = bytecode[pc]
        if op in OPCODES:
            name, size = OPCODES[op]
            operand = bytecode[pc+1:pc+1+size]
            print(f'{pc:04x}: {name} {operand.hex()}')
            pc += 1 + size
        else:
            print(f'{pc:04x}: UNKNOWN {hex(op)}')
            pc += 1

06 / Bytecode Analysis

Running the disassembler over program.bin yielded readable assembly. The logic was a classic flag checker: it read input, looped over each character, XORed it against a hardcoded static array, and compared the result to an expected buffer.

07 / Flag Extraction

Since the XOR operation is reversible, I extracted the expected buffer and the static XOR key from the bytecode. A simple Python script performed the inverse operation, revealing the flag.

Key Takeaways: VM challenges require identifying the architecture type (stack vs register), mapping the dispatcher loop, and building tooling to translate the custom bytecode into a readable format.

Tools Used

[Ghidra] [GDB] [Python] [pwntools] [ltrace]

Practice Questions

  1. How does identifying whether a VM is stack-based or register-based change your reverse engineering approach?
  2. What patterns in a switch statement suggest a VM dispatcher in Ghidra?
  3. How can you automate the mapping of opcodes if the VM is highly obfuscated?