KalmarCTF 2026 0racle Writeup — Peeling Off Heaven's Gate and Self-Modifying Code to Brute-Force FNV-1a

Overview

KalmarCTF 2026 Rev challenge “0racle” (123pts). A Windows PE32 GUI application built with Free Pascal / Lazarus, themed around the “Oracle of Delphi”. Supplying the correct input displays “The gods are pleased”. The flag is the input itself (40 characters), and the goal is to work backward to the correct answer.

At first glance it looks like a plain “input check” challenge, but before you reach the validation logic there are three shells layered on top: XOR encoding → self-modifying code → Heaven’s Gate (32-bit→64-bit switch). We peel them off one at a time.

Analysis

Step 1: Peeling Off the XOR-Encoded Strings

The strings inside the binary are encoded with XOR 0x57. Decoding them reveals the UI messages.

• "Speak your truth and i shall convene with the gods on your behalf!" — the input prompt
• "I feel a bit woozy, what were you saying?" — incorrect answer
• "The gods are pleased and commend your insight." — the success message

Following the reference to the success message reveals the entry point of the validation.

Step 2: Self-Modifying Code Keyed on the Input

At offset 0x154AC2 there is an encrypted code block (1109 bytes) embedded. It is decrypted using the first 7 characters of the input as the key.

# Decrypt: flag_data[i] ^= input[i % 7] ^ 0x57
# Validation passes when the first 7 bytes all become 0x90 (NOP)
# input[i] = flag_data[i] ^ 0xC7 → "kalmar{"

In other words, the input that makes the decrypted prefix a NOP sled (0x90 ×7) — namely kalmar{ — is exactly what is required as the key. The byte sequence after the seven decrypted NOPs is executed as x86 code. A block that looks like “just data” under static analysis alone was in fact the next stage of code.

Step 3: Heaven’s Gate (32-bit → 64-bit Switch)

The decrypted code uses Heaven’s Gate. This is an anti-reversing technique that, from within a 32-bit process running on 64-bit Windows (WOW64), swaps the CS selector to transition into 64-bit mode.

PUSH 0x33               ; CS selector for 64-bit mode
CALL +0                 ; push the return address onto the stack
ADD dword [ESP], 9      ; adjust the return address
RETF                    ; far return → switch to 64-bit mode

The adjustment ADD dword [ESP], 9 runs before RETF, while still in 32-bit mode, so its target is the 32-bit return address on the stack (esp). If you disassemble the same byte sequence as post-switch 64-bit code it appears as [rsp], but at runtime its meaning is a 32-bit [esp] operation. Because 64-bit code coexists inside a 32-bit PE, a standard decompiler cannot interpret it correctly. This was the biggest wall in this challenge.

Step 4: Extracting the 64-bit Code into Ghidra

We manually carve out the 64-bit code that lies past Heaven’s Gate and feed it into Ghidra headless as a 64-bit binary to decompile it. Three functions come into view.

1. toupper function (0x26F): uppercases the input and returns its length
2. FNV-1a hash function (0x09): standard FNV-1a (init=0x811c9dc5, prime=0x01000193)
3. polynomial evaluation function (0x11E): computes each character from a polynomial table

Step 5: Identifying the Validation Logic

The main validation function checked the input by splitting it into seven chunks and matching each independently with FNV-1a hashes.

// Check that the input is exactly 40 characters
// Split into 7 chunks and compare each against its expected FNV-1a hash
if (fnv1a(input+7,  4) == 0x1FDB82EB &&  // [7:11]
    fnv1a(input+12, 3) == 0xC498C8A6 &&  // [12:15]
    fnv1a(input+16, 5) == 0xD451383B &&  // [16:21]
    fnv1a(input+22, 3) == 0xF1D1BDC9 &&  // [22:25]
    fnv1a(input+26, 5) == 0x5768CCA7 &&  // [26:31]
    fnv1a(input+32, 3) == 0xE25D1966 &&  // [32:35]
    fnv1a(input+36, 4) == 0x45B2772B)    // [36:40]
    return SUCCESS;

There is an important observation here. The FNV-1a that performs the final match is computed over the original input (mixed case), before it passes through toupper. toupper and the polynomial check are involved in branching, but the flag match itself is done with the FNV of the raw input. This confusion matters all the way to the end (discussed later).

Note that these FNV conditions only match the seven chunk regions; they do not validate the _ characters that sit at the separator positions (offsets 11, 15, 21, 25, 31, 35). The separators are determined from the skeleton of the 40-character flag (the format where kalmar{ is followed by each chunk joined with _).

Step 6: Brute-Forcing Each Chunk

Since each chunk is short (3–5 characters), brute-forcing FNV-1a lets us enumerate candidates. However, 32-bit FNV-1a is not a collision-free hash. Running over the full printable ASCII (95 characters), the 5-character chunks pick up other preimages (collisions) (for example, 0xD451383B matches both S1gN5 and !#Mi", and 0x5768CCA7 has multiple preimages besides 3NteR). All of these collisions are strings that contain symbols. Under the flag-format assumption that the flag characters are alphanumeric, narrowing the candidate set to alphanumerics (upper- and lowercase plus digits, 62 characters) actually pins these two hashes down to exactly one each, S1gN5 / 3NteR (the other five chunks are likewise unique; enumerating the full printable set and then filtering by the format constraint yields the same conclusion). This makes the entire 40 characters a valid flag format, and we settle on the one that passes the challenge’s validation as the final flag.

def fnv1a(data):
    h = 0x811c9dc5
    for b in data:
        h ^= b
        h = (h * 0x01000193) & 0xFFFFFFFF
    return h

charset = (string.ascii_letters + string.digits).encode()  # 62 alphanumerics (avoid symbol collisions)
for c1 in charset:
    h1 = (0x811c9dc5 ^ c1) * 0x01000193 & 0xFFFFFFFF
    for c2 in charset:
        ...  # brute-force per chunk

The per-chunk results are as follows.

As for the scale of the brute force, a 3-character chunk over 62 alphanumerics is 62^3 ≈ 240,000 combinations — instant. A 5-character chunk is 62^5 ≈ 920 million combinations. A naive CPython loop struggles at minutes-scale, but since the FNV-1a prime is invertible over 2^32, working backward from the target hash / advancing the first half forward and stepping the second half back from the target to meet in the middle (meet-in-the-middle) greatly reduces the search. Reusing intermediate prefix states, a C implementation, or parallelization also brings it into a comfortably practical time (avoid the full printable 95^5 ≈ 7.7 billion, since it picks up the collisions noted above).

Flag

kalmar{M15S_Th3_S1gN5_4Nd_3NteR_tH3_M4Z3

(Exactly 40 characters. By design, the closing brace is not included.)

Pitfalls

1. The polynomial table trap: Initially the polynomial evaluation gave the candidate KALMAR{M15S_TH3_S1GN5_4ND_3NTER_TH3_M4Z3 (all uppercase), but it was unusable. Because FNV-1a is computed over the original input (mixed case), the uppercase version does not match the hash. The polynomial and toupper are involved in structural and skeleton constraints, but they do not bear on the final mixed-case flag match itself; the real target was the FNV over the raw input.
2. Heaven’s Gate: 64-bit code inside a 32-bit PE is not analyzed correctly by a standard decompiler. Only after manually extracting the code and feeding it into Ghidra as a 64-bit binary could it be read.
3. Misidentification of func_0x0000000f: Ghidra mistook a point in the middle of the FNV-1a function (offset 0x0f) for a separate function entry. In reality it is inside the same function, merely entering partway through, skipping the stack-frame prologue.
4. Choosing the charset: At first I ran only “uppercase + digits” and came up empty. Because the flag was mixed case, I widened to alphanumerics including lowercase (62 characters) to pin it down. Conversely, widening all the way to the full printable ASCII including symbols mixes in FNV collisions, so narrowing to alphanumerics is just right.

Key Insights

1. Peel the layers one at a time: XOR strings → input-keyed self-modification → Heaven’s Gate → 64-bit decompilation. Each stage is the “key” to the next. The first 7 characters are forced to be kalmar{ because that is the decryption key for the self-modification.
2. Heaven’s Gate: “cut it out into a separate binary”: For code that mixes 32/64-bit, the fastest route is to carve out the 64-bit part and feed it to the analyzer as a standalone 64-bit binary.
3. Don’t confuse which normalization is used for validation: whether the hash is taken over the toupper-processed value or the raw input. Get this wrong and even a correct brute force will never match.
4. Independent per-chunk validation is weak against brute force: the moment the 40 characters are split into chunks rather than hashed as a whole, each chunk can be cracked independently. Exploit the design weakness.