Someone passed me a binary and said ‘Here, you’ll enjoy this one’. I most certainly did…
The guy also gave one hint, which I needed later. I started checking out the binary, which turned out to be a 32-bit Windows PE executable. I downloaded an XP VM and an old, familiar friend: OllyDbg 1.10, arguably the best debugger for Windows.
Starting up the binary, I was presented with a simple dialog:
I tried entering a string to check, but it didn’t do anything. I set a few breakpoints, but when I switched back to the dialog, the binary crashed!
What was going on here? IsDebuggerPresent was not referenced in the code, but maybe something else was going on. I set a memory breakpoint on a piece of code where I previously had set a breakpoint and pressed F9.
Then, the binary stopped here:
Quite interesting! The binary checks itself for 0xCC bytes, aka INT 3. There were two of those checks, one at 0x4025e0 and one at 0x402536. I modified the following JNZ to a JMP so the code would never do anything when it found a breakpoint. Now I could set breakpoints where ever I liked!
Back to the SendDlgItemMessageA calls.
The binary checks the length of our input and if it is exactly 0x1D or 29 bytes, it continues by fetching the input. It checks the input for - characters at specific places. From this I deduced that the format of the input should be ABCD-DEFG-HIJK-LMNO-PQRS-TUVW. I entered that and pressed check, ending up here:
The code now sends a message to its message queue, but I didn’t know the location of the handler. No worries; I stepped into the call with F7 until I was in NTDLL.dll, and then set a memory breakpoint on the code area at 0x401000. A break-on-access also works.
After pressing F9, this landed me at the handler!
I traced through this function with F7, finally ending up here:
The code takes the first four bytes of our input, XORs them with a certain value and then proceeds to call the code at the resulting value… Only problem is, there was no code at 0x703234BD! I remembered the hint I got at the beginning: “the flag probably starts with HV14”. I changed the input to HV14-ABCD-DEFG-HIJK-LMNO-PQRS and restarted it again, to end up at the same CALL EAX:
With a single F7, I ended up here:
Here, some tricky stuff starts happening! A call to VirtualProtect makes the code in front of us writeable, and the binary starts modifying that code:
Cool and a nice anti-disassembler tactic, this self-modifying code. It can be tricky to reverse, in this case especially the call that emerges:
Carefully using F7, I traced passed this anti-disassembler trick and found myself at this CALL ECX:
I stepped into it, landing at this position. Now it becomes really interesting, the binary apparently calls a function that does something with our input, then sends another message to the message queue. I decided to place a breakpoint at 0x4022B3 and pressed F9; I would examine the function later. First see what happens:
Indeed, we land at the message handler. After decrypting a string, the binary does a byte-by-byte comparison of our mangled input and some other buffer:
The code that comes after it congratulates us, but only if the buffer equals our mangled input. I decided to find out where our input was being mangled. I restarted the binary and set a memory breakpoint on the first four bytes of the input:
Pressing F9, the code breaks at the CALL EAX, of course. Another F9 lands us here:
Ah! The first byte of our input, ‘H’, is being XOR’ed with another value. This is repeated for all the bytes in the input:
I wrote down (literally!) all the values that were used in AL and finally, I ended up at the REPE CMPS instruction:
I took note of the values at EDI and together with the values from the XOR statement earlier, I had all the thing necessary to grab the flag! Sprinkle in some Python magic:
And we have the flag!
It has been a while since I got to use OllyDbg to reverse a Windows binary. Between the self-modifying code, the use of the flag and SendMessage to control code execution, this was a very enjoyable challenge!