This challenge was a simple reversing challenge.
The first step was NOPing out the ptrace “anti-debug” call and finding out the main hash function which was at the 0x401B40 address.
This accepted the flag as an input and used big integer math to calculate it’s hash. This hash was compared to a static buffer (there was a little trick that only every 4th number was used from that buffer).
The hash function can be summarized with this python code:
And we know that the hash of the real flag was:
So we could calculate the flag easily with the following code snippet:
The flag was: