This challenge was an image conversion service which expected a PNG file and converted into a PPM file. PPM is a simple text-based image format, here is the wikipedia page describing it: https://en.wikipedia.org/wiki/Netpbm_format
The main vulnerability was a stack buffer overflow, because it allocated a buffer with the size of width * height * bitsPerPixel and copied the PNG’s decompressed zlib data (except the filter field) to the buffer which does not respected the allocated buffer’s size.
The main problem was that we also overwrote the stack canary and it was a 64-bit ASLR-enabled PIE binary (as the challenge website stated so) which did not fork, so new canary generated for every new connection:
Luckily there was an other vulnerability: the PNG file format uses a method called Filtering which can help to reduce the size of the file by only storing the differences with the pixel to the left or the pixel above etc. You can read about it here: https://en.wikipedia.org/wiki/Portable_Network_Graphics#Filtering
So if we tried to read the previous row in first row, then it read before our buffer. This way we could read the stack cookie from the previous function call. Although I am not 100% sure, but I think the binary was compiled with -fstack-protector-all, because there was stack cookie in every function. In this case maybe this paranoid setting caused more harm than good… :)
I had to experiment a bit with the width / height / with or without alpha channel options. As I had to switch between the “copy the above line” (2) filter (for the stack cookie) and “copy the exact values” (0) filter (for the return address). Finally I created a 6*1 PNG with alpha channel.
This way we could overwrite the stack cookie and we had RIP control. BUT as the binary was a PIE binary we had not fix addresses where we could jump. Or do we?
Basically the return address points to a valid memory address, to the next instruction after the sub_1560 call. So if we partially overwrite the return address, we can jump somewhere in the png2ppm binary.
Practically I jumped to the program’s start again as at this point I already had a lot of leaked addresses as the result of the conversion contained the stack cookie, an address from the stack, and an address from the png2ppm binary (although I had to mix the address bytes from the pixelmap and the alpha map, because the conversion divided the values into two different files).
Only one thing separated me from getting a shell: a libc address :)
Because I know the program’s base address at this point, I could jump to the puts PLT and print out the GOT table. So I got the puts function’s libc address and calculated the system’s address from it.
So in the next ROP chain I could call the system with “sh”. For this I had to calculate the “sh” string’s address from the stack (I could calculate this, because I already had a stack address leak).
Because of the partial overwrite, I also overwrote 4 bits from the ASLR random part, so for the real server I had to run the exploit multiple times to hit the correct address with a 1/16 change (locally I debugged it with disabled ASLR).
Finally I could print the flag out: