This challenge was solved by and the write up was written by one of my teammates, tukan.
The binary implements matrix multiplication.
By specifying a large matrix size, we can trigger an alloca with the size under our control.
Using this, it is possible to pivot the stack pointer to a higher address.
By properly aligning the stack, the scanf function will then eventually overwrite its own saved frame pointer with one of our matrix element input.
Upon returning from main, this gets into rsp.
We can use this to move the stack pointer into a user-buffer at a static location in .bss.
Once done, with some ROP we leak the address of puts from the GOT, calculate the address of system and spawn a shell.
High quality exploit follows: