This challenge is similar to hashcalc1. I used the same method as hashcalc1 to solve it. So read my hashcalc1 writeup first.
The binary uses inetd for running as network service. So the socket fds are 0,1,2. We do not need dup2() like hashcalc1. Also, the calculating hash function is changed. No calling any libc function.
The next libc function call is vsprintf(). But we cannot use it because the GOT entry address is 0x08049108. The address plus 2 is 0x0804910a. 0x0a is bad char.
The next function is strlen() again :). But the program calls vsprintf() with buffer size only is 0x100. To be safe, we should not overflow it. My workaround of this problem is put 0x00 after format string payload. Like this.
payload = payload_fmt + "\x00" + "A"*(0x2e0-len(payload_fmt)-1)
Another weird problem I found is server receive only partial modified GOT table. So I modified it to do recv() 2 times with small data as needed.
The others are same as hashcalc1. Here is my exploit: hashcalc2.py
$ python hashcalc2.py ** Welcome to the online hash calculator ** $ payload len: 840 got GOT table: 104 uid=1008(hashcalc2) gid=1009(hashcalc2) groups=1009(hashcalc2) funkyG_1S_th3_b3$t