int eax = ((int(*)())("\xc3 <- This returns the value of the EAX register"))();
How this works ? String is casted to function pointer
2 Answers
c3 is the RET instruction. When an x86 machine jumps to this string interpreted as code, it will execute RET and therefore jump right back without having done anything (the rest of the string is therefore ignored). Since standard calling convention on x86 is to put your return value in eax, but the code didn't do anything before returning, whatever was already in eax will still be there, and in a position for the C code to interpret it as having been "returned".
This is highly dependent on your machine being x86 and that you're allowed to cast between data and function pointers (and execute the result) - a very system-specific hack. This is not standard compliant or portable C by any stretch!
(\xXX is C's escape syntax for inserting single nonreadable characters into strings via their ASCII code in hex, if you didn't know that part.)
Comments
0xc3 is the ret opcode in x86. The rest of the string is only there for flavor. All of the work is done by the calling convention, and the cdecl calling convention (the default one for the C language) says that a function that returns int returns it in eax.
Your function doesn't actually do anything, it just returns as soon as it's called, but any code that calls int foo = eax() is still going to expect it to have put its return value into the eax register, so it's going to copy the contents of the eax register into foo.
c3in hexadecimal is the opcode of returning the EAX register or something like this. But really, a lot of context is missing.