In order to make Linux machine vulnerable to overflow, I have to use these commands
sudo echo 0 > /proc/sys/kernel/randomize_va_space
and
-fno-stack-protector -z execstack -o bug bug.c
What is the purpose of using these two command before trying to overflow the machine? What security features they disable to cause buffer overflow?
sudo echo 0 > /proc/sys/kernel/randomize_va_space
This is Address Space Layout Randomization (ASLR). On Windows, this is opt-in per app via the /DYNAMICBASE or /HIGHENTROPYVA switches. The Linux command is disabling it for the whole OS, so be sure to turn it back on once you are done with your homework. See also Address Space Layout Randomization in Windows Vista.
-fno-stack-protector
This is Buffer overflow protection. On Windows, it is enabled by building with /GS. See also Compiler Security Checks In Depth for why this is useful.
-z execstack
This is Data Execution Protection. On Windows, this is opt-in per app via the /NXCOMPAT switch. See also /DYNAMICBASE and /NXCOMPAT
As to exactly why you have to disable these security features, that's probably the point of your homework.
On Windows there is one more security mechanism which is controlled by the
/SAFESEHlinker switch. See Preventing the Exploitation of Structured Exception Handler (SEH) Overwrites with SEHOP for what this is about.
