gcc stack optimization

Question

Hi I have a question on possible stack optimization by gcc (or g++)..

Sample code under FreeBSD (does UNIX variance matter here?):

void main() {
   char bing[100];
   ..
   string buffer = ....;
   ..
}

What I found in gdb for a coredump of this program is that the address of bing is actually lower than that buffer (namely, &bing[0] < &buffer).

I think this is totally the contrary of was told in textbook. Could there be some compiler optimization that re-organize the stack layout in such a way?

This seems to be only possible explanation but I'm not sure..

In case you're interested, the coredump is due to the buffer overflow by bing to buffer (but that also confirms &bing[0] < &buffer).

Thanks!

@Stephen, how do you know there wasn't a typedef char* string; before main somewhere? :-) — paxdiablo
– paxdiablo, Commented Feb 23, 2011 at 8:36

paxdiablo · Accepted Answer · 2013-09-04 06:06:14Z

Compilers are free to organise stack frames (assuming they even use stacks) any way they wish.

They may do it for alignment reasons, or for performance reasons, or for no reason at all. You would be unwise to assume any specific order.

If you hadn't invoked undefined behavior by overflowing the buffer, you probably never would have known, and that's the way it should be.

A compiler can not only re-organise your variables, it can optimise them out of existence if it can establish they're not used. With the code:

#include <stdio.h>
int main (void) {
   char bing[71];
   int x = 7;
   bing[0] = 11;
   return 0;
}

Compare the normal assembler output:

main:
    pushl   %ebp
    movl    %esp, %ebp
    andl    $-16, %esp
    subl    $80, %esp
    movl    %gs:20, %eax
    movl    %eax, 76(%esp)
    xorl    %eax, %eax
    movl    $7, (%esp)
    movb    $11, 5(%esp)
    movl    $0, %eax
    movl    76(%esp), %edx
    xorl    %gs:20, %edx
    je      .L3
    call    __stack_chk_fail
.L3:
    leave
    ret

with the insanely optimised:

main:
    pushl   %ebp
    xorl    %eax, %eax
    movl    %esp, %ebp
    popl    %ebp
    ret

Notice anything missing from the latter? Yes, there are no stack manipulations to create space for either bing or x. They don't exist. In fact, the entire code sequence boils down to:

set return code to 0.
return.

Stephen Canon · Accepted Answer · 2011-02-23 01:36:58Z

8

A compiler is free to layout local variables on the stack (or keep them in register or do something else with them) however it sees fit: the C and C++ language standards don't say anything about these implementation details, and neither does POSIX or UNIX. I doubt that your textbook told you otherwise, and if it did, I would look for a new textbook.

answered Feb 23, 2011 at 1:36

Stephen Canon

107k20 gold badges192 silver badges277 bronze badges

3 Comments

Figo Over a year ago

I think I must have read it from some textbook about buffer overflow which assumes the layout of stack regarding how code is written:)

Karl Bielefeldt Over a year ago

Although you are correct about the standard, textbooks have a habit of overly simplifying things at first, then correcting for specifics in more advanced courses. For example, you first learn Newtonian physics, then later they tell you it doesn't work close to the speed of light. It wouldn't surprise me for a compiler text to do the same.

Stephen Canon Over a year ago

@Karl: There's a difference between "overly simplifying" and just being incorrect. Newtonian mechanics is a great approximation at non-relativistic speeds; the claim that the ordering of variables necessarily has any effect on how they are laid out in memory is not a valid approximation or simplification.

Collectives™ on Stack Overflow

gcc stack optimization

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related