Detect assembly overflow?

Question

I'm writing an assembly program that calculates Fibonacci numbers, but I need to find a way to detect for overflow when the numbers get too large. My current code is:

.file   "fib.c"
.text
.globl fib
    .type   fib, @function
fib:
    pushl %ebp
    movl %esp, %ebp
    movl 8(%ebp), %eax
    cmpl $0, %eax
    je .end
    cmpl $1, %eax
    je .end

    pushl %edx

    subl $1, %eax
    push %eax
    call fib
    popl %ebx
    movl %eax, %edx

    movl 8(%ebp), %eax
    subl $2, %eax
    push %eax
    call fib
    popl %ebx
    addl %edx, %eax
    jo .overflow
    popl %edx

.end:
    movl %ebp, %esp
    popl %ebp
    ret
.overflow:
    movl $-1, %eax
    ret

I thought I would be able to just use the jo test to see if there was an overflow (Around line 25), but I get a segmentation fault when I enter a number that should overflow.

Any thoughts how I can do this correctly? (BTW, running on a 32 bit machine if that matters)

Thanks, Mike

EDIT: For anyone who is interested, here is the working version. I wasn't clearing the stack and I wasn't checking for overflow on the movl after the first computation.

.file "fib.c"
.text
.globl fib
    .type   fib, @function
fib:
    pushl %ebp
    movl %esp, %ebp
    movl 8(%ebp), %eax
    cmpl $0, %eax
    je .end
    cmpl $1, %eax
    je .end

    pushl %edx  

    subl $1, %eax
    pushl %eax
    call fib
    popl %ebx
    movl %eax, %edx
    jo .overflow

    movl 8(%ebp), %eax
    subl $2, %eax
    pushl %eax
    call fib
    popl %ebx
    addl %edx, %eax
    jo .overflow
    popl %edx

.end:
    movl %ebp, %esp
    popl %ebp
    ret
.overflow:
    movl $-1, %eax
    movl $-1, %edx
    jmp .end

You posted code that works instead of the code you have a problem with. Not a good idea. Fix the snippet and show exactly where the fault occurs. — Hans Passant
– Hans Passant, Commented Mar 7, 2011 at 7:13
That code actually did cause a segmentation fault because I didn't do an overflow check on "movl %eax, %edx". I did however fix it and I am editing now with working code. — Swift
– Swift, Commented Mar 12, 2011 at 0:54
Mike, without checking, I don't believe MOV instructions set or reset any flags. I don't code in assembly much any more though, so I could very well be wrong. — jcomeau_ictx
– jcomeau_ictx, Commented Mar 12, 2011 at 10:33

jcomeau_ictx · Accepted Answer · 2011-03-07 08:01:30Z

4

since you're using recursion, I'd suspect the stack is what is overflowing, causing your segfault.

on second thought, when you hit numeric overflow, you 'ret' without cleaning up your stack first. jump to '.end' instead:


.overflow:
    movl $-1, %eax
    jmp .end

edited Mar 7, 2011 at 8:01

answered Mar 7, 2011 at 7:19

jcomeau_ictx

38.7k7 gold badges98 silver badges114 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jcomeau_ictx Over a year ago

time for me to get some sleep, I can't think clearly. I believe the answer is correct as of my last edit.

Himadri Choudhury · Accepted Answer · 2011-03-07 07:18:18Z

1

Not sure of this will fix your problem, but there are a couple of typos here where you say push instead of pushl.

answered Mar 7, 2011 at 7:18

Himadri Choudhury

10.4k6 gold badges43 silver badges47 bronze badges

Collectives™ on Stack Overflow

Detect assembly overflow?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related