7

I am trying to decompile an executable for the 68000 processor into C code, replacing the original subroutines with C functions one by one.

The problem I faced is that I don't know how to make gcc use the calling convention that matches the one used in the original program. I need the parameters on the stack to be packed, not aligned.

Let's say we have the following function

int fun(char arg1, short arg2, int arg3) {
    return arg1 + arg2 + arg3;
}

If we compile it with

gcc -m68000 -Os -fomit-frame-pointer -S source.c

we get the following output

fun:
    move.b 7(%sp),%d0
    ext.w %d0
    move.w 10(%sp),%a0
    lea (%a0,%d0.w),%a0
    move.l %a0,%d0
    add.l 12(%sp),%d0
    rts

As we can see, the compiler assumed that parameters have addresses 7(%sp), 10(%sp) and 12(%sp):

illustration of the unwanted parameter positioning on the stack

but to work with the original program they need to have addresses 4(%sp), 5(%sp) and 7(%sp):

illustration of the desired parameter position

One possible solution is to write the function in the following way (the processor is big-endian):

int fun(int bytes4to7, int bytes8to11) {
    char arg1 = bytes4to7>>24;
    short arg2 = (bytes4to7>>8)&0xffff;
    int arg3 = ((bytes4to7&0xff)<<24) | (bytes8to11>>8);
    return arg1 + arg2 + arg3;
}

However, the code looks messy, and I was wondering: is there a way to both keep the code clean and achieve the desired result?

UPD: I made a mistake. The offsets I'm looking for are actually 5(%sp), 6(%sp) and 8(%sp) (the char-s should be aligned with the short-s, but the short-s and the int-s are still packed):

illustration of the updated desired parameter position

Hopefully, this doesn't change the essence of the question.

UPD 2: It turns out that the 68000 C Compiler by Sierra Systems gives the described offsets (as in UPD, with 2-byte alignment).

However, the question is about tweaking calling conventions in gcc (or perhaps another modern compiler).

Improve this question
7
  • 2
    According to this the 68000 and 68010 don't support unaligned access, but the 68020 does. So I would try -m68020. Another possibility is fun(int base) with struct packedArgs *args = &base; where struct packedArgs is a packed structure containing the three arguments. Commented Jun 26, 2020 at 18:16
  • 2
    Reverse the order of the arguments Commented Jun 26, 2020 at 19:46
  • 2
    @user3386109: -m68020 isn't going to change the ABI / calling convention; it might be necessary (so good point) but not sufficient. Commented Jun 26, 2020 at 20:28
  • 2
    With the update your question makes much more sense. Passing the parameters via 4(%sp), 5(%sp) and 7(%sp) would be very expensive on 68000/68010. Commented Jun 29, 2020 at 10:10
  • 3
    Your update is actually a huge change. The first version was super-weird, with a misaligned short and misaligned long, a calling convention no sane compiler would ever use, and literally not usable on 68000 unless the callee uses multiple instructions to load the bytes separately and shift them together. (If comments are correct that 68000 won't do unaligned loads at all; I forget). A calling convention with 2-byte stack slots is totally plausible alignof(int32_t) = 2, if m68k supports 4-byte loads with 2-byte alignment. Commented Jun 29, 2020 at 11:02

3 Answers 3

Reset to default
4

Here's a way with a packed struct. I compiled it on an x86 with -m32 and got the desired offsets in the disassembly, so I think it should still work for an mc68000:

typedef struct {
    char arg1;
    short arg2;
    int arg3;
} __attribute__((__packed__)) fun_t;

int
fun(fun_t fun)
{

    return fun.arg1 + fun.arg2 + fun.arg3;
}

But, I think there's probably a still cleaner way. It would require knowing more about the other code that generates such a calling sequence. Do you have the source code for it?

Does the other code have to remain in asm? With the source, you could adjust the offsets in the asm code to be compatible with modern C ABI calling conventions.

I've been programming in C since 1981 and spent years doing mc68000 C and assembler code (for apps, kernel, device drivers), so I'm somewhat familiar with the problem space.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

For ARM processors there's a qualifier __packed that can be used for function parameters. I stumbled upon it in this question. Maybe there's something similar for 68000?
@Glinka: That's an armcc feature, not GCC. Also, it corresponds to GCC __attribute__((aligned(1))) for the pointed-to data, not for the arg itself on the stack. From the linked Q&A, I don't see any hint that it changes the ABI for stack args, or any reason why it would. So despite the name and syntax, I'm pretty sure that does something else and isn't what we're looking for.
3

It's not a gcc 'fault', it is 68k architecture that requires stack to be always aligned on 2 bytes. So there is simply no way to break 2-byte alignment on the hardware stack.

but to work with the original program they need to have addresses 4(%sp), 5(%sp) and 7(%sp):

Accessing word or long values off the ODD memory address will immediately trigger alignment exception on 68000.

Improve this answer

Comments

2

To get integral parameters passed using 2 byte alignment instead of 4 byte alignment, you can change the default int size to be 16 bit by -mshort. You need to replace all int in your code by long (if you want them to be 32 bit wide). The crude way to do that is to also pass -Dint=long to your compiler. Obviously, you will break ABI compatibility to object files compiled with -mno-short (which appears to be the default for gcc).

Improve this answer

2 Comments

I can't find -mlong on the Option Summary page of GCC online docs, but the M680x0 Options page says that -mno-short is the default.
You are right, I picked that information from a page about the tigcc fork, which apparently is not in mainline gcc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.