Passing Function Pointer in C#

void funct(void*(*handler)(void*));

is a function that accepts a pointer and returns a pointer.

In C# it would be:

IntPtr MyFunc(IntPtr ptr);

So you'll need a delegate like:

public IntPtr delegate MessageHandlerDelegate(IntPtr ptr);

[DllImport("mydll.dll")]
public static extern void register_message_handler(MessageHandlerDelegate del);

Note that P/Invoke (calling native methods) is one of the rare cases where Action<...> and Func<...> delegates don't work, and you have to build "specific" delegate.

BUT to call it, it's a little complex, because you must save a "copy" of the delegate so that if the C functions calls this method at any time, this copy is still "alive" and hasn't been GC (see for example https://stackoverflow.com/a/5465074/613130). The most common way to do it is to encapsulate everything in a class, and put the copy in a property/field of the class:

class MyClass
{
    public delegate IntPtr MessageHandlerDelegate(IntPtr ptr);

    [DllImport("mydll.dll")]
    public static extern void register_message_handler(MessageHandlerDelegate del);

    [DllImport("mydll.dll")] 
    public static extern IntPtr message_handler(IntPtr message);

    public MessageHandlerDelegate Del { get; set; }

    public void Register()
    {
        // Make a copy of the delegate
        Del = Handler;
        register_message_handler(Del);            
    }

    public IntPtr Handler(IntPtr ptr)
    {
        // I don't know what ptr is
        Console.WriteLine("Handled");
        return IntPtr.Zero; // Return something sensible
    }
}

Note that if you use IntPtr then you don't need the unsafe.

If you want to pass message_handler to register_message_handler the safest way is to

// Make a copy of the delegate
Del = message_handler;
register_message_handler(Del);

There is a possibility that you can do directly no there isn't, checked

register_message_handler(message_handler);

and that the CLR will solve this, BUT I'm not sure of this... I can't easily test it, and I wouldn't do it. (if you want to test it, add a GC.Collect() just after the register_message_handler. If after some time you receive a CallbackOnCollectedDelegate error then you know you can't do it :-) )

Mmmh... checked. You can't do the raw register_message_handler(message_handler), you have to use MyClass.

Be very aware of something: it's better to always specify the calling convention C-side and C#-side even in function pointers. C# uses stdcall, while C uses cdecl. In x86 mode you can get very awful silent crashes (in x64 there is a single calling convention)

void __stdcall register_message_handler(void* (__stdcall *handler)(void*));
void * __stdcall message_handler(void *raw_message);

and C# side

[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate IntPtr MessageHandlerDelegate(IntPtr ptr);

[DllImport("Win32Project1.dll", CallingConvention = CallingConvention.StdCall)]
public static extern void register_message_handler(MessageHandlerDelegate del);

[DllImport("Win32Project1.dll", CallingConvention = CallingConvention.StdCall)]
public static extern IntPtr message_handler(IntPtr message);

(or everywhere cdecl)