Okay, there are several problems with this interop scenario, so we'll start with the byte[]. The way an array of bytes is marshalled to unmanaged code that accepts a char* is by using an IntPtr, like this:
private static IntPtr funcCalledByNativeCode()
{
byte[] bytes = getSomeBytes();
IntPtr retVal = Marshal.AllocHGlobal(bytes.Length);
Marshal.Copy(bytes, 0, retVal, bytes.Length);
return retVal;
}
Now, here's the rub. The memory allocated by the call to Marshal.AllocHGlobal() needs to be freed by a call to Marshal.FreeHGlobal(). Since you using callbacks, probably the easiest thing to do is to register a C-style destructor with the native code that it can call whenever it is finished with a block of interop memory.
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
delegate void InteropDtor(IntPtr pMem);
Which you would implement like:
private static void myInteropKiller(IntPtr allocatedMemory)
{
Marshal.FreeHGlobal(allocatedMemory);
}
Another (and simpler) way is to have the native code allocate the buffer on its side and provide it in the callback function, which you would just fill on the managed side, like this:
private static void funcCalledByNativeCode(IntPtr buffer, ref int length)
{
byte[] bytes = getSomeBytes();
lenth = bytes.Length;
if (bytes.Length <= length) // if this isn't enough, the native code can
// check 'length' for the required size
{
Marshal.Copy(bytes, 0, buffer, bytes.Length);
}
}
The other main problem is that when passing function pointers to native code from managed code, you have to make sure that the GC doesn't collect the delegate instance before the unmanaged code has a chance to use it. Normally when you call a method that expects a delegate, you just supply the name of the function that you want to use. That creates a temporary delegate instance that the GC will track just like any other object reference. The problem is that if you pass that delegate over the managed/unmanaged layer, the GC can lose all references to it, so it will collect it.
So any time you have a delegate that you are going to pass to native code as a callback, besides decorating the delegate definition with the UnmanagedFunctionPointer attribute, you need to define a static variable that will hold the actual delegate reference for the implemented method.
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
delegate void Fn_CallbackFunction(IntPtr buffer, ref int length);
private static Fn_CallbackFunction CallbackOne = myCallback;
private static void myCallback(IntPtr buffer, ref int length)
{
// implementation
}
// when calling the native function that wants the callback function
// pointer, use the variable, not the method name:
nativeRegisterFunction(CallbackOne);
byte[]toreturnof a function whose return type seemsbytevalid in C#?extern "C", so it may be the same problem.