c# System.ExecutionEngineError on P/Invoke call from large arrays

Question

I'm writing a small application in c# using P/Invoke to create a window. I'm getting a System.ExecutionEngineException thrown by PeekMessage when working with large arrays. This is weird because when actually working with the arrays no exceptions are thrown and everything is fine. But when I call PeekMessage it throws. Not creating the arrays prevents the issue from happening. When the exception occurrs, this is printed to the console:

Process terminated. A callback was made on a garbage collected delegate of type 'PInvoke.User32!PInvoke.User32+WndProc::Invoke'.
   at PInvoke.User32.PeekMessage(MSG*, IntPtr, WindowMessage, WindowMessage, PeekMessageRemoveFlags)
   at PInvoke.User32.PeekMessage(MSG*, IntPtr, WindowMessage, WindowMessage, PeekMessageRemoveFlags)
   at PInvoke.User32.PeekMessage(IntPtr, IntPtr, WindowMessage, WindowMessage, PeekMessageRemoveFlags)
   at Program.Main(System.String[])

I was able to piece together a small sample below from my project which reproduces the error. I am using .NET Core 3.1.

using System;
using System.Drawing;
using static PInvoke.Kernel32;
using static PInvoke.User32;

static class Program
{
    static IntPtr HWND;

    static unsafe void Main(string[] args)
    {
        // creating the arrays on lines 14, 16, and 26 somehow cause the error and creating the window after line 18 doesn't cause the error.

        HWND = CreateWindow();

        byte[] b = new byte[100000000];

        DoSomething(new int[10000]);

        PeekMessage(HWND, HWND, 0, 0, PeekMessageRemoveFlags.PM_REMOVE);
    }

    public static void DoSomething(int[] data)
    {
        Color[] result = new Color[data.Length];
    }

    static unsafe IntPtr WndProc(IntPtr hwnd, WindowMessage msg, void* wParam, void* lParam)
    {
        return DefWindowProc(hwnd, msg, (IntPtr)wParam, (IntPtr)lParam);
    }

    static unsafe IntPtr CreateWindow()
    {
        var hIsnt = GetModuleHandle(null);

        string classname = "test";
        WNDCLASS wndclass;

        fixed (char* pClassName = classname)
            wndclass = new WNDCLASS()
            {
                lpfnWndProc = WndProc,
                hInstance = hIsnt.DangerousGetHandle(),
                lpszClassName = pClassName
            };

        RegisterClass(ref wndclass);

        var hwnd = PInvoke.User32.CreateWindow(
            classname,
            "test",
            WindowStyles.WS_CAPTION |
            WindowStyles.WS_VISIBLE,
            100, 100, 1280, 720,
            IntPtr.Zero, IntPtr.Zero,
            hIsnt.DangerousGetHandle(),
            IntPtr.Zero
            );

        return hwnd;
    }

    static unsafe void HandleEvents()
    {
        MSG msg;
        while (PeekMessage(&msg, HWND, 0, 0, PeekMessageRemoveFlags.PM_REMOVE))
        {
            TranslateMessage(&msg);
            DispatchMessage(&msg);
        }
    }
}

I'd appriciate any help you could offer!

Best to derive your own class from NativeWindow, call its AssignHandle() method. The class ensures that the object cannot be garbage-collected until the window is closed. — Hans Passant
– Hans Passant, Commented Oct 19, 2020 at 12:44

Redninja1061 · Accepted Answer · 2020-10-18 23:41:08Z

1

I got it! The problem was that the WndProc delegate passed to unmanaged code was being freed by the GC, and using large arrays was causing a garbage collection. The simple fix was to keep a reference the the WndProc delegate passed into the window class.

answered Oct 18, 2020 at 23:41

Redninja1061

313 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

c# System.ExecutionEngineError on P/Invoke call from large arrays

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related