The problem is that you're not properly marshalling the data between the two applications. You are allocating memory in one application, and then passing a pointer to that memory to a second application. But because application's have private memory spaces and cannot read each other's memory, that pointer is useless to the second application.
Depending on what it points to in that application's memory space, it might be triggering an access violation, or just not working properly.
Perhaps you're confused by the naming of the AllocHGlobal and FreeHGlobal functions. Contrary to what first impressions might suggest, they do not actually allocate and free global memory. At least not memory that is globally accessible to all processes running on a machine. The name comes from the Windows HGLOBAL data type, which used to mean exactly this back in the days of 16-bit Windows, where all applications did share a common memory space and could read each others' memory. But that is no longer the case in modern 32-bit Windows. The names were retained for backwards compatibility reasons. HGLOBAL and HLOCAL mean effectively the same thing nowadays. More on the nitty gritty details is available on MSDN. But that's mostly a curiosity. You don't need to know and understand all of it to get the code working correctly.
The point is that all AllocHGlobal does is allocate memory from the process's default heap, readable only to that process. Hence the need to marshal the memory across processes, making it accessible from the other one receiving the message. Doing this manually is, of course, an option. But not a very good one. It's tricky to get right, and there's little point. Like Tim's comment hints at, the easier option is to use the WM_COPYDATA message, which does the marshalling for you. When you use this message, the data you want to share is packaged in a COPYDATASTRUCT structure.
You can keep most of your existing code to allocate memory, you just need to replace your custom RF_TESTMESSAGE window message with WM_COPYDATA. Sample code, including the necessary structure definition, is available over on the pinvoke website.
Something like this (warning—untested and uncompiled):
Private Const WM_COPYDATA As Integer = &H004A
<StructLayout(LayoutKind.Sequential)> _
Public Structure COPYDATASTRUCT
Public dwData As IntPtr
Public cdData As Integer
Public lpData As IntPtr
End Structure
<StructLayout(LayoutKind.Sequential)> _
Public Structure MyData
Public M As String
Public I As Integer
End Structure
Public Function SendTest()
' Create your data structure, MyData, and fill it.
Dim data As New MyData
data.M = "QWERTY"
data.I = 15
Dim pData As IntPtr = Marshal.AllocHGlobal(Marshal.SizeOf(data))
Marshal.StructureToPtr(data, pData, False)
' Create the COPYDATASTRUCT you'll use to shuttle the data.
Dim copy As New COPYDATASTRUCT
copy.dwData = IntPtr.Zero
copy.lpData = pData
copy.cbData = Marshal.SizeOf(data)
Dim pCopy As IntPtr = Marshal.AllocHGlobal(Marshal.SizeOf(copy))
Marshal.StructureToPtr(copy, pCopy, False)
' Send the message to the other application.
SendMessage(New IntPtr(11111), WM_COPYDATA, IntPtr.Zero, pCopy)
' Free the memory we allocated
' (This works because SendMessage is synchronous, and does not
' return until the other application has finished processing
' the data that you have sent it. That also means that the
' other application should not and cannot free the memory.
' If it needs it after processing the message, it needs to
' make a local copy.)
Marshal.FreeHGlobal(pCopy)
Marshal.FreeHGlobal(pData)
End Function
If you decide not to go the easy route using WM_COPYDATA and instead marshal the data yourself, you need to make sure to call the RegisterWindowMessage function (if you are not doing so already in code I can't see) to ensure that the ID of your custom window message is unique.
WM_COPYDATA. A good article explaining this procedure can be found at Simple Interprocess Communications using WM_COPYDATA.