14
$\begingroup$

I have an external function called increment (for the sake of illustration) that I call from Mathematica using MathLink. I.e., 

Install[<path-to-mathlink-executable>]

which is defined as

?increment
increment[x_Real]:=ExternalCall[LinkObject[<path-to-mathlink-executable>,10,10],CallPacket[0,{x}]]

It works fine when I call it like this:

result = Map[increment[#]&, N@Range[10]]

{2., 3., 4., 5., 6., 7., 8., 9., 10., 11.}

When I try to do this in parallel using a ParallelMap[],

ParallelMap[increment[#] &, N@Range[10]]

I get ten $Failed results, and messages like

LinkObject::linkn: Argument LinkObject[<path-to-mathlink-executable>,10,10]
in LinkWrite[LinkObject[<path-to-mathlink-executable>,10,10],
CallPacket[0,{1.}]] has an invalid LinkObject number; the link may be closed.

But running the non-parallel version (Map[]) again works correctly, proving that the link is not closed.

This is on MacOSX Lion, with two CPUs, running Mathematica 8.

Is there a way to run an external command, set up by MathLink, in parallel?

Improve this question
$\endgroup$
6
  • $\begingroup$ Did you make any progress on this? Though I use LibraryLink now, which I find superior for simple applications, I am interested too. $\endgroup$ Commented Jan 24, 2012 at 15:40
  • $\begingroup$ @Szabolcs Thanks for the question - Mma no longer complains about closed links (I had to run LaunchKernels[]), but the processing is seemingly not parallelized as it takes as long to do the calculations in parallel as it does on a single processor. I may edit the question to follow up on this, or ask a separate question. $\endgroup$ Commented Jan 24, 2012 at 20:02
  • $\begingroup$ Did you try the approach described in my answer? As I said I suspect it may have an issue, but in my quick test it worked fine and I managed to run several calculations in parallel. Actually I wanted to know if that issue I mentioned came up. Maybe you are connecting to the same executable from all parallel kernels (if that's possible at all..)? $\endgroup$ Commented Jan 24, 2012 at 20:19
  • $\begingroup$ I'm wondering if it is connecting to the same executable from all kernels. Actually, my external executable is linked against an external library (gel) which itself might be shared amongst the kernels...Maybe I have to parallelize the external function? $\endgroup$ Commented Jan 24, 2012 at 20:33
  • $\begingroup$ @Szabolcs Had to run ParallelNeeds[] (in addition to Needs[]; see docs) to get this to work, since my code uses stuff from other packages. Now it is FAST! I added a note about ParallelNeeds[] in your answer, if you don't mind. But I did indeed follow all else from your answer, so will accept that one. $\endgroup$ Commented Jan 24, 2012 at 21:29

1 Answer 1

Reset to default
6
$\begingroup$

First I'd like to mention that I am currently using LibraryLink, and that LibraryFunctions can be shared with parallel kernels with DistributeDefinitions. When using Parallelize, this happens automatically. (I'm assuming that you have local parallel kernels. For remote ones, see here.)

For MathLink programs, this will not work. Let's try it:

First, let's launch the kernels:

In[6]:= LaunchKernels[]

Then install the addtwo example into the main kernel:

In[7]:= SetDirectory[$InstallationDirectory <> 
   "/SystemFiles/Links/MathLink/DeveloperKit/" <> $SystemID <> 
   "/PrebuiltExamples/"];

In[8]:= link = Install["addtwo"]    
Out[8]= LinkObject["addtwo", 23, 11]

It will not work in parallel kernels even after distribution the definitions because the connection exists only between addtwo and the main kernel:

In[9]:= DistributeDefinitions[AddTwo]    
Out[9]= AddTwo

In[10]:= ParallelEvaluate[Print@AddTwo[2, 2]]

(kernel 2) LinkObject::linkn: Argument LinkObject[addtwo,23,11] in LinkWrite[LinkObject[addtwo,23,11],CallPacket[0,{2,2}]] has an invalid LinkObject number; the link may be closed.

(kernel 1) LinkObject::linkn: Argument LinkObject[addtwo,23,11] in LinkWrite[LinkObject[addtwo,23,11],CallPacket[0,{2,2}]] has an invalid LinkObject number; the link may be closed.

(kernel 2) $Failed

(kernel 1) $Failed

Out[10]= {Null, Null}

Now lets install it into each parallel kernel:

In[11]:= ParallelEvaluate[Install["addtwo"]]    
Out[11]= {LinkObject["addtwo", 6, 6], LinkObject["addtwo", 6, 6]}

In[12]:= ParallelEvaluate[Print@AddTwo[2, 2]]

(kernel 1) 4

(kernel 2) 4

Out[12]= {Null, Null}

As you can see, now it works in each of them, but the addtwo process will also exist in three copies (on a two-core machine). Note: you will have to run ParallelNeeds[] if you load any packages required by the sub kernels.

I must note that unlike with LibraryLink, I have no practical experience with this.

Caveat: This problem didn't come up in my tests, but I think it may be possible that the symbol AddTwo will get auto-synchronized between the kernels, so they might lose the connection to their own addtwo process. This could be avoided by placing the AddTwo symbol into a different context than Global ` (see here and here), but unfortunately you won't have control over this unless you are writing your MathLink programs yourself.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Oops, I must have missed that example, sorry. Removing my comment, and +1. $\endgroup$ Commented Jan 18, 2012 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.