Re: Q1, If you're creating your child processes using a multiprocessing.Pool, then no, the worker processes cannot have children. Attempting to create one will raise an exception:
AssertionError: daemonic processes are not allowed to have children
The reason is stated pretty clearly - the processes in a Pool are daemonic, and daemonic processes can't have children. The reason for this is that terminating the parent process will terminate its daemonic children, but the daemonic children will not be able to terminate their children, which will leave behind orphaned processes. This is stated in the documentation:
Note that a daemonic process is not allowed to create child processes.
Otherwise a daemonic process would leave its children orphaned if it
gets terminated when its parent process exits.
You can get around this by your parent processes creating a set of non-daemonic Process objects, rather than using a Pool. Then, each child can create its own multiprocessing.Pool:
import multiprocessing
def subf(x):
print "in subf"
def f(x):
print "in f"
p = multiprocessing.Pool(2)
p.map(subf, range(2))
if __name__ == "__main__":
processes = []
for i in range(2):
proc = multiprocessing.Process(target=f, args=(i,))
proc.start()
processes.append(proc)
Output:
in f
in f
in subf
in subf
in subf
in subf
This approach seems like it will work ok for you, since your initial dataset just contains four items. You can just create one Process per item in the dataset, and still have some free CPUs to spare for each sub-process to use in a small Pool.
Re: Q2, it sounds like you could use itertools.product to create one large iterable of each pair of tuples you want to compare. You can then use pool.map to parallelize comparing each pair. Here's an example showing how that works:
def f(x):
print(x)
if __name__ == "__main__":
# Create two lists of tuples, like your use-case
x = zip(range(3), range(3,6))
y = zip(range(6, 9), range(9, 12))
pool = multiprocessing.Pool()
pool.map(f, itertools.product(x, y))
Output:
((0, 3), (6, 9))
((0, 3), (7, 10))
((0, 3), (8, 11))
((1, 4), (6, 9))
((1, 4), (7, 10))
((1, 4), (8, 11))
((2, 5), (6, 9))
((2, 5), (8, 11))
((2, 5), (7, 10))
