2

I am trying to run two python files (A.py and B.py) at the same time. I tried to run them in two different and use two separate commands to run them, it works well.

The problem is, I hope to have a script file to run them in parallel. I tried multi-processing in as the following code:

if __name__ == '__main__':
    jobs=[]
    jobs.append(Process(target=A.start))
    jobs.append(Process(target=B.start))

    for job in jobs:
        job.start()

    for job in jobs:
        job.join()

The result is it runs A and B twice, which I hope them to run only once for each of them.

What is the cause for the problem and how can I solve it? Or is there any other solution that I can run two python files in parallel?

Thanks for your help in advance.

For my import information, I have three files: A.py, B.py and run.py.

In A.py, I have:

from scapy.all import *
from scapy.layers.http import HTTPRequest 
from scapy.layers.http import HTTPResponse
from colorama import init, Fore
import docker
import time
import redis

In B.py, I have:

import json
import docker
import socket
import time
import psutil
import socket
import redis
import prometheus_client
from prometheus_client import Gauge,Counter
from prometheus_client.core import CollectorRegistry
from flask import Response, Flask

And in the run.py, I have:

from multiprocessing import Process
import A
import B

====UPDATE PROBLEM CAUSE====

After hours of playing with the code, I think I found the cause, but I do not understand why.

The cause is in my A.py file, I have a code

#flask starts
app.run(host="0.0.0.0", port=15600,debug=True)

If I removed the debug mode, the code works well. The problem only occurs when the debug mode is on.

Does anyone know why it happens?

Improve this question
3
  • From what you've shown, it shouldn't run the code twice. Maybe look out for an import error? Show more code and maybe we can solve your problem Commented Sep 10, 2020 at 6:17
  • Thanks Dan, I just add the import information. I read some post which talks about the import. In one case, the flask cause the problem. But I am not certain about it. Commented Sep 11, 2020 at 1:16
  • 1
    I updated my answer with an explanation on why this is happening. In the future, please post all the relevant code and you'll get a correct answer much sooner. This has nothing to do with the use of multiprocessing, it is happening because of the way you are starting Flask. Commented Sep 13, 2020 at 17:21

2 Answers 2

Reset to default
5

I think what you are looking for is multiprocessing.Pool, and specifically Pool.apply_async. This will allow you to asynchronously start separate processes that will perform unrelated tasks in parallel. You can use it like this:

A.py:

def square(x):
    print('x squared is: ' + str(x**2))

B.py:

def cube(x):
    print('x cubed is: ' + str(x**3))

then:

import multiprocessing as mp
import A
import B

p = mp.Pool(2)

a = p.apply_async(A.square, args=(5,))
b = p.apply_async(B.cube, args=(7,))

p.close()

Update

OK, given the new info, there is a very simple but perhaps non-obvious reason this is happening, and it has nothing to do with your use of multiprocessing (although that is very likely to give you trouble down the road if you're planning to run this app in a real web server and not the one built in to Flask).

The Flask debug server starts 2 processes when you run it in debug mode with use_reloader turned on. The parent process monitors your code and when a change is detected it signals the child to reload. This is why the Flask server is able to instantly apply any code changes you make without you needing to perform any action. The problem you are having is that your code is being run in both the parent and the child.

There are 2 ways to fix it.

  1. Don't use debug=True.
  2. Use debug=True but also disable the reloader by adding use_reloader=False.

Option (2) is probably the more attractive solution since you can still access the debugging console, but you will also lose the automatic reload functionality, so you'll need to restart the server manually before your changes are applied. To do it, change your app.run line to this:

app.run(host="0.0.0.0", port=15600, debug=True, use_reloader=False)
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Thank you so much for this answer, this perfectly solve my confusion. I did not realize the problem is caused by flask at the beginning. In the future, I will post more code for comprehensive information.
@ShiChang Glad to help. Don't forget to accept an answer that solves your problem and upvote those that are helpful :)
1

First I would make sure, that in your scripts, you have the script you want to run inside a callable function. Then make sure you import the function at the start like from fileA import A.

from multiprocessing import Process
import time
# from fileA import A
# from fileB import B


def A():
    print("Running A")
    time.sleep(2)
    print("Ending A")


def B():
    print("Running B")
    time.sleep(3)
    print("Ending B")

if __name__ == '__main__':
    jobs=[]
    jobs.append(Process(target=A))
    jobs.append(Process(target=B))

    for job in jobs:
        job.start()

    for job in jobs:
        job.join()

Output:

Running A
Ending A
Running B
Ending B
[Finished in 3.2s]
Improve this answer

1 Comment

I have also tried this solution. The result is similar to your output: A starts first, after A finished, B starts. However, A contains a packet sniffer so it does not end unless interrupt. I hope them to run in parallel, which is A start and then B starts. What can i do for that?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.