1

I am developing an application in PyQt (primarily Windows but with the future intention to go multiplatform) which focuses on processing certain numerical data, so the core is basically built around numpy arrays and matplotlib charts. However I wanted to make parts of the system Python scriptable, allowing the users to tweak the data processing a bit. The script code they write is stored in a string and it just needs to pass in and out some parameters of arbirtary types. And add some error handling as well... This can be achieved in a very simple way in Python, like in the code fragment here:

global_dict = {}  # values could go in here
local_dict = {}  # values could go in here
try:
    exec(script_code, global_dict, local_dict)
    value = local_dict[variable_name] # value out
#etc...

The problem is that the user scripts are out of my control, the user can write whatever they like and they can take significant time to execute which could freeze GUI. So I needed to go multithreading. I already have a working solution. I use a class which inherits from QtCore.QRunnable runs and the code above asynchronously. It works fine and correctly notifies the GUI thread when the script is finished, passes the results and destroys the thread. But still I have three serious problems:

1) a crucial problem: how to abort the script execution? Although the asynchronous script does not freeze GUI, it can be very long or even infinite. And the user certainly must have the option to abort the execution without having to terminate the whole application. This does not seem to be possible with QRunnable. So I thinked about switching from QRunnable to QThread which has terminate method. However it is strictly not recommended using this method. It is very questionable how would the application recover from the termination of one of its threads, for example what would happen to reference counting with the objects passed into the script. Is this problem even solvable? I do not know... maybe there is a miraculous solution that I do not see. Could Python multiprocessing module help?

2) an important problem: as users can pass data in and out from their scripts, which is done by reference, they can also change the data. In Python multithreading GIL ensures that no two threads write to the same data at the same time but it might happen that the data is already being processed or displayed also in another part of the application. So I would need to have the inputs into the script read only, throwing exception when user attempts to write in it. Is it even possible?

3) a nice-to-have-solved-but-can-live-with-it problem: Sometimes several scripts can be executed at the same time. Due to Python's GIL all threads are executed in one interpreter and one processor core only. This might be a limitation for the performance. As I learned, for benefiting from multiple processor cores I would need to switch from multithreading to multiprocessing. But how to do that with the given scenario? Is it even compatible with PyQt architecture? How to pass values in and out? How to terminate?

Maybe I am asking too much, maybe my wishes are not from this reality. Nevertheless I would be very grateful for any hint or advice or sources to look into.

Improve this question
2
  • Maybe I have found a solution to at least a part of my problems... killable threads: link ... I am going to test it. Commented Dec 3, 2013 at 20:54
  • Why did this get downvoted? Opinion? Commented Nov 10, 2014 at 3:46

2 Answers 2

Reset to default
1

Your problem sounds very similar to one that my research group has solved. Unfortunately its written in PyGTK (a port to Qt will happen hopefully within 6-12 months). Anyway, the program called lyse in the labscript suite does a lot of what you are asking. It is quite complex, but perhaps looking at the source code could be enough to get you started.

In short we have a GUI that runs in one process, arbitrary user code that runs in another process (this process also has other threads and greats it's own windows for plots) and data is passed around by sending pandas (uses numpy) dataframes and sending them over PyZMQ (0MQ) sockets.

Feel free to check out our source code on http://www.labscriptsuite.org which also contains developer contacts if you have any specific question about the project (should it be of any use). There are quite a few parts that do not rely on GTK, and so could be included in a Qt program if you so wished. Most information on the project (and included programs) is contained in the paper we published on the system (if you don't have access to RSI, there is a download link for the paper on the FAQ page)

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thank you very much for the source, I am really grateful. I will check the code you provided. However before I do complete installation of all the dependencies, could you kindly confirm it has the functionality that I am specifically looking for? I mean the ability to abort a script execution running in another thread or process without corrupting the rest of the application or leaving memory leaks...
I wouldn't necessarily worry about a complete installation of the project (though you can if you like). The specific parts of the code to look at are the module subproc_utils and lyse/analysis_subprocess.py. subproc_utils should only need pyzmq and has some examples of how to setup a connection between two processes. You can then just pass in the filepath of a python script you want run to the subprocess and use execfile() to run it. Because everything is in a separate process, the OS takes care of cleaning up memory when the process is killed. So to abort, just kill a subprocess.
So in short, the best bet is a multiprocess architecture, using something like zeromq (0MQ) to communicate. The subprocesses use execfile or exec to run your arbiratry Python code. You can terminate running scipts by killing the subprocess. This has the added benefit of avoiding GIL issues. An example implementation of this is the program lyse in the labscript suite (you can "abort" a script by deleting it from the list of scripts to execute, which just kills the subprocess) :)
0

It seems that I have found a solution to at least the most critical part of my problem - to abort a thread my mian thread: http://tomerfiliba.com/recipes/Thread2/

According to the link, it uses function PyThreadState_SetAsyncExc which raises exception in another thread. It seems very simple and according to my first test it works perfect with Python 3 on Windows.

I will try to find a solution to my other problems. But if not, I can live with this solution for now.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.