How to interact dynamically with spawned processes using subprocess module in Python

Question

To give some context, I have to grade my students' C codes. This is the folder structure:

    Root
      |_____ID#1
        |____task1.c
        |____task2.c
        |____task3.c
      |_____ID#2
        |____task1.c
        |____task2.c
        |____task3.c
        ...

It is to be noted that not every student answered all the questions, nor is the naming convention uniform. So, I have no way of knowing which file contains which task. All of the tasks require some sort of user input, but since I don't know which file contains which task, I cannot hardcode the input by figuring out the file name.

I want to be able to save:

a. the inputs to the processes, which I must be able to give to them in an interactive way (consider the following screenshot for what I mean by interactivity, after "Code compiles"),

b. their output after processing the input,

c. their errors (if any)

to a json or even a text file for now.

This is what I have so far:

import os
import subprocess
from zipfile import ZipFile

student_folders_root = "/home/mahieyin/Documents/Code/Python/CSE115/Mid #1-20200403T150847Z-001/Mid #1/"

for idx, (directory, _, zipfiles) in enumerate(os.walk(student_folders_root)):
    result = ""
    student_id = directory.rsplit(os.sep, 1)[-1]

    if idx == 0:
        continue
    elif len(zipfiles) == 0:        
        result += f"{student_id} did not attend the mid term exam."
        print(result)
    else:
        zipfiles = list(filter(lambda x: x.endswith(".zip"), zipfiles))

        path_to_zip = os.path.join(directory, *zipfiles)

        if path_to_zip.endswith('.zip'):
            with ZipFile(path_to_zip, 'r') as zip_object:
                zip_object.extractall(directory)

        print(f"\033[1;33;40mProcessing student: {student_id}...")

        for _, _, files in os.walk(directory):
            files = list(filter(lambda x: x.endswith('.c'), files))
            files = list(map(lambda x: os.path.join(directory, x), files))

            executable_name = "a.out"
            exec_path = os.path.join(directory, executable_name)

            for c_file in files:
                try:                
                    print(f"\n\n\033[1;35;40mProcessing file: {c_file}", end="\n\n")
                    compile_command = f"gcc \"{c_file}\" -o \"{exec_path}\""
                    run_command = f"\"{exec_path}\""
                    # command = f"gcc \"{c_file}\" -o \"{exec_path}\""

                    # subprocess.call(command, shell=True)
                    compile_result = subprocess.check_output(compile_command, shell=True)
                    compile_result = compile_result.decode('utf-8').strip()

                    if len(compile_result) == 0:
                        print("Code compiles.")                        
                        run_process = subprocess.run(run_command, shell=True)
                    else:
                        print("Compilation error.")
                        print("\033[1;31;40m", compile_result)
                except subprocess.CalledProcessError:
                    pass

This works as expected. I can see the compilation errors if the source file fails to compile. I can provide input to the spawned processes and see the output. However I am unable to wrap my head around the way in which I can obtain the outputs while preserving the interactability.

Basically, whenever I try to set the stdout and stderr options in either subprocess.Popen() or subprocess.call(), I lose the interactability with the spawned process. The prompts of the spawned processes don't show (e.g. "Please enter the value: ") until after I have provided the input, which isn't really elegant to look at.

Thanks in advance.

Roland Smith · Accepted Answer · 2020-04-03 21:45:58Z

1

Have a look at pexpect:

Pexpect is a pure Python module for spawning child applications; controlling them; and responding to expected patterns in their output.

answered Apr 3, 2020 at 21:45

Roland Smith

43.8k3 gold badges69 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mahieyin-rahmun · Accepted Answer · 2020-04-04 16:22:00Z

Following Roland Smith's advice, I ended up using a solution that is not elegant but saves my time. I replaced the second subprocess call with pexpect.

import pexpect
import subprocess
import os
from zipfile import ZipFile

student_folders_root = "/home/mahieyin/Documents/Code/Python/CSE115/Mid #1-20200403T150847Z-001/Mid #1/"

for idx, (directory, _, zipfiles) in enumerate(os.walk(student_folders_root)):
    result = ""
    student_id = directory.rsplit(os.sep, 1)[-1]

    if (idx == 0):
        continue
    elif len(zipfiles) == 0:        
        result += f"{student_id} did not attend the mid term exam."
        print(result)
    else:
        zipfiles = list(filter(lambda x: x.endswith(".zip"), zipfiles))

        path_to_zip = os.path.join(directory, *zipfiles)

        if (path_to_zip.endswith('.zip')):
            with ZipFile(path_to_zip, 'r') as zip_object:
                zip_object.extractall(directory)

        print(f"Processing student: {student_id}...")

        for _, _, files in os.walk(directory):
            files = list(filter(lambda x: x.endswith('.c'), files))
            files = list(map(lambda x: os.path.join(directory, x), files))

            executable_name = "a.out"
            exec_path = os.path.join(directory, executable_name)

            for c_file in files:
                try:
                    print(f"\n\nProcessing file: {c_file}", end="\n\n")
                    compile_command = f"gcc \"{c_file}\" -o \"{exec_path}\""
                    run_command = f"\"{exec_path}\""
                    # command = f"gcc \"{c_file}\" -o \"{exec_path}\""

                    # subprocess.call(command, shell=True)
                    compile_result = subprocess.check_output(compile_command, shell=True)
                    compile_result = compile_result.decode('utf-8').strip()

                    if len(compile_result) == 0:
                        print("Code compiles.")
                        # the change
                        c_process = pexpect.spawn(run_command)
                        c_process.interact()
                    else:
                        print("Compilation error.")
                        print(compile_result)
                except subprocess.CalledProcessError:
                    pass

And for redirecting the output to a log file, I use:

python GradeStudents.py 2>&1 | tee logfile.txt

Collectives™ on Stack Overflow

How to interact dynamically with spawned processes using subprocess module in Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related