I am using the keyboard and mouse modules to record user interaction with our software so that we can have some high level GUI tests. Currently I am trying to store the recorded events in a text file and later play this recording again.
However when I load the recorded events from said file I only see played mouse events and no keyboard events.
One cause for this problem may be the implementation of KeyboardEvents.
KeyboardEvents does not contain a correct implementation of __repr__. This prevents us from calling print(keyboard_events, file=f) and reading the lines with eval(line). (This works with mouse and ButtonEvent and MoveEvent)
So we have decided to work with the json format of KeyboardEvents. Basically what we're doing is we retrieve the json format of each KeyboardEvent and write the json in the file. Then we load the json file and parse the json as KeyboardEvents.
Currently we're storing both mouse & keyboard input in a single file. However, since mouse supports a correct implementation of __repr__ we can directly print and the mouse events and use eval() on it to retrieve the stored events.
This is the file used for recording and playing:
import threading
import mouse
import keyboard
from mouse import ButtonEvent
from mouse import MoveEvent
from mouse import WheelEvent
from keyboard import KeyboardEvent
import time
import json
import sys
def record(file='record.txt'):
f = open(file, 'w+')
mouse_events = []
keyboard_events = []
keyboard.start_recording()
starttime = time.time()
mouse.hook(mouse_events.append)
keyboard.wait('esc')
keyboard_events = keyboard.stop_recording()
mouse.unhook(mouse_events.append)
#first line = start of recording
#mouse events = second line
#keyboard events = every remaining line = 1 event
print(starttime, file=f)
print(mouse_events, file=f)
for kevent in range(0, len(keyboard_events)):
print(keyboard_events[kevent].to_json(), file = f)
f.close()
def play(file, speed = 0.5):
f = open(file, 'r')
#per definition the first line is mouse events and the rest is keyboard events
lines = f.readlines()
f.close()
mouse_events = eval(lines[1])
keyboard_events = []
for index in range(2,len(lines)):
keyboard_events.append(keyboard.KeyboardEvent(**json.loads(lines[index])))
starttime = float(lines[0])
keyboard_time_interval = keyboard_events[0].time - starttime
keyboard_time_interval /= speed
mouse_time_interval = mouse_events[0].time - starttime
mouse_time_interval /= speed
print(keyboard_time_interval)
print(mouse_time_interval)
#Keyboard threadings:
k_thread = threading.Thread(target = lambda : time.sleep(keyboard_time_interval) == keyboard.play(keyboard_events, speed_factor=speed) )
#Mouse threadings:
m_thread = threading.Thread(target = lambda : time.sleep(mouse_time_interval) == mouse.play(mouse_events, speed_factor=speed))
#start threads
m_thread.start()
k_thread.start()
#waiting for both threadings to be completed
k_thread.join()
m_thread.join()
if __name__ == '__main__':
if len(sys.argv) > 2 and sys.argv[1] == 'play':
play(sys.argv[2])
elif len(sys.argv) >= 2 and sys.argv[1] == 'record':
if(len(sys.argv)) == 3:
record(sys.argv[2])
else:
record()
else:
print("missing either 'play' or 'record' or filename")
I expect the same behavior with this code like when it is run in a single function (see edit in https://stackoverflow.com/a/57670484/7345513).
Meaning: I expect the playback in the threads to be synced and the keys to be pressed. What i actually get is that the mouse events are played back as desired but no KeyboardEvents are being processed. When I use the function from the linked SO it works.
Can someone please point me to the right direction?
repr(arbitrary_object)to provide a representation that can be reversed witheval(). Therefore it's not an incorrect implementation as long as it returns a string. Third parties can return anything they want as__repr__(), so you can't even rely onmouseto always give you what you want in the future.repr()is a debugging tool, not fit for business logic. It would be better to rely on more robust serialization methods, such as JSON, XML, YAML, pickle or whatever your prefer.keyboardalready implementsKeyboardEvent.to_json()as far as I can tell: github.com/boppreh/… So this would likely be the easiest path forward if you want robustness.python record.py record) and now both input is loaded. However the timings ofkeyboardseem to be off. I think this is due to it starting right off with its first input. This can be worked around... And I see now that not all input is forwarded to other applications. Which may be a bigger issue (ctrl+N i.e.)record()stores absolute timestamps (unix timestamps). Therefore, the only way forplay()to play the events back is by playing the first event immediately, which is not what you want. To fix that, you will have to also keep a timestamp of when you started recording and then when you want to play the events back calculatefirst_interval = keyboard_events[0].time - timestamp_when_recording_startedand use the result tosleep(first_interval)before callingplay().