1

I am using the keyboard and mouse modules to record user interaction with our software so that we can have some high level GUI tests. Currently I am trying to store the recorded events in a text file and later play this recording again.

However when I load the recorded events from said file I only see played mouse events and no keyboard events.

One cause for this problem may be the implementation of KeyboardEvents. KeyboardEvents does not contain a correct implementation of __repr__. This prevents us from calling print(keyboard_events, file=f) and reading the lines with eval(line). (This works with mouse and ButtonEvent and MoveEvent) So we have decided to work with the json format of KeyboardEvents. Basically what we're doing is we retrieve the json format of each KeyboardEvent and write the json in the file. Then we load the json file and parse the json as KeyboardEvents.

Currently we're storing both mouse & keyboard input in a single file. However, since mouse supports a correct implementation of __repr__ we can directly print and the mouse events and use eval() on it to retrieve the stored events.

This is the file used for recording and playing:

import threading
import mouse
import keyboard

from mouse import ButtonEvent
from mouse import MoveEvent
from mouse import WheelEvent

from keyboard import KeyboardEvent

import time

import json
import sys

def record(file='record.txt'):
    f = open(file, 'w+')
    mouse_events = []
    keyboard_events = []
    keyboard.start_recording()
    starttime = time.time()
    mouse.hook(mouse_events.append)
    keyboard.wait('esc')
    keyboard_events = keyboard.stop_recording()
    mouse.unhook(mouse_events.append)
    #first line = start of recording
    #mouse events = second line
    #keyboard events = every remaining line = 1 event
    print(starttime, file=f)
    print(mouse_events, file=f)
    for kevent in range(0, len(keyboard_events)):
        print(keyboard_events[kevent].to_json(), file = f)
    f.close()

def play(file, speed = 0.5):
    f = open(file, 'r')
    #per definition the first line is mouse events and the rest is keyboard events
    lines = f.readlines()
    f.close()
    mouse_events = eval(lines[1])
    keyboard_events = []
    for index in range(2,len(lines)):
        keyboard_events.append(keyboard.KeyboardEvent(**json.loads(lines[index])))

    starttime = float(lines[0])
    keyboard_time_interval = keyboard_events[0].time - starttime
    keyboard_time_interval /= speed
    mouse_time_interval = mouse_events[0].time - starttime
    mouse_time_interval /= speed
    print(keyboard_time_interval)
    print(mouse_time_interval)
    #Keyboard threadings:
    k_thread = threading.Thread(target = lambda : time.sleep(keyboard_time_interval) == keyboard.play(keyboard_events, speed_factor=speed) )
    #Mouse threadings:
    m_thread = threading.Thread(target = lambda : time.sleep(mouse_time_interval) == mouse.play(mouse_events, speed_factor=speed))
    #start threads
    m_thread.start()
    k_thread.start()
    #waiting for both threadings to be completed
    k_thread.join() 
    m_thread.join()


if __name__ == '__main__':
    if len(sys.argv) > 2 and sys.argv[1] == 'play':
        play(sys.argv[2])
    elif len(sys.argv) >= 2 and sys.argv[1] == 'record':
        if(len(sys.argv)) == 3:
            record(sys.argv[2])
        else:
            record()
    else:
        print("missing either 'play' or 'record' or filename")

I expect the same behavior with this code like when it is run in a single function (see edit in https://stackoverflow.com/a/57670484/7345513).

Meaning: I expect the playback in the threads to be synced and the keys to be pressed. What i actually get is that the mouse events are played back as desired but no KeyboardEvents are being processed. When I use the function from the linked SO it works.

Can someone please point me to the right direction?

8
  • You cannot rely on repr(arbitrary_object) to provide a representation that can be reversed with eval(). Therefore it's not an incorrect implementation as long as it returns a string. Third parties can return anything they want as __repr__(), so you can't even rely on mouse to always give you what you want in the future. repr() is a debugging tool, not fit for business logic. It would be better to rely on more robust serialization methods, such as JSON, XML, YAML, pickle or whatever your prefer. Commented Aug 30, 2019 at 10:03
  • keyboard already implements KeyboardEvent.to_json() as far as I can tell: github.com/boppreh/… So this would likely be the easiest path forward if you want robustness. Commented Aug 30, 2019 at 10:08
  • 1
    I understand. But I am using the to_json() functionality already. My problem is that when I load the json string back to python the KeyboardEvents are not called. Maybe I am leading/writing them incorrectly? Everything I have is in the pasted source code... Commented Aug 30, 2019 at 10:14
  • Ok I have retried my code and basically tried to debug it within a single python session (usually this code will be started by calling python record.py record) and now both input is loaded. However the timings of keyboard seem to be off. I think this is due to it starting right off with its first input. This can be worked around... And I see now that not all input is forwarded to other applications. Which may be a bigger issue (ctrl+N i.e.) Commented Aug 30, 2019 at 10:34
  • 1
    The reason why the timing is not the way you expect it (i.e. the keyboard output starts too early) is that record() stores absolute timestamps (unix timestamps). Therefore, the only way for play() to play the events back is by playing the first event immediately, which is not what you want. To fix that, you will have to also keep a timestamp of when you started recording and then when you want to play the events back calculate first_interval = keyboard_events[0].time - timestamp_when_recording_started and use the result to sleep(first_interval) before calling play(). Commented Aug 30, 2019 at 10:47

1 Answer 1

0

I forgot to answer my own question but here it goes:

Thanks to the guidance from blubberdiblub I have tracked the start time for the tracking and can then add an offset to when the first input occurred. This allows playbacks to be timed somewhat accurately.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.