What I need

I am working on a music player with a sample analyzer. My issue is that I am currently using pydub to get the samples from the song, but as the song gets longer, the more out of sync the two become. I have made the two literally in sync, pydub just so happens to read out of it [I even started them at the same time, the two actually progressively de-synced with audio alone and more so as the song continues playing].

Basically, is there a python-vlc equivalent to pydub.AudioSegment.get_array_of_samples()?

You can look at the python-vlc docs here: https://www.olivieraubert.net/vlc/python-ctypes/doc/

What I've tried

1. python-vlc has sound.get_time() which returns the last updated time in milliseconds. The issue with this command is not that it only updates every half a second or so [I've found a way around that] but that it doesn't return the accurate time. I can start a timer from when it starts playing using time.monotonic(). As time progresses, the get_time() wildly varies from the timer. I've gotten a difference of 195ms, 294ms and 217ms.

2. I tried using an external timer using the threading module. It appears that pydub.AudioSegment[index] is not in sync whatsoever.

3. Scaling from sound.get_time() in py-vlc to len(sound) in pydub. This doesn't work as expected. I cannot tell you why but it is still desynced.

4. Using sound.get_time() with an average offset that increases over time. It appears that pydub.AudioSegment[index] doesn't line up properly.

5. Using aubio. It only reads WAV files, and for a live analyzer, converting to a WAV first would take too long.

Things I found out

• I've looked at how long each song is in milliseconds, len(sound) for pydub and sound.get_length() for python-vlc is different usually by 10 or so seconds.

Things that won't work

• Using pydub's play() command. I don't like it because it's very limiting.
• Using something else than py-vlc to play the audio, VLC has many features that just cannot be replicated.

Previous suggestions

• Use How can I extract audio from video with ffmpeg?
  • This won't work because I am not looking to extract audio from an existing video file, I'm trying to get samples like pydub.AudioSegment.get_array_of_samples()

I'm curious

• Is there a python module for Audacity? I've been looking and couldn't find it. I can also use some other command line tool that can interact with Audacity if possible [I know the built in command line utility doesn't do much at all]