This issue often comes down to how your software, hardware, drivers, and system settings interact.
The mismatch between hardware acceleration and scaling depends on your specific setup, like your GPU, installed drivers, FFmpeg configuration, and even environment variables.
There’s no one-size-fits-all solution, you’ll need to experiment with different settings (e.g. switching between hardware and software decoding) to see what works for your system and your setup.