Qemu/kvm audio quality became much worse after updating to 5.4.0-96-generic

Updated kernel from 5.4.0-52-generic to 5.4.0-96-generic on a 20.04 LTS installation.

After the update, output audio quality of guests running under Qemu/KVM's worsened significantly, with constant crackle and stuttering in the sound.

Happens in multiple different guests, including Gentoo, Ubuntu live-cd and Windows 10.

It is most noticeable with -soundhw hda and PA audio backend. If I use ALSA backend and -soundhw ac97, the sound quality is a lot better (still worse than 5.4.0-52-generic but not nearly as noticeable).

If I boot the old kernel (5.4.0-52-generic) instead, the audio quality issues are gone.

The low latency kernel ( 5.4.0-96-lowlatency) also works better than the 5.4.0-96-generic one, but still has crackle and stuttering.

Qemu packages were also updated from 4.2-3ubuntu6.7 to 4.2-3ubuntu6.17, however since the issues don't happen when running the old 5.4.0-52-generic kernel, it might not be related.

Any suggestions on how should I go about resolving this?