For reference the rest of the game runs perfectly smooth at any settings I throw at my computer. Is it (just like the asset loading lag) something that is only fixable through using an SSD or super high end graphics card?
I avoid the sewer areas and the spider caves because of their weird frame drops.
I have not heard about the sound bug, so can't confirm that.
But if you ask whether a high-end graphics card or an SSD could solve that issue, it's definitely the latter. Some of my friends with absolute high-end graphic cards have the same problem you describe, and I had this problem as well. The solution was to install D3 on an SSD. Oftentimes an SSD is said to have no (or only a small) effect on games, and this might true for the overall performance or load times for D3 (it was fast enough before, so not really any noticeably improvement). The lag issue, however, can definitely be solved by moving the D3 installation to an SSD.
Not sure about the USB drive recommendation, it would need to be pretty fast and pretty big and therefore quite expensive, and even a USB 3.0 bus does not reach SSD speed and latency (maybe in theory, but not in practice). But would be interesting to see if this has changed since I tried it last time, as SSD upgrades are not always possible or affordable. That being said, an SSD is the best performance boost in the computer industry I've seen in the last 20+ years.