I pretty much feel the same. I think they share the loot table with legendaries, so I even feel they take away a chance of a good item dropping. Town in party games gets so trashed with them after a few hours as well.
I can not remember the last time (if ever) I thought "that 60 extra all res for 5 seconds really saved me" or "I wish I had potion x to use the effect now". Only on my DH I ever wished for the one to break walls, only to remember I can not use it when full health and will probably die anyway when taking damage
On the other hand, I often end up leaving a belt/ring/amu behind because my inventory is full.
Thanks for reminding me I can just drop it and have that extra slot
PS: If you have multiple potions in your inventory, you can right click on the bar and pick the one you want to use. While, in theory, this might enable some high end plays, the bonus is too small to matter. However, this system would prevent you putting a potion directly on your bar, or as a gear slot.