Attack speed only benefits pets in that your weapon dps goes up, and that's where their damage comes from, they don't actually gain attack speed. I would personally reroll the attack speed, crit damage is much more useful for pets. You would technically summon less fetishes from the passive, but as your pet gear gets better and better you'll start turning away from attack speed because you'll rely more and more on casting fetish army, which is counter productive to the passive since everytime you cast fetish army it kills your existing fetishes.
1