Short answer is no. It doesn't have to be physical.
But if you want the most damage then it should be.
Most people are using:
Ancient Spear - Boulder Toss
Leap - Death from Above
Ground Stomp - Wrenching Smash
Threatening shout - Falter
War Cry - Veteran's Warning/Impunity
Earthquake - Cave in
The reason this is good is because everything is physical. And the base skill Avalanche is physical. So even though you don't have it on your bar it still gets a bonus. This way everything benefits from +physical dmg.
If you switch to cold then you have to sacrifice some things. If you run the same setup and just switch the rune on Earthquake and stack cold damage then your Bouldertoss and Avalanche don't benefit. If you also want to run with the cold Avalanche rune then you have to sacrifice ALOT of damage. Cause then you have to remove one of the skills that procs your weapon or belt, removing alot of "free" damage.
There is sort of better control with cold but not by alot. And not nearly enough to make it worth sacrifcing a big chunk of your damage. You get two pull effects and two more then decent stuns with practically no CD considering Fury dumping.
Hope this explains why most people are going physical. But if you find cold effective, fun or anything else positive then by all means play it like that. Another possibility is fire. Earthquake unruned does fire damage so you could swap Earthquake for Avalanche on your bar and get fire damage from both. But with this scenario you loose utility aswell. And Boulder toss damage takes a loss. But experiment away.
How are you liking the set in general?