I have further read that found something like what unReality said.
Armor Damage Reduction = A%
TotalArmor / ((50*(Creature Level)) + TotalArmor)
Resist Damage Reduction = B%
TotalResist / ((5*(Creature Level)) + TotalResist)
Total Damage Reduction = 1-(1-A%)*(1-B%)
Of course increasing both will definitely be better, but when you are put to a choice to increase ARMOR or AR, then you might want to whip out the calculator and see which is better for your total DR.
The above definitely helps me, hope it will help others.
Quote from unRealityEHP is a combinatino of your total life (HP) and total damage reduction (DR).
Total DR is based on "DR from armor", "DR from resists", and "Other DR". Other DR are stats like "reduce damage from melee attacks by %". So, to answer your question, you want to increase BOTH armor and AR.
10 armor = 1 AR if you have armor and AR at a 10:1 ratio. If you have 5000 armor and 800 AR, adding in 100 AR will provide more total DR than adding 1000 armor. If you have 2000 armor and 300 AR, adding 1000 armor will provide more total DR than adding 100 AR.