For years I was firmly in the frame-bag camp: narrow, quiet and good on rough ground. This season I rode the same bike with panniers for half of it, to see whether I had been wrong.
Panniers win on packing. You open them, you find things, you close them. They also carry heavy water far better than anything strapped to the frame.
Where frame bags still win
On singletrack and anywhere you have to push or lift the bike, frame bags are much easier to live with. The bike stays narrow and balanced, and nothing snags on rocks or brush.
My answer after 3,000 kilometres: frame bags for rough routes, panniers for long road days, and a rear rack left on the bike so I can switch in ten minutes.