You have to split your awareness between breathing and kicking. It's hard, I know. But this is where most people develop bad habits - when they are distracted by breathing.
This article has become more about kicking technique than shoulders. But if you don't kick well, I pretty much can guarantee that you are using your arms/hands in inefficient ways to compensate, or are swimming much slower than you could be, with the biggest drag happening when you try to rotate to breathe.
And if you are not rotating well, you may be swimming with the chicken recovery or some other such evilness.
Words of wisdom from Xiao Jin, kung fu master in the ancient art of You Yong:
"Kick well to rotate. Rotate well to breathe and slip past the water dragon. And slip past the water dragon to move your arms with shoulders that will last a thousand years, my young minnow."