You could also argue that wearing target shooting clothing and using target rifles gives the impression to the general public that this is a serious and legitimate target shooting sport.
They're used in the Olympics after all, and the general public are used to seeing such things in the Olympics and associate Olympic shooting jackets with legitimate target sports.
So you could argue that it's actually a very positive thing for the sport as a whole and we should ban people not wearing a target jacket
p.s. I have a canvas Creedmoor jacket - it's not the hardback version, and I don't find that it gives me much if any of an advantage. I bought it because of the rubber pads on the elbow and sides. Since I broke my back and had 3 of my vertebrae bolted together, I find it a bit painful to cock my hip out for extended practice on standers. So I wear a creedmoor so that I can use the rubber pads to take some of the weight of the rifle that I would have taken on my hip if I could.
I don't think this gives me an advantage at all, i think it helps to level the playing field against other shooters who don't have metalwork in their spine. But I'm still at a disadvantage.
My Creedmoor is an NSRA style jacket, not Olympic ISSF so wouldn't be affected by your proposed ban. It's green and after extensive Mori poll research 62.83% of general public say they think it looks cool and makes me look a bit like an Elf. (Elves are cool).