Had mine milled off but not for weight issues , in fact as the frame / shroud is aluminum I don't think that there would be much of a saving. Did mine so I could fit a UK Neil air stripper to it. Then just got some carbon wrap from the bay to cover the now bare barrel.
Some people did report on POI shift and IIRC there were 2 school's of thought about thie remedy for this ....
1). Remove the 'O' ring from the end of the barrel that keeps the barrel central to the shroud.
2). Mill the shroud completely.
Both options allow the barrel to free float and I suppose milling the shroud removes any chance of the barrel touching the internal surface of the shroud if just the 'O' ring is removed.