It's advice aimed at the young and pregnant mums. Sensible really if you consider that lead goes for the nervous system and it's exposure is cumulative.
Don't go nuts but take care with lead shot. I wouldn't want to crunch another pellet that I missed when preparing the meat, but I wouldn't want to swallow it or bits that dissolved in the cooking process either.
Just take care and minimise the risk through some sensible precautions.
It makes me wince, if I see shooters with pellets between their lips before putting them in their barrels. It all adds up.