there's a natural pause between the lungs exhaling and inhaling... where as inhaling immediately triggers an exhale. Emptier lungs use less muscle, rather than holding your breath. Diaphragmatic breathing is a good technique.
But if you're wandering off then your position sounds like it could do with work. A good position you can take a breath in and although the POI will wander off it will come back each cycle.
BFTA/NSRA County Coach
CSFTA Chairman/BFTA Rep