5M is a lot for pure temp shift. Most likely there are multiple factors. Temp could be part of the equation, so could light, but also your eyes will vary for example depending on how tired you are.
Are you ranging on the faceplates or something else? Are those things consistent in how much detail they have?
I've found the harder you try to rangefind precisely, the easier it is to misrange. I think because you subconsciously squint slightly, tweaking the focus of your eyes. I try to make a point of relaxing my eyes, gazing at a distant object for a second or two before looking through the scope.