Wow- that's a dead deer man, nice!

Just thinking of all the variables involved it's a wonder the slug hits the paper sometimes. There's human, mechanical, natural, that's a lot to deal with that can affect the flight path!

I can only speak from my own experience, but for me, its wind.

Think about a bullet that is traveling say 300 yards to its target.
The amount of time that wind is there to push that bullet off course is fairly significant.
We might say well gee, that bullet is doing around 3,000 FPS, so to travel say not even about 900 feet, its barely exposed for any time, not even 1/3 of a second, but its still capable of drifting a bullet a long ways.

The OCW method has worked well for me. I take the midpoint charge weight between the min and max listed, looking at three or more manuals. Then I step up from the min to the midpoint load in .5 grain increments, one round each, as proof loads. I divide the "top half" of the load data (from the midpoint to the Max) into 5 steps, loading four rounds each, as OCW test loads.
As an example, for my .204 Ruger the OCW loads increase in .2 or .3 grain increments - depending on the powder used. For my .300 WBY the increments run .4 to .6 grains, or even a bit more with some powders.
MAKE SURE to carefully examine each charge weight after firing for signs of excess pressure.