One reason may be that the homing switches will have some level of inaccuracy themselves. Unless they trigger in exactly the same spot, your saved zero will move by the same amount.
If you are comfortable fiddling with some of the settings on the controller, there is a setting that may help. You can change the speed it looks for the homing switches, a slower speed may result in more consistent triggering.
I’m not in front of my machine right now, but GRBL setting $24 sets the homing feed rate.
Here’s a walk-through for changing GRBL settings. Just use $24 instead of the example setting.