Yep, your analysis is correct, losing steps upon retract which then translates to erroneous depth because the Z axis reference is lost.
You should reduce the retract height to the minimal value that will still clear the top of the material, try and use a shorter endmill if possible, and you may also want to temporarily raise the router in its mount for this job, to gain some Z margin.