I agree with Julien on checkiung the machine and making sure it’s all tight-and-proper
also have you seen How I Make REALLY Round Holes on the Shapeoko ? that sounds like very promising technique for what you want
another thing is that for max precision I would recommend first cutting “short” by say 0.5mm (0.02") and then do a 2nd pass just to shave off that last bit… that reduces forces on this critical part