Without knowing the details of what you are machining, the software you are using to produce toolpaths, machining strategy/toolpath form, it is hard to give a meaningful response.
I’ll make a guess that you are plowing slots from the description (the tool cutting a channel, machining both sides and the bottom at the same time). If this is the case, I would expect maybe 0.02 to 0.05mm variation due to the tool flex, tool runout (With these small tools, the runout from the spindle and collet, and any lack of perfect cleanliness on the tool shank, collet bore, and spindle bore are likely to dominate. If the tool is remounted, the runout will likely be different), and the mechanical limits of the machine. Slotting operations are really, really hard on the tool, and can play heavy with tool flex. Don’t EXER count on a precise size slotting with a small tool.
At this size tool, it is unlikely, in my opinion, that there is significant flex in the machine and the anti-backlash setup is pretty tight, on my machine.
If you are not slotting, on tools this small, I would still put runout as the most likely issue, and add that tool flex can be compensated for in a finish pass.
Even on a $US200K precision machining center, with so much rigidity you can’t measure the deflection, and $US2500 bearings that run to within 0.001mm, running tools this small is an art. It is pretty much standard to run the toolpath on a test piece, shoot small then check measurements and rerun with the tool offset adjusted to compensate, and a thousand other techniques to get the final part in tolerance.