Have you measured the actual diameter of that first bit you used ? (I can’t wrap my mind around how a hole can end up smaller than the bit used to drill it, but then again 0.22" actual diameter for a 0.25" endmill would be something new…)
about that wiggle, did you check it by holding the endmill tip itself and wiggling it around ? i.e. what @fenrus implied to check for excessive play in the router bearings.
It seems like you have a Dewalt router, and while I never owned one and…statistics, there have been a couple of reports of people getting bad mechanical precision from theirs (@fenrus mentioned one, yesterday @Microwave_Monkey had a similar testimony, and there are more on the forum)
As a double check you could maybe use a pointy Vbit, and engrave small (and very shallow) circles on the surface, check if they come out round or oval.
Side note: I’m surprised by your calibration values of around 39.7 steps per mm. Back when I believed in machine calibration I remember getting much closer to 40 (typically like 39.94 or 40.03 or values like that). I’m not saying this is not normal, I’m just surprised. Have you checked over longer distances than 4"? Sorry for the trivial question and forgive me if you already addressed that, but how tight are your belts ? (as in, post a pic of you pulling the middle of your X belt with the gantry pushed to one side)