Sounds like what you’re looking for is a “Slip Fit”. Usually a hole 0.0005" - 0.002" larger than the dowel.
Best bet is to machine the hole exactly 0.500" & measure it, then adjust from there.
You might just end up with a slip fit the first time due to spindle/cutter runout. Or if you’re using CC, the fact that it likes to conventional cut.
Yes, humidity is definitely a factor that I foolishly didn’t think of. I was practicing with MDF, but it’s a sponge. I’ll test out on some scrap. Worst case, I can enlarge the holes with my drill press if they almost work.