This helps a lot! Good call on the plunge rate and I should have known better because I get better results running my regular cuts faster.
So are you plunging almost to the bottom, then letting it cut out the pocket? My holes are slightly bigger than the bit, so my approach was to plunge it down almost to the bottom, let it do its pocket, then one more plunge to the bottom and let it do its pocket (2 passes). The other option would be to plunge just slightly further than the bottom and letting it pocket (one pass).