Delrin machines very nicely, but I suspect the torque limit will get you much above 3mm. I have drilled 3.3mm in acrylic for 4mm taps and it is satisfactory, but generally just mill them, if I can, to avoid the tool change. I wouldn’t try any larger in acrylic. Delrin is less gooey than acrylic, but it still takes a fair bit of torque to drill. Note that the torque required is approximately proportional to the square of the diameter of the drill.
For 5mm and 6.75mm, I mill the hole using a 1/8" endmill, and, if I can’t go deep enough (25mm is about the limit with tools I keep on hand, but, then again, I would be hard pressed to get much more with a drill bit on the Nomad), follow up on the drill press or using a handheld drill using the milled hole as a guide.
That said, I have not tried a drill as large as 5mm on delrin in this machine. I discovered early on that much over 1/8" were impractical on the materials I most work with, and that milling holes generally saves my time anyway, so I stopped exploring the limits for every material. I can say with confidence that it is not practical to drill a 5mm hole in aluminum, nor a 5mm hole of significant depth in acrylic (actual size was 0.166 for a #10-32 machine screw, so actually about 4.2mm. Coolant would have helped, but it is easier to just mill the hole dry at that point).
When you drill, watch the machine carefully until you get your settings, especially with larger bits. You can and likely will stall the motor at some point. If you find you can drill 5mm, that would be very nice, but my suspicion is that the machine will bog.
As for collets getting tight, I haven’t had that problem, but I typically only bring a 1/8 (3.2mm) collet down to 3.0mm. It is so much easier, and more efficient, to just get as much tooling as I can with standard shank sizes, which, for me, on the Nomad, is 1/8" and 3mm, with some 1/4" (6.35mm) shank. I use a different flavour of collet on the lathe.