I do a reasonable amount of drilling with the VFD spindle in my Shapeoko, I run carbide drills, not HSS or other softer materials to keep a sharp cutting face, I only use steel drills in woods like MDF and they blunt quickly in those abrasive materials.
Seems like your chipload is very small and your drill is doing more rubbing and burning than cutting in the hole which will wear it very quickly, especially in an abrasive material such as HPL.
Take a look at this thread
I run a feed per revolution of at least 0.04mm in Aluminium, 0.1mm in wood and more in plastics based on both recommendations and experience. The carbide drills I buy are not expensive
I’m sure there are similar suppliers where you are.
Check out the recommended feeds in FR-4 from Precisebits here too
For a 4mm drill they are suggesting 20kRPM and 60 in/min (1,524mm / min in proper units) feed rate to keep the feed per rev up at 0.08mm/rev
It’s also worth bearing in mind that each drill and material combo has an optimum range of surface speed which you should stay within, this gives a working RPM range for a given diameter of bit, from there you work out what feed rate you need to keep chips coming out and not burnt up dust.
I run between 2k and 8k on the VFD spindle for drills in the 2-8mm range, I run at the bottom of the surface speed range so that I can keep the feed rate down so machine rigidity doesn’t become the constraint. If your router only goes down to 10kRPM then that’s going to set the min feedrate, then the rigidity and torque set the max drill diameter you can usefully run in any given material.
So, I suggest trying some carbide drills and jack up the feedrate.
HTH