Deflection is your issue. When your endmill is cutting material there are forces acting on the bit from multiple directions… backwards from the direction of travel, but also sideways, due to the torque of the bit against the material. Corners are a particular problem because the torque is multiplied as the bit is hitting material on a lot more of the bit at once. One technique I’ve used successfully to combat that deflection issue in the corner is to make my inside corner fillets a little bigger than my bit, and draw a circle in that corner with the same center as the corner, but the size of the endmill. If I’m doing a pocket, for example, I cut a relief pocket in the corners, plus I cut a rough profile around the inside of the pocket. Only then do I do the pocket toolpath.
For example, I want to make a 1/4" deep pocket with a 1/4" Endmill. Here’s my plan:
0.3" corners, 0.25" concentric-to-the-corner circles.
There’s almost no deflection when plunging straight down.
Next, cut the rough outline to a depth of 0.2" leaving .05" side margin
Then I cut the pocket to a depth of 0.25" with 0.1" plunges plus a final 0.05" pass.
All of these cuts can be run faster than doing a pocket alone, because you’re able to control the feed/depth for each independent toolpath. If you want a perfect pocket in a single toolpath, you’re going to have to run it slow and shallow in order to reduce deflection along the edges and the corners, but can’t speed things up for the main part of the pocket where you’re just hogging material away without concern about deflection. Inside corners, when doing a cutout, can be done the same way.