Sideways plunge during cut....very frustrating

Is there any mechanical interference?

Are you cutting a slot as narrow as the tool?

Where possible avoid slotting and add geometry and cut as a pocket

and/or

and consider leaving a roughing clearance and taking a finishing pass.