Using a keypad and the smooth jogging mode in GRBL could get you that manual milling mode (just "jog in the material with the router turned on 
).
The problem would be the ability to manually adjust the jog speed depending on the material/cutter/RPM.
Jog speed is fixed in CM, but using another G-code sender you could add a manual control of jog speed, this is what I did in this project, one of the knobs on that pendant is used as a jog speed selector. The selector only had three positions, but one could come up with a finer way to manually adjust jog speed/feedrate (a rotary dial managed by an arduino maybe).