Is it chatter or just lines left from step over with a round nose cutter? Hard to tell from the picture what scale the piece is, but could you use a 1/4" ball nose cutter and small step over, like say 10%? What % step over are you using with the 1/8" cutter?
Here’s a pretty good article:
http://www.cnccookbook.com/CCCNCMillFeedsSpeedsStepover.htm
If that’s not the case and it really is the machine chattering I’d check the usual suspects starting with V-wheels, belt tightness, pulls set screws.
Dan