Difference in bits

You have two conflicting issues:

  • cut went too deep — this argues for less acute endmill
  • finer detail lost — this argues for a more acute one.

I would suggest breaking the file up and using different endmills on different sections/areas, and possibly adding geometry so as to depth limit things. Please see:

and