How much material are you removing on your final pass?
I haven’t experimented a lot, but I found that having a very small amount of material removed helped the surface finish.