The problem you are seeing is almost certainly that the bit you are using is a so-called ‘fishtail’ bit: