The time step dt need to be smaller enough (which is depend on velocity V and diameter of the ball R).

V*dt<< R (?dt<0.1*R/V should be fine).

If V*dt ? R , the approximation used in the calculation (finding the time collision really occurred) is not valid.

So you should adjust time step (or set maximum value for V) to satisfy the above condition.