This paper introduces an intermediate virtual representation, called ideal fisheye image, which obeys the ideal simple projection without camera distortion. By using a look-up-table from the ideal fisheye image to the input fisheye image with distortion, a view-direction-free perspective display can be generated fast in comparison with the method of solving a set of nonlinear equations of camera distortion parameters.