The vibration pattern of a rotor system reflects the mechanical parameter changes in the system. Hence, the use of vibration monitoring is considered as a possible means of detecting the presence and growth of the cracks. In this paper, a pattern search based method for shaft crack detection is proposed and described which formulates the shaft crack detection as an optimization problem by means of finite element method and utilizes pattern search algorithm to search the solution. Using direct search method avoids some of the weaknesses of the traditional gradient based analytical search method, including the difficulty in constructing well-defined mathematical models directly from practical inverse problems. First, a finite element code was developed for analyzing a rotor system with open cracks. To extract the flexibility matrix of an element containing cracks an exact integration scheme was adopted which is more accurate than the conventional methods. Then, the crack detection method was formulated as an inverse problem which can be solved by optimization algorithms. The numerical simulations suggest that good predictions of shaft crack location and depth are possible and the proposed method is feasible.