Operation and control of large‐scale power systems is a complex task. Since the primary purpose of power system stabilizer (PSS) units is damping of local and interarea modes, identification of network electromechanical oscillatory modes for designing PSS is very important. In this research, continuous wavelet transform (CWT) was used to study low‐frequency electromechanical oscillations of power systems. The complex Morlet mother wavelet and guidelines for the selection of center frequency, bandwidth parameters, and scaling factor were used to estimate reliable modal specifications of the power system. After estimation of power system modes, the Prony method was used to identify the transfer function residue. Thus, the parameters of PSS were calculated by the residual method. In order to improve damping and meet the constraints of the proposed PSS parameters, the sequential quadratic programming algorithm was used as an optimization method. Finally, with the help of MATLAB and PST softwares, modes estimation methods and the proposed controller design approach was simulated on the two known test case systems, IEEE two‐area, four‐generator and 68‐bus, 16‐Gen test cases. The simulation results show efficacy and good performance of the proposed PSS design method to improve damping and small signal stability of the systems.
