Abstract In this paper, a novel method for investigation of unified power flow controller (UPFC) effects on transient stability of a multi machine power system has been introduced. Based on the combination of output results of time domain simulation (TDS) and transient energy function (TEF) analysis, the study of power system transient stability is converted to the study of transient stability of only one machine, the so called critical machine. The effects of UPFC in three basic control modes, namely, in-phase voltage control (IVC), quadrature voltage control (QVC) and shunt compensation control (Sh.C), on the transient stability margin and for various fault clearing times has been studied. Based on acquired results, it can be seen that in-phase and quadrature voltage controls are the most effective on the improvement of transient stability margin. Also, ivc is more robust than other control modes against the increase of fault clearing time. The proposed method determines the critical machine based on the swing curves resulting from TDS, and then provides a stability margin (SM) of the critical machine with trivial computation by use of machine speeds and inertia constants. The method does not need to include a UPFC description equation in the transient energy function. The proposed method was successfully demonstrated on the 7-machine CIGRE test system.