In this paper, an online adaptive optimal controller is firstly designed to optimize the performance of an automatic voltage regulator (AVR). Towards this end, an optimal quadratic tracking problem is defined based on the error between the synchronous generator's terminal voltage and its desired value. Then, this optimal control problem is solved using an adaptive dynamic programming (ADP) method, called the policy iteration technique. Using this technique, the optimal performance is achieved without knowledge of the AVR parameters. To eliminate the steady state error between the AVR output and its desired value, a modification is made in the original policy iteration technique base on the integral action control and another adaptive optimal controller is designed. In addition, an approach is proposed to represent a large-scale power system through several Single-Machine Infinite-Bus (SMIB) systems. The equivalent SMIB system then may be linearized through modal analysis to be employed by the proposed control scheme. Simulation results show that the designed adaptive optimal controllers are so effective and can be used in practical power systems.