In this paper, optimal tuning the parameters of a power system stabilizer (PSS) controller for the power system transient stability enhancement is introduced. The design problem of the proposed PSS is converted to an optimization problem with the time-domain based objective function which is solved by using particle swarm optimization (PSO) technique with a robust ability in order to find the most promising results. The dynamic performance PSS controller is evaluated on the basis of a multi-machine power system exposed to the diverse disturbances by comparison with the genetic algorithm-based damping controller. By virtue of the nonlinear time-domain simulation and some performance indices studies, the performance of the proposed PSS controller is tested and observed. The results show that the tuned PSO based PSS damping controller by the proposed fitness function has an excellent capability in damping power system low frequency oscillations, as well as it significantly improves the dynamic stability of the power systems. In addition, the results reveal that the performance of the designed controller is better than the genetic algorithm based stabilizer.