Human action recognition is applicable in different domains. Previously proposed methods cannot appropriately consider the sequence of sub-actions. Herein, we propose a semantical action model based on the sequence of sub-actions. A technique is used to segment actions on the time axis based on body movements via an energy diagram. After dividing actions into sub-actions, a novel bi-level deep structure is used to extract their features. Then, the sequence of sub-action features is modeled by a deep network to create the action model. As extracted sub-actions have fewer variations in execution manner, their representation is more stable, and modeling their sequence would be an efficient model. Experimental results on UCF-YouTube, UCF-Sport, and Human Motion DataBase (HMDB) datasets indicate the sustainable performance of this method. Overall, the accuracy of the proposed method is 0.972 on average, while the value for the second-best method is 0.925.
