Community detection in complex networks faces persistent challenges in modeling hierarchical organization, higher-order structures, and directional interactions. Deep Nonnegative Matrix Factorization (Deep NMF) models effectively reveal network hierarchical structures, supporting community detection. However, they often focus on first-order connections, overlooking higher-order patterns like network motifs, recurring subgraphs that capture complex interactions. By bridging the concept of motifs with deep representation learning, we establish new possibilities for analyzing modern network paradigms. We propose Deep Motif- Regularized Asymmetric NMF (DMRA-NMF), a deep model that synergizes multi-scale factorization with motif-aware regularization to address these limitations. Our method constructs hybrid similarity matrices integrating edge connectivity and statistically significant motifs, while employing asymmetric factorization to preserve directional network flows. A novel diagonal dominance constraint sharpens community boundaries by suppressing inter-cluster noise without sacrificing intra-community cohesion. The unified optimization strategy balances local neighborhood preservation with global structural coherence, offering interpretable community structures while maintaining computational efficiency. Extensive experiments on biological, social, and technological networks demonstrate the framework’s superiority in detecting nested communities and resolving boundary ambiguities compared to eleven state-of-the-art methods.
