In this paper, we utilize intelligent reflecting surfaces (IRSs) in full-duplex (FD) systems to enhance diversity gain and system performance. The issue of self-interference in FD systems causing an outage floor is addressed. We propose a beamforming (BF) method that mitigates this effect, even when there is no direct channel between the source and the far user in non-orthogonal multiple access (NOMA) systems. We examine two designs, named fixed antenna mode (FAM) and adaptive antenna mode (AAM), for 𝐾 decode-and-forward (DF) FD near users in a Rayleigh fading environment. To improve system performance, we employ joint antenna mode and near user selection techniques in IRS-NOMA systems. We evaluate the performance of the proposed schemes and derive closed-form expressions for outage probability and ergodic capacity, which are validated through simulations. The results demonstrate that the proposed scheme achieves additional spatial diversity gain through the use of the IRS structure and antenna mode selection at the relay nodes. Furthermore, the new BF technique improves power efficiency compared to conventional FD NOMA structures, and the analysis can be extended to 6G Massive MIMO systems using machine learning algorithms, which are applicable for millimeter wave channels.
