In this study, the behavior of reinforced concrete deep beams containing natural coarse aggregates (NCA), recycled concrete aggregates (RCA), and recycled asphalt pavement aggregates (RAP) was numerically investigated. The required parameters for use in the nonlinear finite element analysis were discussed in detail. To examine the accuracy of the nonlinear analysis, the numerical analysis results for deep beams containing NCA, RCA, and RAP were validated with experimental results. The most important parameters that could affect the behavior of concrete deep beams were proposed. The numerical analysis results for 63 concrete deep beam specimens indicated that the deep beams containing NCA and RCA exhibited nearly similar behavior for any a/h and L/h ratios, whereas RAP deep beams behaved differently from the two corresponding specimens. The deep beams made with RCA and NCA had brittle behavior for any value of the a/h and L/h ratios, while those containing RAP showed more ductility. The energy absorption was far greater for RAP deep beams compared with the NCA and RCA specimens. The absorbed energy in NCA and RCA deep beams was nearly the same. The size effect phenomenon was almost similar in the deep beams made with NCA and RCA, whereas in the RAP deep beams, the intensity of this effect was less severe.