The next generation of wireless networks demands transformative solutions to achieve ultra-reliable, high-capacity, and energy-efficient communications. Conventional systems based solely on relay-assisted transmission or intelligent reflecting surfaces (IRS) face inherent trade-offs in spectral efficiency, coverage, and energy consumption. In this work, we propose a novel cooperative hybrid communication system that integrates IRS with relay-assisted transmission to form a robust, scalable, and energy-aware design for beyond 5G (B5G) networks. Unlike traditional architectures, the proposed system enables mobile users to receive signals from the base station through three cooperative transmission paths: direct relay transmission, IRS reflection, and relay-assisted IRS reflection. This cooperative signal propagation enhances path diversity, improves link robustness, and increases spatial efficiency. Additionally, we introduce a joint optimization algorithm to determine the optimal IRS placement and reflecting element configuration, maximizing system throughput under practical rate constraints. Simulation results under Rayleigh fading conditions demonstrate the effectiveness of the proposed system across various deployment scenarios, including SISO, MISO-OMA, and MISO-NOMA. In the MISO-NOMA setting with 40 dBm transmit power, the system achieves a sum rate of 49.80 bits/s/Hz and an energy efficiency of 16.60 bits/Joule, outperforming benchmark hybrid relay-IRS-aided and relay-dominant cooperative hybrid systems. These findings establish the proposed system as a high-performing and energy-efficient solution for future wireless networks.