In underlay cellular networks supporting Device-to-Device (D2D) communication, efficient r esource allocation remains a key challenge due to spectrum sharing and crosstier interference. This work proposes a Deep Reinforcement Learning (DRL)-based framework for joint spectrum and power allocation aimed at maximizing system throughput while limiting interference to cellular users (CUs). The network environment is modeled as a Markov Decision Process (MDP) that captures essential wireless characteristics, including path loss, interference, and signal-to-interference-plus-noise ratio (SINR). Two DRL algorithms, namely Deep Q-Networks (DQN) and Proximal Policy Optimization (PPO), are implemented using the Stable Baselines3 framework to dynamically assign resource blocks and transmission power levels to D2D pairs in a shared spectrum setting. Simulation results demonstrate that the PPO-based agent consistently outperforms DQN and baseline allocation schemes in terms of average throughput, SINR distribution, energy efficiency, and convergence stability. Overall, the proposed DRL-based approach achieves an effective trade-off between throughput maximization and interference mitigation under dynamic network conditions. While federated learning is discussed as a promising direction for distributed and privacy-preserving optimization, this study focuses on centralized DRL-based resource allocation and leaves federated extensions for future investigation.