This research focuses on the development of a Deep Learning (DL) System that is intended to provide accurate detection of Oil Spills in Port Images (RGB) using a DL framework. The method overcome issues such as noise and poor visibility. The first step of this process involving using an Adaptive Scharr Filter (ASF) to preprocess the image so that image contrast is improved while removing the majority of the noise but keeping important edge information in the image. After preprocessing the image, a Residual Segmentation Network (Res-SegNet) is created using a deep residual architecture that connects the multiple layer of neural network together to create connections that incorporate multiple scales of spatial data into one final output. The Grizzly Bear Optimization (GBO) is used to further improve Res-SegNet accuracy by tuning hyperparameters and eliminating segmentation errors caused during image processing and training. In addition, a clustering module incorporating into the final network architecture that refines and combines previously segmented region into larger with larger groups with more consistent classifications of oil spill areas while reducing the number of false positives. The findings of the experiments shows that Res-SegNet is more accurate than standard models at finding oil spills in a variety of port images, with an overall accuracy of 95%.