Remote underwater picture and video capture is used by marine biologists to monitor different fish species in their natural environments. This aids in their comprehension and forecasting of the responses of fish to fishing pressure, habitat degradation, and climate change. Having this knowledge is crucial for creating environmentally friendly, sustainable fisheries for human use. Humans, on the other hand, find it difficult and time-consuming to extract useful information from massive amounts of collected videos. Deep learning (DL) appears to have an issue with this. With the help of DL, marine biologists can rapidly and effectively parse massive amounts of film, uncovering specialized information that is not accessible via manual monitoring techniques. We present a two-step deep learning technique in this study that can recognize and classify temperate fishes without the use of pre-filtering. Every fish in a picture must first be identified, regardless of species or gender. For this, we employ the You Only Look Once (YOLO) object detection technique. The classification of each fish in the image is done in the second stage using a squeeze-and-excitation (SE)-designed convolutional neural network (CNN). Despite the short training sample size of temperate fishes, we use transfer learning to improve classification accuracy. For this, the fish classifier was trained using a public dataset, and the object detection model was trained using ImageNet. Both models were then updated with pertinent temperate fishes. Weights are always added both before and after a workout. The CNN-SE model performed admirably, with a 96.22% accuracy. Extensive comparative research revealed that the CNN-SE model outperformed more recent approaches.