Due to the abundance and richness of spectral-spatial information, hyperspectral images (HSIs) obtained from hyperspectral imaging have been widely used in a variety of applications, including target or anomaly identification. However, due to its low processing complexity, onboard real-time anomaly identification has always been challenging in hyperspectral image analysis. To achieve high detection accuracy, most existing anomaly detection systems inevitably compromise on high computational complexity. In this paper, a new lightweight field-programmable gate array (FPGA) accelerator is proposed for hyperspectral anomaly detection using HSIs. The proposed approach consists of two stages. In the first stage, average fusion is used to reduce the dimensions of the HSIs. In the second stage, an optimized TinyYOLOv3 accelerator is utilized to extract features and detect anomalies. This optimized TinyYOLOv3 accelerator uses a hardware-friendly shift-based floating-fixed multiply accumulator (MAC) operator and a shift-based quantization method. The shift-based floating-fixed MAC operator is further optimized using a compact LUT-based multiplier (C-LUT-MUL) and an effective floating point adder. The proposed lightweight FPGA Accelerator is implemented on the coding tool Xilinx Verilog using San Diego, Urban-Beach, and EI Segundo datasets. The evaluation results reveal that the proposed accelerator has a higher resource consumption and processing speed (62.5 FPS) while maintaining maximum detection accuracy. This shows the benefits of the proposed lightweight FPGA accelerator over existing research.