This paper proposes a semi-fragile video watermarking scheme suitable for data authentication of surveillance video. To authenticate any video it is not mandatory to watermark all the frames within it. Instead of that watermarking a few frames such that modification of those frames will lead to change the semantic is also sufficient. A novel scene change detection algorithm is proposed to detect the major scene changes in video and a few selected frames in each scene have been used for watermarking. The scene change is identified by a combined measure of weighted block based mean absolute frame differences and count of edge pixels. To watermark a specific frame an authentication code has been generated from the highest informative block based on the discrete cosine transform (DCT) coefficients. The authentication code is further inserted into the less informative blocks of same frame by a new watermarking algorithm based on pixel intensity swapping within a four neighborhood region. Information within a block of frame is measured in terms of entropy. Experimental study has done on video from standard video data set shows good performance. The comparative study of performance with existing technique has been done.