Internet of Things(IoT) has been immensely progressed through recent years both in academic as well as in the industrial field. IoT has been widely used in Smart Home System(SHS) to provide the wide variety of facilities. In IoT enabled Smart Home environment various things such as lights, home appliances, computers, cameras, and many others which all are connected to the Internet and allowing the user to monitor and control things at any time and from any location. However, we need to ensure the proper security to maintain the quality of the SHS. In this paper, we propose the novel security protocols based on Matrix inversion method to prevent the loss of user data in IoT based SHS. Further, we observe that data transfers within the SHS are protected by an encryption scheme. We use the partition of matrices to larger size matrices over rings for encryption and decryption using matrix inversion method. Data transfers within the SHS more secured in our proposed system.