In this paper, we investigate an encoder–decoder-based on the bi- directional transformers with a self-attention function to generate abstractive text summaries for an input document. In the proposed approach, we fine-tune the transformer by changing the activation function from the rectified linear activation unit (ReLU) to the parametric rectified linear activation unit (PReLU). By introducing the PReLU activation function, we can store the long-term dependencies from the input document by reducing the data loss which was more when we used the ReLU function. Apart from that, we introduce a self-attention function for keeping track of the important keywords and the out-of-vocabulary words presented in the input document. We employ CNN/DailyMail dataset, Inshorts dataset, and customized IndiaToday data to achieve abstractive text summarization. The proposed model achieves a better ROUGE score when compared with the sequence-to-sequence with long short-term memory network and the traditional transformers model.