Federated Learning (FL) has garnered widespread attention in finance, banking, and healthcare due to its decentralized, distributed training and the ability to protect privacy while obtaining a global shared model. However, FL faces challenges such as communication overhead and limited resource capability. This motivated us to propose a first-of-its-kind, two-stage FL approach as follows: (i) During phase I, under non-federated settings, synthetic dataset is generated by employing two different probability distributions as noise to the vanilla conditional tabular generative adversarial neural network (CTGAN) resulting in modified CTGAN. We also employed standard metrics to assess the quality of synthetic datasets. (ii) In phase II, the Federated Probabilistic Neural Network (FedPNN) is developed for building globally shared classification model. Despite PNN being a one-pass learning classifier, its complexity depends on the training data size. Therefore, we employed a modified evolving clustering method (ECM), another one-pass algorithm, to cluster the training data, in between the input and pattern layers of the FedPNN. The effectiveness of our approach is validated on credit card fraud detection and Polish bankruptcy prediction datasets.