Ultra-short-term wind power forecasting (UST-WPF) under non-stationary conditions demands dynamic policy adaptation, a capability that supervised learning approaches fundamentally cannot provide. Distributional shift in wind time series is inherently a sequential decision problem, and inter-agent coordination in parameter space, rather than output space, is critical for stable distributed forecasting across spatially heterogeneous turbines. To validate this perspective, Asynchronous Advantage Actor-Critic (A3C)-TimesNet- Long Short-Term Memory (LSTM) is proposed, recasting UST-WPF as a decentralized multi-agent reinforcement learning problem where each turbine agent learns adaptive forecasting policies through environment interaction rather than fixed regression targets. TimesNet provides frequency-aware state representations by encoding intra- and inter-period spectral variations, while LSTM preserves sequential temporal context across non-stationary regimes. The proposed Local Consensus-Based Policy Synchronization (LCPS) mechanism enforces structural alignment directly in parameter space, avoiding gradient interference and temporal coupling limitations inherent to loss-based consistency regularization in asynchronous settings. Empirical evaluation on the Baidu Knowledge Discovery and Data Mining (KDD) Cup 2022 dataset demonstrates a 12.9% MAE reduction over static baselines and, critically, a 4.0% improvement over an online fine-tuning baseline. This demonstrates reinforcement-based policy learning as a viable and superior alternative to gradient-based adaptation for non-stationary wind power forecasting.