This paper deals with a novel approximation technique for constrained optimal feedback control problems. We consider a conventional optimal control processes governed by the closed-loop dynamics and apply the β-relaxation approach. We next establish some useful convergence properties of the resulting relaxations for the extended and for the originally given Feedback Optimal Control Problems (FOCPs). The approximative approach involving the β-relaxation reduces the nonlinear closed-loop dynamic system to a control-affine model. This system reduction makes it possible to apply the generic Hamilton–Jacobi–Bellman (HJB) solution methodology and determine an exact optimal feedback in the relaxed FOCP. The resulting explicit solution scheme is finally applied to the originally given constrained FOCP. The obtained theoretical results provide a rigorous mathematical basis for the possible numerical approaches to the optimal policy iterations.