Abstract
While many current AI applications use Large Language Models (LLMs), most commercial companies provide LLM services and charge based on token usage, meaning that longer prompts increase inference latency and operational cost, especially for high-volume API applications. Most current methods for compressing prompts rely on some combination of summarization, token pruning, or soft prompt encoding, and therefore do not preserve the original structure or directive intent of user instructions, potentially altering task intent or removing critical information required for accurate downstream responses. In this paper, we frame prompt optimization as rewriting instructions without changing their intent, rather than as a summarization task. We used a synthetic dataset called PromptRewrite-1200 consisting of 1,200 pairs of verbose and concise prompts, in a wide range of domains, to fine-tune a Small Language Model, i.e., google/flan-t5-base (220M parameters). The proposed method achieves an average token reduction of 44.9% while maintaining high semantic similarity (BERTScore F1 = 0.9337) and 77% response-level semantic consistency when evaluated using a Gemini model.