Prompt Optimization Using Instruction-Tuned Small Language Models for Cost-Efficient LLM Deployment

Publications

Prompt Optimization Using Instruction-Tuned Small Language Models for Cost-Efficient LLM Deployment

Year : 2026

Publisher : Institute of Electrical and Electronics Engineers Inc.

Source Title : 2026 1st International Conference on Emerging Trends in Advancements and Applications of Computational Intelligence Techniques, ETAACT 2026

Document Type :

Abstract

While many current AI applications use Large Language Models (LLMs), most commercial companies provide LLM services and charge based on token usage, meaning that longer prompts increase inference latency and operational cost, especially for high-volume API applications. Most current methods for compressing prompts rely on some combination of summarization, token pruning, or soft prompt encoding, and therefore do not preserve the original structure or directive intent of user instructions, potentially altering task intent or removing critical information required for accurate downstream responses. In this paper, we frame prompt optimization as rewriting instructions without changing their intent, rather than as a summarization task. We used a synthetic dataset called PromptRewrite-1200 consisting of 1,200 pairs of verbose and concise prompts, in a wide range of domains, to fine-tune a Small Language Model, i.e., google/flan-t5-base (220M parameters). The proposed method achieves an average token reduction of 44.9% while maintaining high semantic similarity (BERTScore F1 = 0.9337) and 77% response-level semantic consistency when evaluated using a Gemini model.