Poster: Comparative Study of Human and Machine level prompts for LLM driven software testing

Publications

Poster: Comparative Study of Human and Machine level prompts for LLM driven software testing

Author : Dr Vivek Yelleti

Year : 2026

Publisher : Institute of Electrical and Electronics Engineers Inc.

Source Title : Proceedings - 2026 IEEE International Conference on Software Testing, Verification and Validation, ICST 2026

Document Type :

Abstract

The quality of the prompts significantly affects the performance of Large Language Models (LLMs). Human experts tend to design prompts based on their level of expertise and cognitive abilities. This leads to multiple interpretations of the same problem. Therefore, factors such as coherence, phrasing, and end-user expectations must be considered when designing prompts. This motivated us to investigate two types of prompts in the context of test case generation for C programs: (i) Humanlevel prompts, crafted entirely by humans, and (ii) Machinelevel prompts, prompts crafted by humans but later refined by various machine tools. We validated the proposed algorithm on 160 C -programs. Experimental results demonstrate that prompts refined by machine tools outperformed human-crafted prompts by 7% across both branch and line coverage criteria.