In-Context Example Ordering for LLM-Based API Sequence Generation

We demonstrate OptiSeq [1], an edge-cloud middleware system that converts natural language queries into executable API sequences with high reliability in distributed environments. A central challenge in such systems is that large language models (LLMs) can generate incorrect API sequences, causing costly retries that consume edge bandwidth and add cloud latency. Our demonstration shows how OptiSeq, a lightweight inference-time example-ordering optimization engine, reduces these failures by intelligently arranging in-context examples before prompting the LLM. In our distributed architecture, (1) natural language queries originate at the edge, (2) relevant examples are retrieved from an edge-hosted vector store, (3) OptiSeq runs within the middleware pipeline to optimize prompt construction, (4) a cloud-hosted LLM generates API sequences, and (5) a distributed API execution layer carries out the calls. Through live demonstrations on real-world API sequencing tasks, we show that OptiSeq-guided prompting reduces retry attempts by 35-45 % while maintaining high accuracy. This work presents the first demonstration of ordering-aware middleware for natural-language-to-API sequence generation deployed across edge-cloud systems.