This article is intended to correct some very common misrepresentaions of virtual assistants.

When you read a book or course about Prompt Engineering that says that LLMs simply do prompt completion, you might think that they seem to do more. And the ones you likely encounter do do much more.

Most books and courses teach that LLMs just complete prompts by guessing what is likely to follow a prompt provided to it. That however is an oversimplification. Base LLMs do predict sequences statistically, yet the LLMs in virtual assistants you interact with daily are instruction-tuned LLMs that go far beyond that fundamental capability. Instruction-tuned LLMs do rely on next-token prediction internally. Instruction tuning is extensive additional software and data added to a base LLM to provide understanding of intent, the following of instructions, refusal of harmful requests and to match response style to the user’s stated or implied needs. They also decompose complex requests into manageable parts. Responses are then composed to be appropriate for the prompt, not just a dump of material found elsewhere.

Another feature of virtual assistants that is not simple completion is the use of tools, such as search tools, calculators, APIs and generation of images and videos and documents.

When virtual assistants were asked to review the preceding they insist that system prompts (also called system instructions) should also be mentioned. System prompts are hidden instructions that are also included in prompts you enter. System prompts guide behavior, enforce policies and shape responses. OpenAI Codex system prompt includes explicit directive to "never talk about goblins" - Ars Technica is an interesting article about system prompts.

And it is more complicated than that. Instead of there being prompts going into an instance of a LLM and something coming out, the reality is that prompts are processed by an orchestrator that might select one of multiple possible LLM instances or might even use more than one LLM instance at a time. It might augment its knowledge by searching external sources. It might even write and execute a computer program. An orchestrator typically begins by attempting to determine intent. It might create a proposed response then internally critique it and revise it before returning a final response.

Virtual assistants do more than just complete a prompt. When you read a book or see or read a course saying LLMs just complete prompts, ignore that part and be skeptical of everything else the book or course says. There is a good chance that your favorite virtual assistant will guide you more accurately.