Day 3B Study Guide

An API Is a Standardized Way for Software to Make Requests and Receive Responses

API stands for Application Programming Interface. The word "interface" is the key: an API is a defined contract that specifies how one piece of software can ask another piece of software to do something. The requester sends a structured message to a specific address (the endpoint) with specific parameters; the responder processes it and returns a structured result. Neither side needs to know how the other works internally. They only need to agree on the format of the request and the response.

This is not unique to AI. When a weather app displays the current temperature, it is making an API call to a weather service. When you pay for something online, the checkout page is making an API call to a payment processor. APIs are how modern software systems are composed from separate, specialized services rather than built as a single monolithic program.

In AI systems, APIs serve the same purpose. When a chatbot needs a language model to generate a response, it makes an API call to the model provider, sending the conversation as input and receiving the generated text as output. The chatbot does not contain the model; it accesses it remotely through the API.

AI Systems Are Composed of Multiple APIs Talking to Each Other

A deployed AI product is rarely a single system. It is a chain of API calls. The user interacts with an interface (a website, a survey, an app). That interface calls a backend server. The backend server calls the language model API. The language model may in turn call tool APIs to retrieve information or take actions. Each call is a separate request-response exchange, and each has its own latency, cost, and potential failure mode.

In the Day 1 framework, these API calls are part of the behavior layer: the pipeline is executing, each component is doing its job, and the results flow from one layer to the next. Understanding that each step is a discrete, billable call changes how you think about system design. A slow tool call adds latency the user feels. A verbose tool response adds tokens the operator pays for. A failed API call at any step can break the whole chain.

┌──────────────────────────────────────────────────────────────────────┐
│  HOW AI SYSTEMS COMMUNICATE: A CHAIN OF API CALLS                   │
└──────────────────────────────────────────────────────────────────────┘

  USER
    │  types a message
    ▼
  ┌─────────────────────────┐
  │  Interface              │   (Qualtrics survey, website, mobile app)
  │  (browser / frontend)   │
  └────────────┬────────────┘
             │  HTTP request → backend server
             ▼
  ┌─────────────────────────┐
  │  Backend server         │   Adds API key, assembles full context,
  │  (your application)     │   enforces constraints, logs usage
  └────────────┬────────────┘
             │  API call (with key + context)
             ▼
  ┌─────────────────────────┐
  │  Language model API     │   Reads context window, generates response
  │  (OpenAI, Anthropic...) │   Returns: text + token counts + metadata
  └────────────┬────────────┘
             │  Tool call (if needed)
             ▼
  ┌─────────────────────────┐
  │  External tool APIs     │   Product database, order system,
  │  (optional)             │   web search, calendar, CRM...
  └────────────┬────────────┘
             │  Tool result returned to model
             ▼
  ┌─────────────────────────┐
  │  Language model         │   Incorporates tool result,
  │  (continues)            │   generates final response
  └────────────┬────────────┘
             │  Response text returned to backend
             ▼
  ┌─────────────────────────┐
  │  Backend server         │   Logs response, appends to history,
  │                         │   sends to interface
  └────────────┬────────────┘
             │  Displays to user
             ▼
  USER sees response
    

An API Key Is a Credential That Links Requests to a Billing Account

Every API call to a commercial model provider must include an API key: a long string of characters that identifies the account making the request. The key serves two purposes simultaneously. First, authentication: the provider verifies that the request comes from an authorized account before processing it. Second, billing: every token processed under your key is charged to your account. The key is the link between computation and cost.

This means an exposed API key is not just a security problem; it is an open billing account. Anyone with your key can make API calls charged to you, up to your account's spending limit. The risk is proportional to scale: a key exposed in a public code repository can be found and exploited by automated scrapers within minutes.

This is why, in the LUCID setup for this course, the API key lives on the backend server and never appears in the Qualtrics survey code. When a participant sends a message, Qualtrics calls the backend, which adds the key and forwards the request to OpenAI. The participant never sees the key, and neither does anyone inspecting the survey source.

Every Turn Is a Fresh API Call with the Full History Re-Sent

A chatbot conversation is not a continuous session where the model remembers what was said. It is a series of discrete API calls. Each time a user sends a message, the entire conversation history is assembled and sent to the model from scratch, along with the system prompt. The model has no memory between calls. Everything it knows about the conversation is in the input it receives at that moment.

The input to each API call has three parts: the system prompt (instructions about how the model should behave), the conversation history (every prior user and assistant message), and the new user message. Together these form the context window for that turn. The model reads the full context, generates a response, and returns it. That response is then appended to the history and included in the next call.

This has a direct consequence for cost: input tokens accumulate with every turn. The system prompt is paid for on every call. Each assistant response becomes part of the history and is re-sent as input on all future turns. A conversation does not cost the same each turn; it gets progressively more expensive as history grows.

┌─────────────────────────────────────────────────────────────────────────┐
│  4-TURN HOME DEPOT CONVERSATION                                         │
└─────────────────────────────────────────────────────────────────────────┘

TURN 1
  ┌─────────────────────────────┐
  │ System prompt        120 tk │  ─────────────────────────────────┐
  │ Scenario context      50 tk │                                    │
  │ User: "Does this come        │                                    ▼
  │        with a battery?" 10tk│                             ┌──────────────┐
  │                             │                             │  API call    │
  │ INPUT TOTAL:        180 tk  │  ──────────────────────────►│  OpenAI      │
  └─────────────────────────────┘                             │              │
                                                              │ OUTPUT:95 tk │
                                                              └──────┬───────┘
                                                                     │
                                           ┌─────────────────────────┘
                                           │  Response appended to history
                                           ▼
TURN 2
  ┌─────────────────────────────┐
  │ System prompt        120 tk │
  │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
  │ History (Turn 1)     124 tk │  ◄── Turn 1 output is now input
  │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
  │ User: "What batteries?"  10 │                                    ▼
  │                             │                             ┌──────────────┐
  │ INPUT TOTAL:        254 tk  │  ──────────────────────────►│  API call    │
  └─────────────────────────────┘                             │ OUTPUT:75 tk │
                                                              └──────┬───────┘
                                                                     │
                                           ┌─────────────────────────┘
                                           ▼
TURN 3
  ┌─────────────────────────────┐
  │ System prompt        120 tk │
  │ History (Turns 1-2)  213 tk │  ◄── Grows with every turn
  │ User message          14 tk │                                    ▼
  │ INPUT TOTAL:        347 tk  │  ──────────────────────────►┌──────────────┐
  └─────────────────────────────┘                             │ OUTPUT:120tk │
                                                              └──────┬───────┘
                                                                     │
                                           ┌─────────────────────────┘
                                           ▼
TURN 4
  ┌─────────────────────────────┐
  │ System prompt        120 tk │
  │ History (Turns 1-3)  352 tk │
  │ User: "with the             │
  │        same drill"    4 tk  │  ◄── Only 4 new tokens from user
  │                             │                                    ▼
  │ INPUT TOTAL:        476 tk  │  ──────────────────────────►┌──────────────┐
  └─────────────────────────────┘                             │ OUTPUT:137tk │
                                                              └──────────────┘

  Total input across all turns:  1,257 tokens
  Total output across all turns:   427 tokens
    

Input Tokens Grow Every Turn. Output Tokens Become Input.

A token is roughly three to four characters of text, or about three-quarters of a word. Models do not process words or sentences; they process tokens. Every API call is priced on two quantities: input tokens (everything sent to the model) and output tokens (everything the model generates in response). Input and output are priced separately, and output is usually more expensive per token.

The key dynamic is that output tokens from one turn become input tokens on the next. The assistant's response is appended to the conversation history and re-sent in full on every subsequent call. This means a long assistant response in Turn 2 will be paid for again as part of the input in Turns 3, 4, 5, and so on. Prompt efficiency matters because the cost compounds across turns.

Note that the system prompt (120 tokens) alone accounts for 480 tokens of input across the four turns, simply because it is re-sent every time. A poorly written system prompt is not a one-time cost.

Prompt Caching: Paying Once for What Gets Re-Sent Every Time

Because the system prompt is re-sent on every turn, API providers have introduced prompt caching as a cost optimization. When caching is enabled, the provider stores the processed representation of a long, stable prefix (typically the system prompt and any fixed context) after the first call. On subsequent calls in the same conversation, if that prefix has not changed, the provider reads from the cache instead of reprocessing it. Cached tokens are charged at a significantly reduced rate.

OpenAI charges cached input tokens at 50% of the standard input price. Anthropic's prompt caching is similar in structure, with cache write tokens slightly more expensive than standard input and cache read tokens substantially cheaper. The economics favor any system where the same large prompt is sent repeatedly across many turns or many conversations.

In the Home Depot example, the 120-token system prompt is re-sent on all four turns. With caching, after the first call that prefix is stored and the three subsequent reads are cheaper. At scale across millions of conversations, caching the system prompt can meaningfully reduce the total input token bill without changing the model's behavior at all.

Tool Calls: When the Model Reaches Outside the Context Window

A language model by itself can only work with what is in the context window. It cannot look up a product's current inventory, check an order status, or retrieve a customer's purchase history unless that information is explicitly included in the input. Tool calls extend what the model can do by allowing it to request external data or trigger external actions mid-conversation.

When a tool call is made, the exchange looks like this: the model generates a structured request specifying which tool to call and with what parameters; the application runs the tool (a database query, an API call, a function) and returns the result; that result is then added to the conversation context as a new message and the model continues generating its response. In the Day 1 framework, tool calls are part of the behavior layer, and their outputs become part of the representation on the next step.

The token cost implication is the same as for any other content: the tool call request and the tool result both consume tokens and are included in the history going forward. A tool that returns a verbose response — a full product catalog, a long JSON object — can add significantly to the input token count for every subsequent turn. Designing tool outputs to be concise is a direct cost lever.

Choosing a Model Is a Cost-Quality Tradeoff, Not Just a Quality Decision

The conversation structure described above works identically regardless of which language model you use. The same system prompt, history, and user message are sent; the same token counts apply. What changes is the price per token, and that difference can be substantial.

OpenAI prices models by input and output tokens per million. Output tokens are more expensive than input tokens because generating text requires more compute than reading it. Prices as of the time of this lecture:

The open-source row illustrates an important point: API cost is not the same as total cost. Running your own model requires cloud GPU infrastructure, which at the scale of this example costs more than using GPT-4o via the API. Self-hosting only becomes cost-effective at very high token volumes, where the per-unit infrastructure cost drops below the per-token API price.

For the Home Depot 4-turn conversation (1,257 input tokens, 427 output tokens), the cost on GPT-4o mini is approximately $0.00057 per conversation, well under a tenth of a cent. At one million conversations that becomes $570. On GPT-4o the same conversation costs roughly $7.40 per thousand conversations.