AI concepts from scratch

This page is the glossary-you-can-think-with for the rest of the documentation. It explains, from first principles, the artificial-intelligence ideas the Space builds on. You don’t need a machine-learning background — by the end you’ll know what a token, an embedding and a “context window” are, why temperature matters, how retrieval-augmented generation works, and what makes something an agent rather than a chatbot.

Each section ends with a short “In the Space” note tying the concept to the page that uses it.

Large language models (LLMs)

A large language model is a program trained on an enormous amount of written text that learns to predict the next piece of text given everything written before it. Given The capital of France is, a model assigns a probability to the most plausible next word — most likely Paris.

Two things make this more than autocomplete:

1. Scale. Trained on far more text and with far more parameters than a toy model, it learns grammar, facts, reasoning patterns, programming languages and even style.
2. The next-token dance, repeated. To answer a question, the model is prompted, produces one token, appends it to the prompt, predicts the next one, and continues — each step conditioned on all previous steps. That loop, repeated thousands of times, is how it writes an essay or a function.

A raw model is probabilistic: it samples from the possible continuations rather than picking only the single most-likely one. That randomness is what makes output feel creative, and it’s also why you get a different answer if you re-run the same prompt.

Your model is stateful only through its input. After training, the model has no memory, remembers nothing between calls, and cannot look anything up by itself. Everything it “knows” for a given answer must be written into the request you send — this one fact explains most of the architecture in this documentation (system prompts, memory, embeddings, RAG, tools).

In the Space: Models & API keys lets you choose which model runs; chat and the agent pipeline all send requests through the same token-by-token loop.

Tokens

A token is the unit of text an LLM reads and writes — not a character, not a word, but a subword chunk from the model’s fixed vocabulary. The word “tokenization” splits text into these pieces:

Text	Rough tokenization
Hello world	[Hello] [ world]
unbelievable	often [un] [believ] [able]
café	[caf] [é] (accented chars can split or cost extra)

Roughly, English runs ~0.75 words per token for everyday text (about 4 characters), but it varies by language and content. Code, math and non-Latin scripts tend to use more tokens per “word.”

Why tokens matter:

• They are the currency of cost. Providers bill per token — typically a cheaper input (prompt) price and a pricier output (completion) price.
• They cap how much a model can handle at once. A model with a 128k context window can read about as much text as fits in 128k tokens in a single request.
• They size what gets sent. When the Space shows “estimated tokens in context,” it’s estimating how much of the model’s window your current messages and memory would consume.

Metrics like “reasoning tokens” refer to the tokens the model spends on thinking before answering (see Reasoning later).

In the Space: the usage dashboard (/space/governance) reports prompt/completion/cached/reasoning tokens per request; the model catalog shows per-token pricing.

The context window and “in-context learning”

A model processes a fixed amount of text per request, called its context window (a.k.a. context length). Everything you want the model to “see” for one answer — system instructions, conversation history, past memory, retrieved documents — must fit inside that window for that request.

There is no separate short-term storage that grows as you chat: a conversation is a growing prompt. Each assistant reply is not remembered by the model; the client re-sends the accumulated history (up to a limit) so the model appears to remember. This is called in-context learning: the model learns what to do not by updating itself, but from what you put in the context.

Consequences:

• Long conversations eventually exceed the window, so the platform applies a context limit — it keeps the most recent and most relevant messages and drops older ones.
• This is why memory and the knowledge base exist: to compress what’s relevant into something that fits in the window.

Think of the window as a desk. You can’t hold infinite papers — when it fills, you have to decide which papers stay on the desk. The Space’s context limit and RAG are that “what stays on the desk” logic.

In the Space: the contextLimit global setting and the per-page memory/knowledge tools exist precisely because the window is finite.

Sampling parameters: temperature, top-p, top-k, penalties

When the model produces the next token, it computes probabilities over its whole vocabulary. Sampling parameters shape which of those tokens is actually picked — how “creative” or how “deterministic” the model behaves.

• Temperature. Scales the probability distribution. Low (e.g. 0.1) makes the model almost always pick the most likely token → focused, deterministic answers. High (e.g. 1+) flattens the distribution → more diverse, sometimes surprising output. 0.7 is a common default balancing coherence with creativity.
• Top-p (nucleus sampling). Only consider the smallest set of tokens whose combined probability reaches p. 1 means no truncation; 0.9 restricts output to the top 90% probability mass.
• Top-k. Only consider the k most likely tokens. Lower k = more conservative. 0 typically means disabled.
• Presence / frequency penalties. Reduce the likelihood of tokens that have already appeared — discouraging repetition. Presence penalizes reappearing tokens regardless of count; frequency penalizes proportional to how often they repeat.

None of these are “checks a science box” — they’re the knobs that decide whether the assistant answers dryly, offers options, or re-explains patterns. This is also why the same prompt can yield different results: sampling is stochastic.

In the Space: Global Settings exposes temperature, max tokens, top-p, top-k and presence/frequency penalties, persisted in synthhires:globalParams.

System prompt vs. user prompt

A request to an LLM typically carries two kinds of instructions:

• System prompt — the standing instructions that define the assistant’s role and rules (tone, boundaries, what tools it may use, output format). It sits at the top and applies on every turn.
• User prompt / messages — the specific request and the ongoing conversation.

The boundary is a design choice: the system prompt is where the platform injects agent identity (see below), and where your global system instruction (You are a helpful AI assistant…) lives. Put the standing rules in the system prompt, the current task in the user message.

In the Space: each agent ships a structured XML system prompt (<identity><capabilities><tools><constraints>); your global system instruction is a setting.

Embeddings and semantic distance

An embedding is a numerical vector (a list of numbers) that represents the meaning of a piece of text. The trick is geometric: texts with similar meaning end up at similar positions in the vector space, so you can measure “how related are these two things” by computing how close their vectors are (typically by cosine similarity).

Text	~Meaning interpreted as
“cat”	a point near “kitten”, “feline”, far from “invoice”
“refund policy”	near “return policy”, far from “deploy”

Embeddings give you semantic search: instead of matching keywords, you compare meaning. Search “how do I refund this invoice?” and a store of documents about return policies surfaces even though no query word appears verbatim.

Why this matters here: a plain substring search fails on paraphrase; embeddings don’t care about exact wording, only about being about the same thing.

In the Space: the knowledge base stores each ingested document split into chunks, computes an embedding per chunk, and searches by nearest vector — see the RAG section below and Memory & knowledge.

Chunking

Chunking is splitting a long document into small, digestible pieces before embedding or feeding them to a model. Because the context window is finite and search works per piece, a whole chapter is useless — a model can’t retrieve “the bit about refunds” if it’s buried in ten thousand tokens.

Good chunking respects natural boundaries (paragraphs, headings, code blocks) so each chunk is one coherent idea, small enough to embed well and to fit in the window alongside the rest of a prompt.

In the Space: chunkDocument (src/lib/rag/vector.ts) splits uploads into chunks that are individually embedded and indexed — the unit you later retrieve.

RAG (Retrieval-Augmented Generation)

RAG is the pattern of augmenting a model’s answer with retrieved context it wasn’t trained on. The generator doesn’t invent the facts from nothing — it’s handed relevant material up front and asked to answer grounded in it.

The flow:

1. Ingest — split a document into chunks and embed each one.
2. Query — embed the user’s question.
3. Retrieve — find the chunks whose embeddings are closest to the question’s embedding.
4. Generate — put those chunks in the prompt as grounding context, and let the model compose the answer from them.

RAG solves three hard problems at once: it brings in up-to-date/private knowledge the model never saw, it grounds answers in sources (reducing confident hallucinations), and it keeps the context window affordable by sending only the relevant snippets rather than whole documents.

RAG vs. fine-tuning: fine-tuning changes the model itself; RAG leaves the model untouched and changes the context you feed it. RAG is fast to update (just add documents) and fully auditable — you know exactly which chunk grounded a given answer.

In the Space: the Knowledge Base (/space/knowledge) is a RAG store. Ingest uploads/URLs/YouTube, and POST /api/rag with search returns the relevant chunks you can inject as “grounding context.”

What makes something an “agent” (vs. a chatbot)

A chatbot completes a conversation. An agent is a loops program that can act: besides generating text, it can call tools, observe results, and decide the next action, iterating until it reaches a goal.

The defining loop:

1. Reason about the task with an LLM.
2. Decide what to do next — answer, or call a tool.
3. Act by calling an external function (read a file, run code, query GitHub, send a message).
4. Observe the tool’s result, feed it back into context.
5. Repeat until done, then answer.

Tools therefore turn a text generator into an actor: without tool-calling the model can only describe actions; with tools it can perform them. This is why agents need scoped capabilities, consent gates, and audit logs (see Runtimes & bridge).

Beyond a single loop, a system can use multi-agent orchestration: a lead agent plans and delegates subtasks to specialized agents, which may themselves spawn workers — a tree of cooperating agents.

In the Space: the chat uses a Lead Orchestrator that delegates to the 16-agent catalog; the hierarchy panel lets you spawn executors/verifiers; the pipeline studio runs whole graphs of these agents.

Function/tool calling

Tool calling (a.k.a. function calling) is the formal mechanism behind “an agent can act.” The model is told the schema of available tools (name, parameters, description) but it cannot run them — it can only emit a request to call one, with arguments. The application runs the real function, gets the result, and feeds the result back to the model as a tool message, letting the model continue.

Example with GitHub:

1. LLM announces call get_issue({"number": 42}).
2. Your code calls the GitHub API, gets the issue body.
3. That body is added to the conversation; the model reasons over it and answers.

Tool calls are how the platform bridges the model to your codebase, filesystem, network, databases and messaging — each scoped by a capability and consented to.

In the Space: agents declare recommended tools; MCP servers supply them via a standard protocol; the bridge exposes desktop.*/mobile.*/sync.* capabilities as callable actions.

MCP (Model Context Protocol)

MCP — the Model Context Protocol — is an open standard that gives models tools and resources from external servers through one common interface. Think of it as “USB-C for model tools”: instead of each integration needing custom code, any MCP server (GitHub, Notion, Slack, a company-internal API) exposes its capabilities uniformly, and the model calls them through MCP.

• Tools: actions a server exposes (e.g. create_issue, search_issues).
• Resources: data the model can read (e.g. files, schemas).
• Prompts: reusable prompting templates from the server.

On edge runtimes (like Workers) there is no process to spawn, so MCP runs over Streamable HTTP rather than spawning a local stdio process.

In the Space: the MCP Servers tab manages a registry of connectors (GitHub, GitLab, Jira, Asana, Linear, Notion, Slack, Discord…), each with its server package and setup mode.

Streaming

Streaming is sending the model’s answer to the client as it’s generated, token by token, instead of waiting for the whole output. It’s a perceived-latency win: the first words appear in milliseconds even if the full answer takes many seconds, and it turns a long wait into a “typing” experience.

Architecturally it means the response is a stream (SSE or chunked), not a single buffered string — the server flushes partial completions continuously and the client renders them incrementally.

In the Space: chat streams by default via streamText; the route never buffers the full response. Streaming is a deliberate product requirement.

Reasoning models and “thinking” tokens

Reasoning models (e.g. GPT-5/“thought”, Claude reasoning, Gemini think, DeepSeek reasoner) spend extra tokens on an explicit internal chain-of-thought before producing the final answer. Instead of answering in one shot, they draft reasoning steps, self-check, and then answer — which improves complex/mathematical/debugging performance at the cost of more latency and more (reasoning) tokens.

The platform exposes a thinking level (off / low / medium / high) that maps to the provider’s own reasoning-effort control. Providers differ in whether reasoning is optional or always-on, so the client derives the capability and sends the right parameter per provider.

In the Space: each conversation stores a thinking level; src/lib/thinking.ts computes the correct reasoning-control parameter for OpenAI, Anthropic, Gemini and DeepSeek.

Structured output and “plan extraction”

LLMs produce free text, but applications often need structured data (a plan with steps, a JSON result, an object). Structured output constrains the model to return something that matches a schema — typically by supplying a JSON schema and/or parsing + validating.

Two complementary techniques used together:

1. Schema-guided output — ask/train the model to return JSON matching a defined schema (Output.object({ schema })).
2. Tolerant parsing — separately, scan whatever text came back for a JSON-like block and validate it strictly, so output survives even when the model wraps it in prose or forgets code fences.

In the Space: the orchestrator’s plan is extracted from the assistant’s reply via a tolerant JSON scanner + strict Zod validation (safeParsePlan), then opened as a graph in the Pipeline studio.

Fine-tuning vs. prompts, briefly

Fine-tuning updates the model’s weights on a dataset; prompting learns in-context from what you type. A base model is often powerful enough for most tasks; fine-tuning is reserved for specializing a model on a domain, format or tone. RAG and prompting are cheaper, reversible, and auditable (you always know what context was fed in) — which is why the Space leans on them and on BYOK rather than hosting or fine-tuning models.

The thread that ties it together

That’s the mental model: a token-by-token sampler over a finite context window, steered by prompts and sampling knobs, grounded by retrieval (RAG) when facts are needed, extended with tools/MCP to act, composed into agent loops and pipelines, and delivered streamed to your screen. With this vocabulary, the rest of the documentation describes how the Space wires those ideas to its pages, APIs and storage.

Next: Overview for the product map, or Architecture for the request path.