Tokenization, transformers, embeddings, attention, completion generation, prompt engineering, and multi-agent systems
Suggested study time: 32 minutes • Beginner level • Original rewrite based on Microsoft Learn objectives
By João Ricardo Dutra••Complete original content
1. Introduction to generative AI
Learning objectives
Explain how large and small language models represent linguistic and semantic relationships.
Describe tokenization, vectors, embeddings, attention, encoders, decoders, and completion prediction.
Distinguish system and user prompts and apply history, RAG, and effective prompting practices.
Identify the components of AI agents and collaboration in multi-agent systems.
Generative AI has become highly visible because it can create original material that resembles human work, including poetry, prose, images, and code. The result can feel extraordinary, but it comes from mathematical methods refined over many years of research in statistics, data science, and machine learning.
A high-level understanding makes today’s technology easier to evaluate and helps reveal future possibilities. It also explains how generative models support a new generation of agents that can find information, choose next steps, and carry out tasks.
The original Microsoft Learn material can be completed through video or through text and images. The written version generally contains more detail and can supplement the video presentation.
2. Language models and completion prediction
Large language models, or LLMs, and their more compact relatives, small language models, or SLMs, encode relationships among words, passages, and meanings in a vocabulary. Those relationships support reasoning over natural-language input and the generation of coherent, relevant responses.
Fundamentally, training teaches a model to complete sequences that begin with a prompt. It resembles predictive text on a phone, but at a far larger scale: at each step, the model considers what has appeared, estimates which elements most influence the continuation, and chooses a likely next token.
In “I heard a dog...”, for example, “heard” suggests that a sound will follow, while “dog” points toward sounds associated with that animal. A person predicts “bark” because they have a vocabulary, understand language structure, and connect words to concepts. Training aims to give the model comparable capabilities.
Foundations of linguistic prediction.
Foundation
Contribution
Large vocabulary
Provides many possible units for a continuation.
Linguistic structure
Represents ordering patterns and relationships that form meaningful sentences.
Semantics
Connects words and passages with concepts, contexts, and similar uses.
3. Tokenization: from text to identifiers
The first step is to build a very large vocabulary from extensive training corpora, including public Internet content and other sources. Current models may work with hundreds of thousands of tokens.
A token is not limited to a whole word. The vocabulary can also contain subwords, punctuation, and common character sequences. A prefix such as “un” in “unbelievable” can therefore be its own unit. Every distinct token receives a unique integer ID, and repeated occurrences reuse that identifier.
Simplified word-level tokenization.
Token
ID
I
1
heard
2
a
3
dog
4
bark
5
loudly
6
at
7
cat
8
This teaching example uses whole words for readability. A production model also includes subwords and punctuation, and its vocabulary grows as more training material is processed.
4. Transformer: encoder, attention, and decoder
IDs alone do not express meaning. Each token first receives a vector, which is an array of numbers. A transformer converts these initial vectors into representations containing linguistic and semantic properties. Because those properties are embedded in the vectors, the results are called embeddings.
A simplified view divides a transformer into two blocks. The encoder creates embeddings with attention: it examines a token in the context of neighboring tokens, assigns influence weights, and passes the result through a fully connected neural network. Multi-head attention evaluates several characteristics in parallel to make the process more efficient.
The decoder uses the embeddings to estimate the next token in a prompt-started sequence. It also combines attention with a feed-forward network. The real architecture is more elaborate; the key idea is that attention captures token characteristics from the contexts in which tokens occur.
The SVG redraws the conceptual architecture: tokens and position enter the encoder, embeddings provide context, and the decoder predicts a continuation.
5. Position, attention, and embeddings
Before training, token-vector values may start randomly. They enter the model together with positional encoding because sequence order changes meaning and token relationships. A teaching table might use only three dimensions, while real model vectors contain thousands of elements.
During training, attention evaluates every token in context. In “I heard a dog bark,” “heard” and “dog” influence “bark” more strongly than articles or pronouns. The model does not know those weights initially; repeated exposure to large text collections reveals proximity and frequency patterns, and its parameters are adjusted iteratively.
The result is a vector space where tokens used in similar settings point in similar directions. Embeddings for “dog,” “puppy,” and “cat” are likely to be closer than those for “car” or “skateboard.” Cosine similarity provides a mathematical measure of this semantic proximity.
Semantically related vectors point in similar directions; a real model uses far more than three dimensions.
6. How the decoder generates a completion
Once embeddings represent contextual relationships, the decoder can predict a continuation one token at a time. During training it uses masked attention: when predicting a position, the model can consider only preceding tokens and ignores everything after it.
Because the correct sequence is known during training, the prediction can be compared with the actual next token. The error guides weight adjustments in later iterations. During generation, the next token is unknown; attention and the feed-forward network score candidates, the selected token is appended, and the cycle repeats until the model predicts an ending.
Given “When my dog was a...,” context makes “puppy” more likely than “cat” or “skateboard.” The full output emerges from repeatedly conditioning each prediction on everything generated so far.
7. System prompts and user prompts
A prompt is the input sent to an LLM to obtain a completion. It may be a question, a command, or a casual remark that starts a conversation.
The two main prompt types.
Type
Responsibility
Reworded example
System prompt
Sets behavior, tone, role, and persistent constraints.
Act as a helpful assistant and respond in a friendly manner.
User prompt
Requests an answer to a specific question or instruction.
Summarize generative AI adoption points for an executive in no more than six professional bullets.
The application usually supplies the system prompt. A user prompt may be written by a person in a chat or generated by the application on that person’s behalf. The response should address the specific request while following the overall guidance.
The system establishes global rules; the user provides the task that produces a completion.
8. Conversation history and RAG
Conversational applications often retain history and include summarized versions of earlier turns in later prompts. This preserves continuity and lets the model understand a follow-up such as “What are the privacy risks?” after discussing corporate generative AI adoption.
Earlier messages and the follow-up form context that supports a coherent answer.
Retrieval-augmented generation, or RAG, adds external context. The application searches documents, email, or other sources, selects relevant passages, and inserts them into the prompt. The answer is then grounded in retrieved information rather than relying only on the model’s general knowledge.
An expense assistant might receive a question about the business-travel allowance, search the company policy, retrieve the relevant section, and send it with the question. Without retrieval, a model would probably offer generic advice to consult the policy; with RAG, it can answer from the organization’s actual rules.
RAG connects a question to relevant sources and grounds the model’s completion.
9. Writing better prompts
Response quality depends on both the selected model and the instructions it receives. A well-designed prompt reduces ambiguity and explicitly communicates the desired outcome.
Be clear and specific: express the question or task without vague wording.
Add context: state the subject, audience, goal, and required format.
Provide examples when a particular style or pattern should be followed.
Ask for structure: request bullets, tables, numbered steps, or another useful layout.
Clarity, context, examples, and structure make an instruction easier to follow.
10. AI agents and multi-agent systems
An AI agent is a generative-AI application that does more than answer. It reasons over natural language, uses tools, considers contextual conditions, and performs an appropriate action on a user’s behalf.
The three elements of an agent.
Element
Purpose
Large language model
Provides the core language understanding and reasoning capability.
Instructions
A system prompt describes the agent’s role, behavior, and boundaries.
Tools
Knowledge tools access search and databases; action tools send email, update calendars, or control devices.
Together, the three components let an assistant turn context into a controlled action.
Agents can also collaborate. In a multi-agent system, each participant has a specialty: one gathers data, another analyzes it, and a third takes action. They exchange prompts to determine required work, assign responsibility, and share results, creating a workflow that can handle more complex processes.
Prompt-based coordination divides a complex workflow among specialized agents.
11. Hands-on exploration and knowledge check
The module exercise uses a chat playground to interact with a generative model and observe the practical effects of system prompts, tools, and grounding with data.
The real interface capture is preserved as a PNG in accordance with the screenshot rule.
