Tokenization, Decoding, and Context Engineering: The Low-Level Mechanics Behind LLM Behavior

Many engineers work with LLMs for months before they develop a precise intuition for three mechanisms that dominate model behavior in practice: tokenization, decoding, and context construction. These topics are often treated as implementation details. In reality, they are core drivers of quality, cost, latency, and reliability.

This article takes a low-level engineering view of these mechanisms. The goal is to explain how they shape model behavior, how they interact, and why strong LLM systems require explicit control over all three.

1. Tokenization Is the Real Input Layer

An LLM does not process raw text. It processes tokens. Those tokens may correspond to full words, subwords, punctuation patterns, bytes, or fragments of source code. That means the true input length of a request is not its character count or even its visible word count. It is the tokenized representation the model actually sees.

This matters operationally because tokenization affects:

• context window usage
• inference cost
• batching behavior
• multilingual performance
• code and table handling

A common beginner mistake is to estimate context usage visually. Professionals measure tokens directly.

2. Tokenization Behavior Varies by Modality and Domain

The same sentence can tokenize very differently depending on:

• natural language versus code
• English versus morphologically rich languages
• clean prose versus logs or markup
• short identifiers versus long-form documentation

In enterprise systems, documents with lots of tables, IDs, JSON, or source code often consume more tokens than teams initially expect. This has direct consequences for retrieval chunking and prompt budgets.

3. Why Tokenization Changes Product Design

Once you understand tokenization, several product decisions become easier to reason about:

• whether to summarize before prompting
• how large retrieval chunks should be
• whether few-shot examples are too expensive
• when a long-context model is worth the extra serving cost

Good context engineering starts with token awareness.

4. Decoding Is Where Model Probability Becomes Product Behavior

After the model computes token probabilities, the serving system must decide how to decode output. This choice has major effect on answer style, determinism, and hallucination risk.

The common levers are:

• temperature
• top-k sampling
• top-p sampling
• max token limits
• stop sequences
• repetition penalties

These are not cosmetic settings. They shape the model’s behavioral distribution.

5. Greedy, Sampling, and Constrained Generation

Greedy decoding

Greedy decoding always selects the highest-probability next token. It is stable and simple, but can become repetitive or rigid.

Probabilistic sampling

Sampling allows lower-probability tokens to be selected under controlled conditions. This increases diversity, but can also increase variance and error rates.

Constrained or structured generation

When the output must satisfy a schema or grammar, constrained generation is often preferable to free-form sampling. This is especially important in extraction, function calling, and structured agents.

6. Why Decoding Strategy Should Match the Task

Different tasks justify different decoding policies:

• classification and extraction benefit from lower variance
• brainstorming may benefit from controlled diversity
• code generation often needs tighter output constraints
• tool invocation should be highly structured and conservative

A professional system should not use one default decoding profile for every workload.

7. Context Engineering Is More Than Prompt Writing

Prompt engineering focuses on instructions. Context engineering focuses on everything surrounding the prompt that influences model behavior.

That includes:

• retrieved documents
• system policies
• conversation history
• tool outputs
• examples
• summaries of earlier steps

Most bad outputs are not caused by the model being weak. They are caused by the model seeing the wrong context, too much context, or badly ordered context.

8. Ordering Effects Are Real

The placement of information inside a context window matters. Models do not treat every token equally. In practical terms:

• conflicting instructions can cause unstable behavior
• long context can bury critical evidence
• examples placed too far away may lose influence
• irrelevant retrieval passages can distract the generator

This is why context packing is an engineering task, not just a formatting task.

9. Retrieval and Context Engineering Must Be Designed Together

A retrieval system that returns good documents can still produce weak answers if context assembly is poor. Strong systems usually decide:

• how many passages to include
• in what order to include them
• whether to compress or summarize them
• whether some evidence should be quoted verbatim
• whether metadata should be exposed to the model

This is one reason retrieval quality and generation quality cannot be tuned independently.

10. Practical Patterns

Pattern 1: Small trusted context

Best for highly controlled workflows such as extraction, classification, or tool selection.

Pattern 2: Retrieval plus selective packing

Best for enterprise QA where documents are large but only some fragments are relevant.

Pattern 3: Summary memory plus fresh context

Best for longer multi-turn tasks where old state must be compressed before new evidence is added.

11. Minimal Implementation Sketch

def build_context(question, docs, budget_tokens):
    selected = []
    used = 0
    for doc in docs:
        doc_tokens = count_tokens(doc)
        if used + doc_tokens > budget_tokens:
            break
        selected.append(doc)
        used += doc_tokens
    return "\n\n".join(selected)


def decode_request(question, context, llm):
    prompt = f"Question: {question}\n\nContext:\n{context}"
    return llm.invoke(prompt, temperature=0.2, max_tokens=600)

This example is deliberately simple, but it captures the production reality: token budget and decoding policy are part of the application logic.

12. What to Measure

Useful metrics in this layer include:

• input token count
• output token count
• context truncation frequency
• schema-valid output rate
• latency by prompt size
• answer quality by decoding profile

Without these, teams often guess at the effect of token budgets and sampling policies.

13. Common Failure Modes

• token budgets silently exceeded and context truncated
• wrong decoding profile applied to a structured task
• too many retrieved passages included without ranking discipline
• conversation history grows until useful evidence is diluted
• prompt examples consume most of the available context

14. Tokenization in Practice: Code and Counting

BPE with tiktoken

import tiktoken

# GPT-4 / GPT-4o tokenizer
enc = tiktoken.get_encoding("cl100k_base")

text = "Hello, how are you? 你好，你好吗？"
tokens = enc.encode(text)
print(f"Text:   '{text}'")
print(f"Tokens: {tokens}")
print(f"Count:  {len(tokens)}")

# Decode back to text
decoded = enc.decode(tokens)
print(f"Decoded: {decoded}")

# Inspect token boundaries
for token_id in tokens:
    piece = enc.decode([token_id])
    print(f"  {token_id:6d} → {repr(piece)}")

Token counting for cost and context management

import tiktoken
from functools import lru_cache

@lru_cache(maxsize=8)
def get_encoder(model: str = "gpt-4o") -> tiktoken.Encoding:
    try:
        return tiktoken.encoding_for_model(model)
    except KeyError:
        return tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    return len(get_encoder(model).encode(text))

def count_messages_tokens(messages: list[dict], model: str = "gpt-4o") -> int:
    """Count tokens for a list of chat messages, including role overhead."""
    enc = get_encoder(model)
    total = 0
    # Each message has 4 tokens of overhead (role, content, separator, newline)
    for msg in messages:
        total += 4
        total += len(enc.encode(msg.get("content", "")))
        total += len(enc.encode(msg.get("role", "")))
    total += 2  # reply priming overhead
    return total

# Usage
messages = [
    {"role": "system",    "content": "You are a helpful assistant."},
    {"role": "user",      "content": "Explain transformer attention."},
]
print(f"Messages token count: {count_messages_tokens(messages)}")

# Context budget management
def fits_in_context(text: str, max_tokens: int = 4096, model: str = "gpt-4o") -> bool:
    return count_tokens(text, model) <= max_tokens

Tokenization behavior across domains

examples = {
    "English prose":    "The quick brown fox jumps over the lazy dog.",
    "Code":             "def fibonacci(n): return n if n <= 1 else fibonacci(n-1) + fibonacci(n-2)",
    "JSON":             '{"user_id": "usr_12345", "action": "login", "timestamp": 1703123456}',
    "Arabic":           "مرحباً كيف حالك؟",
    "Technical IDs":    "CVE-2024-12345 RFC-7231 ISO-8601",
    "Repeated symbols": "==========================================================",
}

enc = tiktoken.get_encoding("cl100k_base")
for name, text in examples.items():
    n = len(enc.encode(text))
    print(f"{name:20s}: {len(text):4d} chars → {n:3d} tokens ({len(text)/n:.1f} chars/token)")

Expected output patterns:

• English: ~4 chars/token
• Code (Python): ~3 chars/token (identifiers split)
• JSON with IDs: ~2.5 chars/token
• Arabic: ~2 chars/token (morphologically rich)
• Repeated symbols: ~1–2 chars/token

---

15. Decoding Strategies in Practice

Temperature, Top-K, and Top-P (Nucleus Sampling)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import torch.nn.functional as F

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
model     = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

def decode_with_params(
    prompt:      str,
    temperature: float = 1.0,
    top_k:       int   = 50,
    top_p:       float = 0.95,
    max_tokens:  int   = 200,
) -> str:
    inputs  = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=max_tokens,
        do_sample=temperature > 0,
        temperature=temperature,
        top_k=top_k,
        top_p=top_p,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )
    # Slice off the input tokens, decode only generated tokens
    generated = outputs[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(generated, skip_special_tokens=True)

prompt = "[INST] What are the top 3 causes of climate change? [/INST]"

# Deterministic (temperature=0 or greedy)
print("=== Greedy (temp=0) ===")
print(decode_with_params(prompt, temperature=0))

# Diverse creative output
print("\n=== High temperature (temp=1.2) ===")
print(decode_with_params(prompt, temperature=1.2, top_p=0.95))

# Conservative, focused
print("\n=== Conservative (temp=0.2, top_k=10) ===")
print(decode_with_params(prompt, temperature=0.2, top_k=10))

Decoding strategy selection guide

Task	Temperature	Top-K	Top-P	Notes
Factual QA	0.0–0.2	10–20	0.9	Minimize variance
Code generation	0.0–0.3	20–50	0.95	Syntax must be correct
Creative writing	0.8–1.2	0 (off)	0.95	Diversity welcome
Brainstorming	1.0–1.3	0	0.97	Maximum diversity
Structured extraction	0.0	1–5	0.9	Schema compliance first
Summarization	0.3–0.5	40	0.92	Accurate but fluent
Tool calling / JSON	0.0	1–5	0.9	Constrained output

---

16. Constrained / Structured Decoding

When output must conform to a schema, constrained generation ensures compliance at the token level:

# Using Outlines for grammar-constrained generation
import outlines
from pydantic import BaseModel
from typing import Literal, Optional

class TicketClassification(BaseModel):
    category:  Literal["billing", "technical", "general", "complaint"]
    urgency:   Literal["low", "medium", "high"]
    summary:   str
    escalate:  bool

# Load model
model = outlines.models.transformers(
    "mistralai/Mistral-7B-Instruct-v0.2",
    device="cuda",
)

# JSON generator constrained to schema — ALWAYS valid JSON
generator = outlines.generate.json(model, TicketClassification)

result = generator(
    "[INST] Classify this support ticket:\nMy account was charged twice! [/INST]"
)
print(result)
# TicketClassification(category='billing', urgency='high', summary='Double charge', escalate=True)

# Using Guidance for complex structured templates
import guidance
from guidance import gen, select

lm = guidance.models.Transformers("mistralai/Mistral-7B-Instruct-v0.2", echo=False)

@guidance
def analyze_ticket(lm, ticket_text):
    lm += f"""
Ticket: {ticket_text}

Category: {select(["billing", "technical", "general"], name="category")}
Urgency: {select(["low", "medium", "high"], name="urgency")}
Summary: {gen("summary", stop="\\n", max_tokens=50)}
Escalate: {select(["Yes", "No"], name="escalate")}
"""
    return lm

result = analyze_ticket(lm, "I've been waiting 3 weeks for my refund!")
print(result["category"], result["urgency"], result["escalate"])

---

17. Context Engineering: Ordering and Position Effects

Lost-in-the-Middle

Research shows LLMs attend much less to content in the middle of long contexts. Critical information should be positioned at the beginning or end of the context window.

def position_sensitive_context_pack(
    documents:     list[dict],
    question:      str,
    token_budget:  int = 3000,
) -> str:
    """
    Place most relevant docs at beginning and end, less relevant in middle.
    This counteracts the 'lost-in-the-middle' effect.
    """
    if not documents:
        return ""

    # Sort by relevance score
    sorted_docs = sorted(documents, key=lambda d: d.get("score", 0), reverse=True)

    # Budget allocation
    remaining_budget = token_budget
    selected         = []
    for doc in sorted_docs:
        tok_count = count_tokens(doc["text"])
        if remaining_budget - tok_count < 0:
            break
        selected.append(doc)
        remaining_budget -= tok_count

    # Interleave: high-relevance at edges, mid-relevance in middle
    if len(selected) <= 2:
        ordered = selected
    else:
        # Snake pattern: best first, second-best last, rest in middle
        ordered = [selected[0]] + selected[2:] + [selected[1]]

    return "\n\n---\n\n".join(
        f"[Source {i+1}]\n{doc['text']}"
        for i, doc in enumerate(ordered)
    )

Conversation History Compression

def compress_conversation_history(
    messages:        list[dict],
    max_hist_tokens: int = 2000,
    llm = None,
) -> list[dict]:
    """Summarize old conversation turns to stay within token budget."""
    if not messages:
        return []

    # Always keep the system prompt (index 0) and last N turns
    system_msg = [m for m in messages if m["role"] == "system"]
    convo_msgs = [m for m in messages if m["role"] != "system"]

    total_tokens = count_messages_tokens(convo_msgs)
    if total_tokens <= max_hist_tokens:
        return messages   # no compression needed

    # Summarize oldest half of the conversation
    n_to_summarize = len(convo_msgs) // 2
    old_msgs = convo_msgs[:n_to_summarize]
    new_msgs = convo_msgs[n_to_summarize:]

    summary_prompt = (
        "Summarize this conversation excerpt in 3-5 sentences, "
        "preserving key decisions and context:\n\n"
        + "\n".join(f"{m['role'].upper()}: {m['content']}" for m in old_msgs)
    )
    summary = llm.invoke(summary_prompt).content if llm else "[summary unavailable]"

    compressed_history = [{"role": "system", "content": f"Previous conversation summary: {summary}"}]
    return system_msg + compressed_history + new_msgs

---

18. Token Budget Management

class ContextBudgetManager:
    """Manages token budget for a multi-component LLM prompt."""

    def __init__(self, model: str = "gpt-4o", context_limit: int = 128_000):
        self.model          = model
        self.context_limit  = context_limit
        self.safety_margin  = 200    # reserve for metadata and formatting
        self.allocations:  dict[str, int] = {}
        self.used:         dict[str, int] = {}

    def allocate(self, component: str, max_tokens: int):
        self.allocations[component] = max_tokens

    def measure(self, component: str, text: str) -> int:
        n = count_tokens(text, self.model)
        self.used[component] = n
        return n

    def remaining_budget(self) -> int:
        total_allocated = sum(self.allocations.values())
        return self.context_limit - total_allocated - self.safety_margin

    def report(self):
        print(f"\n{'Component':25s} {'Allocated':>10s} {'Used':>10s} {'Utilization':>12s}")
        print("-" * 60)
        for comp in self.allocations:
            alloc = self.allocations[comp]
            used  = self.used.get(comp, 0)
            pct   = used / alloc * 100 if alloc else 0
            print(f"{comp:25s} {alloc:>10,} {used:>10,} {pct:>11.1f}%")
        print(f"{'Remaining':25s} {self.remaining_budget():>10,}")

# Example usage for a RAG system
budget = ContextBudgetManager(model="gpt-4o", context_limit=128_000)
budget.allocate("system_prompt",    512)
budget.allocate("rag_context",    8_192)
budget.allocate("conversation",   2_048)
budget.allocate("output",         1_024)

system  = "You are a helpful AI assistant with expertise in finance."
context = retrieved_context     # from retriever
history = format_history(messages)

budget.measure("system_prompt",  system)
budget.measure("rag_context",    context)
budget.measure("conversation",   history)
budget.report()

---

19. Sliding Window and Long-Context Patterns

def sliding_window_inference(
    document:    str,
    question:    str,
    llm,
    window_size: int = 3000,   # tokens per window
    stride:      int = 1500,   # overlap between windows
) -> str:
    """Process a document larger than the context window using sliding windows."""
    enc          = get_encoder()
    doc_tokens   = enc.encode(document)
    answers      = []
    n_windows    = 0

    for start in range(0, len(doc_tokens), stride):
        window_tokens = doc_tokens[start : start + window_size]
        window_text   = enc.decode(window_tokens)

        prompt = f"""Read the following excerpt and answer the question if the answer is present.
If the excerpt does not contain the answer, respond ONLY with: "NOT_IN_THIS_SECTION"

Excerpt:
{window_text}

Question: {question}

Answer:"""

        answer = llm.invoke(prompt).content.strip()
        if answer != "NOT_IN_THIS_SECTION":
            answers.append(answer)
        n_windows += 1

        if start + window_size >= len(doc_tokens):
            break

    if not answers:
        return "The document does not contain a clear answer to the question."

    if len(answers) == 1:
        return answers[0]

    # Synthesize multiple partial answers
    synthesis_prompt = f"""Combine these partial answers to the question "{question}" into one coherent answer:

{chr(10).join(f'{i+1}. {a}' for i, a in enumerate(answers))}

Synthesized answer:"""

    return llm.invoke(synthesis_prompt).content

---

20. Few-Shot Example Selection

Examples dramatically influence model behavior. Selecting the right ones for each query:

from sentence_transformers import SentenceTransformer
import numpy as np

class DynamicFewShotSelector:
    """Selects the most relevant few-shot examples for each query."""

    def __init__(self, examples: list[dict], embed_model: str = "all-MiniLM-L6-v2"):
        self.examples   = examples
        self.embedder   = SentenceTransformer(embed_model)
        self.embeddings = self.embedder.encode([e["input"] for e in examples])

    def select(self, query: str, n: int = 3) -> list[dict]:
        query_emb   = self.embedder.encode([query])
        similarities = np.dot(self.embeddings, query_emb.T).squeeze()
        top_indices  = np.argsort(similarities)[::-1][:n]
        return [self.examples[i] for i in top_indices]

    def format_examples(self, examples: list[dict]) -> str:
        return "\n\n".join(
            f"Input: {e['input']}\nOutput: {e['output']}"
            for e in examples
        )

# Example usage
examples = [
    {"input": "What is the capital of France?", "output": "Paris"},
    {"input": "Who invented Python?",            "output": "Guido van Rossum"},
    {"input": "What is gradient descent?",       "output": "An optimization algorithm..."},
    # ... hundreds more
]

selector = DynamicFewShotSelector(examples)
relevant = selector.select("Who created the Rust programming language?", n=2)
few_shot_text = selector.format_examples(relevant)

---

21. Common Failure Patterns and Fixes

Failure	Root Cause	Fix
Silent context truncation	Token budget exceeded without warning	Add explicit budget check; log truncation events
Greedy decoding for creative tasks	Wrong decoding profile	Use temperature=0.8, top_p=0.95 for creative tasks
Schema failure on JSON extraction	High temperature causes token deviation	Use temperature=0, constrained decoding
History dilutes recent context	Unbounded conversation append	Compress or summarize old turns
Important evidence lost in middle	Lost-in-the-middle effect	Move critical content to start/end of context
Repetitive outputs	Low diversity, no repetition penalty	Add repetition_penalty=1.1–1.15
Verbose unnecessary output	No max_token constraint	Set appropriate max_tokens per task type

---

System Prompt Engineering Patterns

# Pattern 1: Role + Constraints + Format
ANALYST_PROMPT = """You are a senior financial analyst with 20 years of experience.

CONSTRAINTS:
- Only make claims supported by the provided data
- Flag uncertainty explicitly: "Based on available data..."
- Never project beyond the provided time horizon

OUTPUT FORMAT:
1. Executive Summary (2-3 sentences)
2. Key Findings (bulleted)
3. Risks and Uncertainties
4. Data: {data}"""

# Pattern 2: Few-shot system prompt
FEW_SHOT_SYSTEM = """You classify customer support tickets.

Examples:
Q: "My order hasn't arrived" → billing: false, shipping: true, technical: false
Q: "I can't log into my account" → billing: false, shipping: false, technical: true
Q: "I was charged twice" → billing: true, shipping: false, technical: false

Classify tickets using the same JSON format."""

# Pattern 3: Chain-of-thought forcing
COT_SYSTEM = """You are a math tutor.
When solving problems:
1. State what you know from the problem
2. Identify what formula or method applies
3. Show each calculation step
4. State the final answer clearly
Never skip steps, even for simple problems."""

# Pattern 4: Persona with expertise calibration
EXPERT_PERSONA = """You are Dr. Sarah Chen, a molecular biologist with expertise in CRISPR-Cas9 systems.
- Use precise scientific terminology
- Cite mechanism of action, not just effects
- Acknowledge uncertainty in cutting-edge research
- Recommend peer-reviewed sources for clinical decisions
- Audience: graduate students and lab researchers"""

---

Context Window Comparison (2025)

Model	Context Window	Effective Window	Notes
GPT-4o	128K tokens	~100K practical	Lost-in-middle above 32K
Claude 3.5 Sonnet	200K tokens	~150K practical	Better long-context retention
Gemini 1.5 Pro	1M tokens	~500K practical	Best-in-class long context
Llama-3-70B	128K tokens	~32K practical	Open model
Mistral Large	128K tokens	~32K practical	Open model
Command R+	128K tokens	~64K practical	RAG-optimized

Key insight: Claiming X tokens of context ≠ using X tokens effectively. All models show attention degradation for content beyond 32K tokens. For very long documents, hierarchical or sliding-window approaches still outperform brute-force long-context injection.

---

Tokenization Cheat Sheet by Language

Language	Chars per token	Notes
English	3.5–4.5	Baseline
Spanish/French/German	3.0–4.0	Similar to English
Russian/Ukrainian	1.5–2.5	Cyrillic less efficient
Arabic	1.5–2.5	Right-to-left, morphologically rich
Chinese	1.0–2.0	Characters often map to single tokens
Japanese	1.5–2.5	Mixed scripts
Source code	2.5–3.5	Depends heavily on identifier length
JSON/structured data	2.0–3.0	Keys are often tokenized efficiently
Markdown	3.0–4.0	Headers/emphasis tokens add overhead

Practical rule: When estimating context usage across languages, use 3 chars/token for English and divide by 1.5× for other languages to be safe.

---

Conclusion

Tokenization, decoding, and context engineering are foundational to professional LLM work because they directly control what the model sees and how it speaks. Teams that understand token budgeting, decoding strategy selection, constrained generation, context ordering, and system prompt design build systems that are cheaper, more reliable, and easier to tune. Teams that ignore them usually end up blaming the model for failures caused by their own context construction and decoding choices. These skills — along with awareness of language-specific tokenization costs, context window limitations, and few-shot ordering effects — separate practitioners who can prototype from those who can operate reliable, cost-efficient production systems.

---

Decoding Strategy Quick Reference

Task	temperature	top_p	top_k	Special
JSON extraction	0.0	0.9	5	Constrained decoding
Classification	0.0–0.1	0.9	10	—
Code generation	0.1–0.3	0.95	40	Stop on syntax
Summarization	0.3–0.5	0.92	40	—
Creative writing	0.8–1.2	0.95	0 (off)	Repetition penalty
Brainstorming	1.0–1.3	0.97	0	High diversity
Tool calling	0.0	0.9	5	Constrained decoding

Context Window Usage Guide

# Practical budget allocation for an 8K context window
CONTEXT_BUDGET = {
    "system_prompt":       400,    # 5%
    "few_shot_examples":   800,    # 10%
    "rag_context":       4_000,    # 50%
    "conversation_history": 1_600, # 20%
    "current_query":       400,    # 5%
    "output_reserve":      800,    # 10%
    # Total: 8,000 tokens
}

# Compress when budget exceeded:
# 1. Remove oldest conversation turns first
# 2. Summarize long RAG passages
# 3. Reduce few-shot examples count
# 4. Truncate RAG context from the middle

---

Token Efficiency Techniques

# Technique 1: Compress few-shot examples
# Instead of full examples, use compact format:
COMPACT_EXAMPLES = """
Examples (format: input → output):
positive review → {"sentiment": "positive", "score": 4}
negative review → {"sentiment": "negative", "score": 1}
"""
# Saves ~60% tokens vs full chat-style examples

# Technique 2: Remove redundant context
def deduplicate_chunks(chunks: list[str], threshold: float = 0.90) -> list[str]:
    """Remove near-duplicate retrieved chunks to save tokens."""
    from sentence_transformers import SentenceTransformer
    from sklearn.metrics.pairwise import cosine_similarity
    import numpy as np

    model   = SentenceTransformer("all-MiniLM-L6-v2")
    embeddings = model.encode(chunks)
    kept    = [0]   # always keep first
    for i in range(1, len(chunks)):
        sims = cosine_similarity([embeddings[i]], [embeddings[j] for j in kept])[0]
        if max(sims) < threshold:
            kept.append(i)
    return [chunks[i] for i in kept]

# Technique 3: Summarize old conversation turns
def compress_history(messages: list[dict], max_tokens: int, llm) -> list[dict]:
    system  = [m for m in messages if m["role"] == "system"]
    history = [m for m in messages if m["role"] != "system"]

    from tiktoken import encoding_for_model
    enc = encoding_for_model("gpt-4o")
    current_tokens = sum(len(enc.encode(m["content"])) for m in history)

    if current_tokens <= max_tokens:
        return messages   # nothing to compress

    # Summarize oldest half
    n_summarize = len(history) // 2
    to_summarize = history[:n_summarize]
    text = "\n".join(f"{m['role']}: {m['content']}" for m in to_summarize)
    summary = llm.invoke(f"Summarize concisely (< 100 words):\n{text}").content

    compressed = [{"role": "system", "content": f"Earlier context: {summary}"}]
    return system + compressed + history[n_summarize:]

---

Tokenizer Comparison

Tokenizer	Algorithm	Used By	Vocabulary
tiktoken (cl100k_base)	BPE	GPT-3.5, GPT-4, GPT-4o	100,277 tokens
tiktoken (o200k_base)	BPE	GPT-4o, o1	200,019 tokens
LlamaTokenizer	BPE + SentencePiece	Llama 2, Llama 3	32K / 128K
Mistral tokenizer	BPE + SentencePiece	Mistral, Mixtral	32,000 tokens
T5 tokenizer	SentencePiece (Unigram)	T5, Flan-T5	32,100 tokens
BERT WordPiece	WordPiece	BERT, RoBERTa	30,522 tokens

# Compare tokenization across models
import tiktoken
from transformers import AutoTokenizer

text = "The financial results for Q3 2024 showed a 23.5% increase in revenue."

# tiktoken (GPT-4o)
enc_4o = tiktoken.get_encoding("cl100k_base")
tokens_4o = enc_4o.encode(text)

# Llama 3 tokenizer
llama_tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
tokens_llama = llama_tok.encode(text)

print(f"Text: '{text}'")
print(f"GPT-4o tokens: {len(tokens_4o)}")
print(f"Llama-3 tokens: {len(tokens_llama)}")
# Typically similar: 15-20 tokens for this sentence

---

Advanced Decoding: Contrastive Search

Contrastive search is an alternative decoding strategy that reduces repetition while maintaining coherence, without relying on repetition penalties:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model     = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

input_ids = tokenizer.encode("The future of AI is", return_tensors="pt")

# Contrastive search: balance between probability and diversity
output = model.generate(
    input_ids,
    penalty_alpha=0.6,    # diversity penalty (0 = greedy, 1 = random)
    top_k=4,              # candidate pool size
    max_new_tokens=100,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

# Greedy (repetitive for long sequences):
greedy_out = model.generate(input_ids, max_new_tokens=100, do_sample=False)

# Sampling (diverse but may be incoherent):
sample_out = model.generate(input_ids, max_new_tokens=100, do_sample=True, temperature=0.8)

---

Context Engineering Summary

Effective context engineering involves four decisions:

1. What to include: Select only relevant evidence; exclude noise
2. How much to include: Stay within token budget; leave room for output
3. In what order: Critical evidence first and last; supporting detail in the middle
4. In what format: Clear structure (headers, bullets) helps the model parse; prose is token-efficient

# Production context engineering template
def engineer_context(
    question:    str,
    docs:        list[dict],
    history:     list[dict],
    max_tokens:  int = 6000,
) -> str:
    from tiktoken import encoding_for_model
    enc    = encoding_for_model("gpt-4o")
    budget = max_tokens

    # Reserve for question and output
    budget -= len(enc.encode(question)) + 800

    # Add history (compressed if needed)
    hist_text    = "\n".join(f"{m['role']}: {m['content']}" for m in history[-4:])
    hist_tokens  = len(enc.encode(hist_text))
    budget      -= min(hist_tokens, 800)

    # Pack documents within remaining budget
    packed_docs  = []
    for doc in sorted(docs, key=lambda d: d.get("score", 0), reverse=True):
        tok = len(enc.encode(doc["text"]))
        if tok <= budget:
            packed_docs.append(doc)
            budget -= tok

    # Build context: most relevant first
    context_parts = []
    if hist_text:
        context_parts.append(f"Recent conversation:\n{hist_text}")
    if packed_docs:
        context_parts.append("Relevant documents:\n" + "\n\n".join(
            f"[{i+1}] {d['text']}" for i, d in enumerate(packed_docs)
        ))

    return "\n\n".join(context_parts)

---

Conclusion

Tokenization, decoding, and context engineering are the invisible substrate of every LLM application. Understanding them converts debugging from guesswork into systematic diagnosis: is the model seeing the right tokens? Is the decoding policy appropriate for the task? Is the critical evidence positioned where the model will pay attention to it? These skills compound: a practitioner who has internalized token counting, decoding strategy selection, and context ordering makes fewer architecture mistakes, catches performance problems earlier, and builds systems that are more reliable at lower cost.