LLM Engineering Cheat Sheet: Prompting, RAG, Fine-Tuning, Evaluation, and Guardrails

Posted Dec 12, 2025 Updated Aug 20, 2026

By eya_garci 23 min read

This cheat sheet condenses the main engineering rules across modern LLM systems. It is intended as a practical companion to longer articles on prompting, RAG, fine-tuning, evaluation, observability, inference, and safety.

1. Decoding Parameters

Every API call exposes sampling parameters that directly control output behavior. Using wrong defaults is one of the most common sources of brittle production systems.

Parameter	Range	Effect	Recommended Default
`temperature`	0.0 – 2.0	Randomness of sampling	0 for extraction/code; 0.7–1.0 for creative
`top_p`	0.0 – 1.0	Nucleus sampling cutoff	1.0, or pair with temperature
`top_k`	1 – vocab size	Limits candidate tokens	Not exposed in all APIs
`frequency_penalty`	−2.0 – 2.0	Penalizes already-used tokens	0.1–0.3 to reduce repetition
`presence_penalty`	−2.0 – 2.0	Penalizes any token used at all	0.1–0.5 to encourage variety
`min_p`	0.0 – 1.0	Filter tokens below p_min × p_max	0.05–0.1 (alternative to top_p)
`max_tokens`	1 – ctx limit	Hard output length cap	Always set; prevents runaway cost
`stop`	list of strings	Halt generation on match	Use for structured output delimiters
`seed`	int	Reproducible outputs	Set for eval and regression tests
`logprobs`	bool / int	Return token log-probabilities	Useful for calibration and confidence routing

  
from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Classify sentiment: 'The product is okay.'"}],
    temperature=0.0,          # deterministic
    top_p=1.0,
    frequency_penalty=0.1,
    max_tokens=20,
    stop=["\n", "###"],       # stop on newline or separator
    seed=42,
    logprobs=True,            # include token log-probs in response
)
print(response.choices[0].message.content)
print(response.choices[0].logprobs)

2. Prompting Techniques

Zero-Shot

  
messages = [
    {"role": "system", "content":
        "Classify the sentiment as POSITIVE, NEGATIVE, or NEUTRAL. Return only the label."},
    {"role": "user", "content": "The product works but shipping was terrible."}
]
# → NEGATIVE

Few-Shot

  
messages = [
    {"role": "system",    "content": "Classify ticket priority as P1, P2, or P3."},
    {"role": "user",      "content": "Ticket: Server is down in production"},
    {"role": "assistant", "content": "P1"},
    {"role": "user",      "content": "Ticket: Update the homepage banner image"},
    {"role": "assistant", "content": "P3"},
    {"role": "user",      "content": "Ticket: Login fails for 20% of users"},
]
# → P1

Chain-of-Thought (CoT)

  
system = """Solve problems step by step.
Step 1: Identify the question type.
Step 2: Extract relevant numbers or facts.
Step 3: Reason through each sub-problem.
Step 4: Write the final answer on a new line as "Answer: ..."
"""

ReAct (Reason + Act)

  
system = """You have access to these tools:
- search(query: str) -> str       — search the web
- calculator(expr: str) -> float  — evaluate a math expression

Use this format for every response:
Thought: [what you need to do and why]
Action: tool_name(arguments)
Observation: [tool result — filled in by the system]
... (repeat Thought/Action/Observation as needed)
Thought: I now have enough information.
Final Answer: [your answer]
"""

Structured Output with Pydantic

  
from pydantic import BaseModel, Field
from openai import OpenAI

client = OpenAI()

class InvoiceExtraction(BaseModel):
    vendor:     str        = Field(description="Vendor company name")
    amount:     float      = Field(description="Total invoice amount in EUR")
    date:       str        = Field(description="Invoice date in YYYY-MM-DD format")
    line_items: list[str]  = Field(description="List of billed items")

response = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": f"Extract invoice data:\n\n{raw_text}"}],
    response_format=InvoiceExtraction,
)
invoice = response.choices[0].message.parsed
print(invoice.vendor, invoice.amount, invoice.line_items)

Prompt Template Best Practices

  
# Version prompts like code artifacts
EXTRACTION_V2 = """\
You are a {role}.
Task: {task}

Constraints:
- Output only valid JSON matching the schema below.
- If a field cannot be determined, use null.
- Do not add any explanation outside the JSON.

Schema: {output_schema}

Input:
{user_input}"""

def build_prompt(role: str, task: str, schema: str, user_input: str) -> str:
    assert len(user_input) < 8000, "Input too long for context window"
    return EXTRACTION_V2.format(
        role=role, task=task, output_schema=schema, user_input=user_input
    )

3. RAG — Parameters and Configuration

Chunking

  
from langchain.text_splitter import RecursiveCharacterTextSplitter, TokenTextSplitter

# RecursiveCharacterTextSplitter — recommended default
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,         # 256–1024 chars depending on doc type
    chunk_overlap=64,       # 10–15% of chunk_size
    separators=["\n\n", "\n", ". ", " ", ""],  # split priority
    add_start_index=True,   # adds positional metadata
)

# TokenTextSplitter — when billing by token
token_splitter = TokenTextSplitter(
    chunk_size=256,
    chunk_overlap=32,
    encoding_name="cl100k_base",
)

Embedding Models

Model	Dimensions	Cost	Best For
`text-embedding-3-small`	1536	$	General English, low cost
`text-embedding-3-large`	3072	$$$	Best quality, multilingual
`nomic-embed-text`	768	Free	Open-source, strong on code
`BAAI/bge-m3`	1024	Free	Multilingual, self-hosted

Retrieval Configuration

  
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vs = Chroma.from_documents(chunks, embeddings)

# search_type options
retriever = vs.as_retriever(
    search_type="mmr",              # "similarity" | "mmr" | "similarity_score_threshold"
    search_kwargs={
        "k": 5,                     # final number of docs returned
        "fetch_k": 20,              # mmr: initial candidate pool size
        "lambda_mult": 0.5,         # mmr: 0=max diversity, 1=max relevance
        "score_threshold": 0.7,     # similarity_score_threshold mode only
        "filter": {"dept": "hr"},   # metadata pre-filter
    }
)

# Reranking — apply after initial retrieval to improve precision
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

reranker = CrossEncoderReranker(
    model=HuggingFaceCrossEncoder(
        model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"
    ),
    top_n=3
)

RAG Evaluation Metrics

Stage	Metric	What It Measures
Retrieval	Recall@k	Were relevant docs returned?
Retrieval	MRR / NDCG	Rank quality of results
Generation	Faithfulness	Does answer stay grounded in context?
Generation	Answer relevancy	Does answer address the question?
End-to-end	Exact match / F1	Factual accuracy vs. ground truth

  
# pip install ragas
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

result = evaluate(
    Dataset.from_list([{
        "question":    "What is parental leave duration?",
        "answer":      "16 weeks",
        "contexts":    ["The policy allows 16 weeks of paid leave."],
        "ground_truth": "16 weeks of paid leave"
    }]),
    metrics=[faithfulness, answer_relevancy, context_recall]
)
print(result.to_pandas())

4. Fine-Tuning Hyperparameters

LoRA / QLoRA Configuration

  
from peft import LoraConfig

lora_config = LoraConfig(
    r=16,                   # rank: 4–64 (higher = more capacity, more params)
    lora_alpha=32,          # scaling factor — typically 2×r
    target_modules=[        # which layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",   # attention
        "gate_proj", "up_proj", "down_proj",        # MLP (Llama/Mistral)
    ],
    lora_dropout=0.05,      # 0.0–0.1
    bias="none",            # "none" | "all" | "lora_only"
    task_type="CAUSAL_LM",
)

Training Arguments

  
from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="./checkpoints",
    num_train_epochs=3,                  # 1–5 for most LoRA fine-tunes
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,       # effective batch = 4 × 4 = 16
    learning_rate=2e-4,                  # 1e-4 to 3e-4 for LoRA
    lr_scheduler_type="cosine",          # "cosine" | "linear" | "constant"
    warmup_ratio=0.03,
    fp16=True,                           # or bf16=True on Ampere+ GPUs
    optim="paged_adamw_8bit",            # memory-efficient optimizer for QLoRA
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    report_to="wandb",                   # "none" | "wandb" | "tensorboard"
)

Quantization Options

Method	VRAM Reduction	Quality Loss	Use Case
`load_in_8bit=True` (bitsandbytes)	~50%	Minimal	Inference + fine-tune
`load_in_4bit=True` (QLoRA)	~75%	Small	Training on consumer GPU
GPTQ (post-training)	~75%	Small	Fast inference
AWQ (post-training)	~75%	Minimal	Best quality/speed tradeoff
GGUF + llama.cpp	Configurable	Configurable	CPU and edge inference

  
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",          # "nf4" | "fp4"
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    device_map="auto",
)

5. Evaluation Metrics

  
# pip install rouge-score bert-score evaluate

# ROUGE — text overlap (summarization)
from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
scores = scorer.score(reference, prediction)
print(scores["rougeL"].fmeasure)

# BERTScore — semantic similarity
from bert_score import score as bert_score
P, R, F1 = bert_score([prediction], [reference], lang="en", model_type="deberta-xlarge-mnli")
print(f"BERTScore F1: {F1.mean():.3f}")

# Exact match + F1 for QA
def token_f1(prediction: str, reference: str) -> float:
    pred_tokens = set(prediction.lower().split())
    ref_tokens  = set(reference.lower().split())
    common = pred_tokens & ref_tokens
    if not common:
        return 0.0
    precision = len(common) / len(pred_tokens)
    recall    = len(common) / len(ref_tokens)
    return 2 * precision * recall / (precision + recall)

Metric Selection Guide

Task	Primary Metric	Secondary Metric
Classification	Accuracy, F1	Confusion matrix per class
Summarization	ROUGE-L	BERTScore F1
RAG question answering	Faithfulness (RAGAS)	Answer relevancy, Context recall
Information extraction	Exact match	Field-level F1
Code generation	pass@k	Execution accuracy
Safety / refusal	Refusal rate	False positive refusal rate

6. Inference Optimization

  
# vLLM — high-throughput serving
# pip install vllm
from vllm import LLM, SamplingParams

llm = LLM(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    quantization="awq",              # "gptq" | "awq" | None
    gpu_memory_utilization=0.9,
    max_model_len=8192,
    tensor_parallel_size=2,          # number of GPUs
    enable_prefix_caching=True,      # reuse KV cache for repeated prefixes
)
params = SamplingParams(temperature=0.2, max_tokens=512, top_p=0.9)
outputs = llm.generate(prompts, params)

# Speculative decoding — reduces latency 2–3× on long outputs
llm_speculative = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_model="meta-llama/Llama-3.2-1B-Instruct",
    num_speculative_tokens=5,
)

Optimization Techniques

Technique	Latency Impact	Throughput Impact	Quality Impact
Quantization INT8	↓ 20–40%	↑↑	Negligible
Quantization INT4	↓ 40–60%	↑↑↑	Small
Continuous batching	Neutral	↑↑↑	None
Speculative decoding	↓ 30–50%	Neutral	None
KV cache prefix reuse	↓ on repeat prompts	↑	None
Smaller model routing	↓↓	↑↑	Task-dependent
Context pruning	↓ proportional	↑	Minor
FlashAttention-2	↓ 20–30%	↑	None

7. Guardrails

  
# pip install guardrails-ai
import guardrails as gd
from guardrails.hub import ToxicLanguage, PIIDetection

# Input validation
input_guard = gd.Guard().use_many(
    ToxicLanguage(threshold=0.5, on_fail="exception"),
    PIIDetection(
        pii_entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD"],
        on_fail="fix"       # redact PII automatically
    ),
)
safe_input = input_guard.validate(user_input)

# Output schema validation with auto-reask
from pydantic import BaseModel
output_guard = gd.Guard.from_pydantic(output_class=Article, on_fail="reask")
_, validated, *_ = output_guard(
    llm_api=client.chat.completions.create,
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    num_reasks=2            # re-prompt the LLM up to 2 times on failure
)

# Prompt injection detection
INJECTION_PATTERNS = [
    "ignore previous instructions",
    "disregard your system prompt",
    "you are now",
    "pretend you are",
    "forget everything",
]
def detect_injection(text: str) -> bool:
    return any(p in text.lower() for p in INJECTION_PATTERNS)

`on_fail` Options

Value	Behavior
`"exception"`	Raise `ValidationError` — halt execution
`"fix"`	Auto-correct the value (e.g., redact PII)
`"filter"`	Remove the failing field from output
`"refrain"`	Return `None` for the field
`"reask"`	Re-prompt the LLM with error context
`"noop"`	Log only, no action taken

8. Observability — What to Trace

  
# LangSmith (LangChain ecosystem)
import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"]     = "ls__your_key"
os.environ["LANGCHAIN_PROJECT"]     = "production"

# Langfuse (open-source alternative)
from langfuse.openai import openai   # drop-in replacement
# All openai calls are now automatically traced

# Manual span creation
from langfuse import Langfuse
langfuse = Langfuse()
trace = langfuse.trace(name="rag_pipeline", user_id="user-123")
span  = trace.span(name="retrieval")
span.end(output={"docs_returned": 5, "top_score": 0.87})

Minimum Viable Trace Schema

Field	Why
`prompt_version`	Reproduce and compare across deploys
`model` + `temperature`	Explain non-determinism
`input_tokens` + `output_tokens`	Cost attribution
`retrieval_query` + `retrieved_docs`	Debug RAG failures
`tool_name` + `tool_args` + `tool_result`	Debug agent failures
`parser_success` + `validation_error`	Track schema failure rate
`latency_ms` + `ttft_ms`	SLA monitoring

9. Quick Decision Matrix

Problem	Best First Move	Key Parameter / Tool
Output format unstable	Structured output + schema	`response_format`, `PydanticOutputParser`
Knowledge missing or stale	RAG	`chunk_size`, `k`, reranking
Behavior stable but inconsistent	Fine-tuning (LoRA)	`r`, `lora_alpha`, `num_train_epochs`
System fails unpredictably	Tracing	LangSmith, Langfuse
Cost too high	Quantization + model routing	INT4, smaller model for easy tasks
Latency too high	Streaming + speculative decoding	`stream=True`, vLLM
Unsafe input / data leakage	Input guardrails	Guardrails AI, injection detection
Output violates policy	Output guardrails + reask	Schema validation, `on_fail="reask"`
Repetitive long prompts	LoRA compression	Distill prompt patterns into weights
Multi-step agentic failures	Explicit graph (LangGraph)	`StateGraph`, `interrupt_before`

Output format unstable	Structured output + schema	`response_format`, `PydanticOutputParser`
Knowledge missing or stale	RAG	`chunk_size`, `k`, reranking
Behavior stable but inconsistent	Fine-tuning (LoRA)	`r`, `lora_alpha`, `num_train_epochs`
System fails unpredictably	Tracing	LangSmith, Langfuse
Cost too high	Quantization + model routing	INT4, smaller model for easy tasks
Latency too high	Streaming + speculative decoding	`stream=True`, vLLM
Unsafe input / data leakage	Input guardrails	Guardrails AI, injection detection
Output violates policy	Output guardrails + reask	Schema validation, `on_fail="reask"`
Repetitive long prompts	LoRA compression	Distill prompt patterns into weights
Multi-step agentic failures	Explicit graph (LangGraph)	`StateGraph`, `interrupt_before`

Model Selection Quick Reference

Use Case	Model Tier	Examples	Why
Simple Q&A, classification	Small	gpt-4o-mini, Llama-3-8B	Low cost, fast latency
RAG, multi-step reasoning	Medium	gpt-4o, Llama-3-70B	Good quality/cost balance
Complex coding, math, analysis	Large	o1, o3, Claude Opus	Chain-of-thought reasoning
Local / private data	Open + quantized	Mistral-7B, Mixtral-8x7B GGUF	No data leaves machine
Long document processing	Long-context	Gemini 1.5 Pro (1M), Claude (200k)	Large context window
High-throughput batch	Open + vLLM	Llama-3-70B + AWQ	Cost-efficient serving

Token Budget Reference

Component	Typical Budget	Notes
System prompt	200–800 tokens	Keep concise — repeated every request
RAG context	1,000–6,000 tokens	Depends on chunk size × k
Conversation history	500–2,000 tokens	Compress after ~10 turns
Output	256–2,000 tokens	Set `max_tokens` explicitly
Total target	< 8,000 tokens	Safe for most gpt-4o deployments

  
# Token budget guard
from tiktoken import encoding_for_model

def check_budget(messages: list[dict], max_tokens: int = 8000) -> bool:
    enc   = encoding_for_model("gpt-4o")
    total = sum(len(enc.encode(m["content"])) + 4 for m in messages)
    return total <= max_tokens

RAG Configuration Reference

Parameter	Conservative	Balanced	Aggressive
`chunk_size`	256 tokens	512 tokens	1024 tokens
`chunk_overlap`	32 tokens	64 tokens	128 tokens
`k` (retrieved docs)	3	5	10
Embedding model	`text-embedding-3-small`	`text-embedding-3-large`	`bge-large-en-v1.5`
Reranker	None	MiniLM cross-encoder	Cohere Rerank v3
Hybrid weight (BM25 : dense)	0.5 : 0.5	0.3 : 0.7	0.2 : 0.8
Best for	Low latency, simple queries	Most production use cases	Complex technical domains

Fine-Tuning Quick Reference

Parameter	QLoRA 7B	QLoRA 13B	Full SFT 7B
VRAM required	8–12 GB	16–24 GB	80+ GB
Rank r	8–16	16–32	N/A
lora_alpha	16–32	32–64	N/A
Batch size	2–4	1–2	8–32
Grad accumulation	8–16	16–32	2–4
Learning rate	2e-4	2e-4	2e-5
Epochs	1–3	1–3	1–2
Convergence	~500 steps	~500 steps	~1000 steps

Evaluation Quick Reference

Framework	Best For	Install
RAGAS	RAG faithfulness, relevance, precision	`pip install ragas`
LangSmith	Tracing + experiment tracking	`pip install langsmith`
DeepEval	Comprehensive LLM unit tests	`pip install deepeval`
PromptFoo	Prompt regression testing	`npx promptfoo`
Braintrust	A/B testing, human review	`pip install braintrust`

  
# DeepEval example
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

test_case = LLMTestCase(
    input="What is the return policy?",
    actual_output="You can return within 30 days.",
    retrieval_context=["Returns are accepted within 30 days of purchase."],
)
assert_test(test_case, [AnswerRelevancyMetric(threshold=0.7), FaithfulnessMetric(threshold=0.8)])

Guardrail Stack Reference

Layer	Tool	What It Catches
Input filter	Regex + LlamaGuard	Jailbreaks, NSFW, PII exposure attempts
Rate limiting	Token bucket (Redis)	Abuse, cost spikes
Schema validation	Pydantic + `instructor`	Malformed tool calls, invalid outputs
Permission check	RBAC middleware	Unauthorized tool/data access
Output filter	Policy LLM	Policy violations, data leakage
Audit log	Structured logger	Compliance, forensics

Cost Estimation Reference (2025 pricing)

Model	Input $/1M	Output $/1M	Context
gpt-4o	$2.50	$10.00	128k
gpt-4o-mini	$0.15	$0.60	128k
claude-3-5-sonnet	$3.00	$15.00	200k
claude-3-haiku	$0.25	$1.25	200k
Mistral-7B (self-hosted)	$0.02	$0.02	32k
Llama-3-70B-AWQ (A10G)	~$0.05	~$0.05	8k

  
def monthly_cost_estimate(
    daily_requests: int,
    avg_prompt_tokens: int = 1500,
    avg_completion_tokens: int = 300,
    model: str = "gpt-4o-mini",
) -> float:
    pricing = {
        "gpt-4o":      (2.50e-6, 10.0e-6),
        "gpt-4o-mini": (0.15e-6,  0.60e-6),
        "claude-3-5-sonnet": (3.0e-6, 15.0e-6),
    }
    input_price, output_price = pricing.get(model, (2.50e-6, 10.0e-6))
    per_request = avg_prompt_tokens * input_price + avg_completion_tokens * output_price
    return round(per_request * daily_requests * 30, 2)

print(monthly_cost_estimate(10_000, model="gpt-4o-mini"))   # ~$297/month
print(monthly_cost_estimate(10_000, model="gpt-4o"))        # ~$4,950/month

Latency Targets Reference

Application Type	p50 target	p95 target	Key lever
Live chat assistant	< 1s TTFT	< 2s TTFT	Streaming, small model
Document Q&A	< 2s total	< 5s total	Retrieval caching, prefix cache
Batch analytics	Throughput-first	Throughput-first	Larger batches, async
Agent workflow	< 5s per step	< 15s total	Parallel tool calls, fast draft
Code completion	< 200ms TTFT	< 500ms TTFT	Speculative decoding, quantized

Common Antipatterns Reference

Antipattern	Why It’s Harmful	Correct Approach
Using `temperature=1.0` for extraction	High variance, inconsistent schemas	Use `temperature=0`
Logging raw prompts/responses	PII exposure in logs	Log metadata only; redact content
Single global RAG retriever	Different tasks need different retrievers	Use task-specific retrievers with appropriate k and filters
Trusting model self-assessment	“I’m 95% confident” is meaningless	Use calibrated scoring or rejection sampling
Blind synthetic data	Errors propagate at scale	Human spot-check ≥5% of synthetic examples
Using same model for judge and generator	Self-enhancement bias	Use different model family for judging
Ignoring context ordering	Lost-in-the-middle degradation	Put critical evidence at start and end of context
Appending conversation history forever	Context overflow, diluted evidence	Compress after 10 turns; use summary memory
`except Exception: pass` in agent	Silent failures, hard to debug	Log all exceptions; emit to observability
Hardcoded API keys in code	Security vulnerability	Use environment variables + secret manager

Embedding Model Quick Reference

Model	Dim	Max tokens	Specialty	Cost
text-embedding-3-small	1536	8191	General purpose	$0.02/1M tokens
text-embedding-3-large	3072	8191	Best quality	$0.13/1M tokens
text-embedding-ada-002	1536	8191	Legacy default	$0.10/1M tokens
bge-large-en-v1.5	1024	512	MTEB top open source	Free (local)
nomic-embed-text	768	8192	Long context open	Free (local)
e5-mistral-7b-instruct	4096	32768	Best open quality	Free (local)
multilingual-e5-large	1024	512	Multilingual	Free (local)

Vector Database Quick Reference

DB	Deployment	ANN Index	Filtering	Best For
Chroma	Local / Docker	HNSW	Metadata	Dev, small prod
Qdrant	Docker / Cloud	HNSW	Full payload	Prod, complex filters
Pinecone	Cloud only	Proprietary	Metadata	Managed large-scale
Weaviate	Docker / Cloud	HNSW	GraphQL	Multi-modal
Milvus	Docker / K8s	IVF/HNSW	Scalar	Billion-scale
pgvector	PostgreSQL	HNSW / IVF	SQL	Existing PG stack
Redis	Cloud / self-hosted	HNSW	Hash fields	Low-latency caching

Observability Stack Comparison

Tool	Tracing	Evals	Cost tracking	Self-hostable	Open source
LangSmith	✅	✅	✅	❌	❌
Langfuse	✅	✅	✅	✅	✅
Phoenix (Arize)	✅	✅	❌	✅	✅
W&B Weave	✅	✅	❌	❌	❌
Helicone	✅	❌	✅	✅	✅
Braintrust	✅	✅	❌	❌	❌

Model Provider Comparison

Provider	Top Model	Strengths	Weaknesses
OpenAI	GPT-4o, o3	Best general quality, tool calling	Cost, no self-hosting
Anthropic	Claude 3.5 Sonnet/Opus	Safety, long context, coding	Cost, less ecosystem
Google	Gemini 1.5 Pro/Flash	1M context, multimodal, price	Latency, reliability
Meta (open)	Llama-3.1-405B	Open weights, self-hostable	Requires hardware
Mistral (open)	Mixtral 8x22B	MoE efficiency	Smaller ecosystem
DeepSeek (open)	DeepSeek-V3	Best open quality	Chinese company concerns
Cohere	Command R+	RAG-optimized, grounding	Narrow use case

LLM Engineering Maturity Levels

Level	Capabilities	What You’ve Mastered
L1 Beginner	API calls, basic prompting	Chat completions, temperature
L2 Practitioner	RAG, structured output	Embeddings, Pydantic, basic eval
L3 Engineer	Agents, fine-tuning, evaluation	LangChain/LangGraph, LoRA, RAGAS
L4 Senior	Production systems, observability	Guardrails, tracing, CI/CD evals
L5 Expert	Distributed training, serving infra	FSDP, vLLM, custom architectures

Production LLM Debugging Workflow

User reports: "The assistant gave a wrong answer"

1. Find the request in trace logs (request_id)
2. Check route taken: docs_only / tool_required / sensitive / disallowed?
3. Inspect retrieval:
   - Were relevant docs retrieved? (check doc_ids + scores)
   - Was context truncated? (check token count)
4. Inspect prompt:
   - Which prompt version was used?
   - Was the relevant evidence included?
5. Inspect generation:
   - What did the model output before validation?
   - Did schema validation pass?
6. Root cause:
   - Retrieval miss → fix chunking, embedding, or hybrid weights
   - Prompt issue → update template, improve evidence ordering
   - Model hallucination → add faithfulness constraint, check decoding
   - Tool error → fix tool schema or authorization logic
7. Fix + regression test + deploy with eval gate

LLM Request Lifecycle Checklist

Request Received
  ├─ Rate Limit Check (✓ / reject)
  ├─ Input Safety / Jailbreak Detection (✓ / block)
  ├─ Request Classification (route: docs_only / tool / sensitive / disallowed)
  ├─ Retrieval (if needed): hybrid search + rerank
  ├─ Context Assembly: budget check + ordering
  ├─ LLM Invocation: model call + streaming
  ├─ Output Validation: schema + policy
  ├─ Tool Execution (if needed): authorize + execute + verify
  ├─ Observability: emit trace + cost + quality metrics
  └─ Response Returned

Prompt Engineering Quick Reference

Technique	When to Use	Key Pattern
Zero-shot	Simple, well-defined tasks	Clear instruction + output format
Few-shot	Tasks requiring specific format or style	3-5 high-quality examples in prompt
Chain-of-thought	Reasoning, math, multi-step analysis	“Think step by step” or step-by-step examples
Self-consistency	Critical decisions requiring high confidence	Generate N answers, take majority vote
ReAct	Tool-using agents	Interleave Thought/Action/Observation
Step-back	Complex queries requiring broad context	Rephrase to higher-level question first
Meta-prompting	Dynamic task routing	Have model identify which strategy to use

  
# Chain-of-thought prompt pattern
COT_PROMPT = """Solve this problem step by step.

Problem: {problem}

Let's think through this carefully:
Step 1: [identify what's given]
Step 2: [identify what we need to find]
Step 3: [apply the relevant method]
Step 4: [verify the answer]

Final Answer:"""

# Self-consistency pattern (generate 5, take majority)
def self_consistent_answer(problem: str, llm, n: int = 5) -> str:
    answers = [llm.invoke(COT_PROMPT.format(problem=problem)).content for _ in range(n)]
    # Extract final answers and take majority
    from collections import Counter
    final_answers = [a.split("Final Answer:")[-1].strip() for a in answers]
    return Counter(final_answers).most_common(1)[0][0]

Key Hyperparameter Cheat Sheet

LLM Call Parameters

Parameter	Default	Range	Effect
`temperature`	1.0	0–2	Response diversity (0=deterministic)
`top_p`	1.0	0–1	Nucleus sampling threshold
`top_k`	disabled	1–100	Top-k token sampling
`max_tokens`	model max	1–128K	Output length cap
`presence_penalty`	0	-2–2	Penalize repeated topics
`frequency_penalty`	0	-2–2	Penalize repeated words
`seed`	None	int	Reproducibility (not deterministic guarantee)

RAG Parameters

Parameter	Conservative	Balanced	Aggressive
`chunk_size`	256	512	1024
`chunk_overlap`	32	64	128
`k` (top_k)	3	5	10
Reranker	None	MiniLM	Cohere
BM25 weight	0.5	0.3	0.2

LoRA Fine-Tuning Parameters

Parameter	Small/Fast	Balanced	Full Quality
`r` (rank)	4–8	16	32–64
`lora_alpha`	8–16	32	64–128
`learning_rate`	3e-4	2e-4	1e-4
`epochs`	1	2	3
`batch_size × grad_acc`	64 effective	128	256

Production Engineering Notes

Building production LLM systems requires continuous learning and adaptation. The patterns and tools in this ecosystem evolve rapidly, but certain engineering principles remain constant:

On reliability: The most reliable systems are built with defense in depth — multiple independent validation layers, graceful degradation paths, circuit breakers, and comprehensive observability. No single component should be a single point of failure.

On evaluation: Automated evaluation enables speed; human evaluation provides ground truth. Calibrate your automated judges regularly against human labels. A 10-point quality regression detected in CI costs hours to fix; the same regression discovered after deployment costs days or weeks of user trust.

On iteration: LLM systems improve through measurement, not intuition. Every prompt change, model upgrade, and retrieval configuration should be evaluated against a representative dataset before deployment. Blind experimentation is expensive; instrumented experimentation is invaluable.

On operations: Monitor cost, latency, and quality as a unified picture. A system that is fast and cheap but inaccurate is not production-grade. A system that is accurate but 10× over budget is not sustainable. The engineering goal is acceptable quality at acceptable cost within acceptable latency.

Key Takeaways

Tokens > characters: Always count tokens; costs and context limits are token-based
Hybrid retrieval baseline: BM25 + dense + reranker consistently outperforms pure vector search
Schema first: Define Pydantic models before writing prompts for structured output tasks
Eval is a product: Build your evaluation dataset before the first production deploy
Trace everything: LangSmith from day one; debugging without traces is guesswork
Guardrails are layered: Input moderation + output validation + access control; no single point of failure
Profile before optimizing: Measure TTFT, cost, and quality by component; fix the measured bottleneck
LoRA is the entry point: QLoRA fine-tuning on 8–12 GB VRAM is accessible and effective
Version your prompts: A prompt change is a product change; track it like code
Monitor quality, not just errors: A 10% faithfulness drop is as serious as a 10% error rate

Conclusion

This cheat sheet is a living reference for the full LLM engineering stack: from API parameters and token budgets to RAG configuration, fine-tuning hyperparameters, evaluation frameworks, observability tools, and production architecture patterns. The best LLM engineers develop intuition across all these layers because failures rarely isolate cleanly to one component. A slow system might be failing at retrieval, prompt assembly, or model serving — or all three. Navigating this stack with confidence is what separates practitioners who ship reliable AI from those who ship impressive prototypes. The key parameters, trade-offs, and decision rules in this reference are designed to accelerate that intuition — and the discipline to measure before optimizing is what makes it stick.

LLM

This post is licensed under CC BY 4.0 by the author.