LLM Engineering Cheat Sheet: Prompting, RAG, Fine-Tuning, Evaluation, and Guardrails

This cheat sheet condenses the main engineering rules across modern LLM systems. It is intended as a practical companion to longer articles on prompting, RAG, fine-tuning, evaluation, observability, inference, and safety.

1. Decoding Parameters

Every API call exposes sampling parameters that directly control output behavior. Using wrong defaults is one of the most common sources of brittle production systems.

Parameter	Range	Effect	Recommended Default
temperature	0.0 – 2.0	Randomness of sampling	0 for extraction/code; 0.7–1.0 for creative
top_p	0.0 – 1.0	Nucleus sampling cutoff	1.0, or pair with temperature
top_k	1 – vocab size	Limits candidate tokens	Not exposed in all APIs
frequency_penalty	−2.0 – 2.0	Penalizes already-used tokens	0.1–0.3 to reduce repetition
presence_penalty	−2.0 – 2.0	Penalizes any token used at all	0.1–0.5 to encourage variety
min_p	0.0 – 1.0	Filter tokens below p_min × p_max	0.05–0.1 (alternative to top_p)
max_tokens	1 – ctx limit	Hard output length cap	Always set; prevents runaway cost
stop	list of strings	Halt generation on match	Use for structured output delimiters
seed	int	Reproducible outputs	Set for eval and regression tests
logprobs	bool / int	Return token log-probabilities	Useful for calibration and confidence routing

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Classify sentiment: 'The product is okay.'"}],
    temperature=0.0,          # deterministic
    top_p=1.0,
    frequency_penalty=0.1,
    max_tokens=20,
    stop=["\n", "###"],       # stop on newline or separator
    seed=42,
    logprobs=True,            # include token log-probs in response
)
print(response.choices[0].message.content)
print(response.choices[0].logprobs)

2. Prompting Techniques

Zero-Shot

messages = [
    {"role": "system", "content":
        "Classify the sentiment as POSITIVE, NEGATIVE, or NEUTRAL. Return only the label."},
    {"role": "user", "content": "The product works but shipping was terrible."}
]
# → NEGATIVE

Few-Shot

messages = [
    {"role": "system",    "content": "Classify ticket priority as P1, P2, or P3."},
    {"role": "user",      "content": "Ticket: Server is down in production"},
    {"role": "assistant", "content": "P1"},
    {"role": "user",      "content": "Ticket: Update the homepage banner image"},
    {"role": "assistant", "content": "P3"},
    {"role": "user",      "content": "Ticket: Login fails for 20% of users"},
]
# → P1

Chain-of-Thought (CoT)

system = """Solve problems step by step.
Step 1: Identify the question type.
Step 2: Extract relevant numbers or facts.
Step 3: Reason through each sub-problem.
Step 4: Write the final answer on a new line as "Answer: ..."
"""

ReAct (Reason + Act)

system = """You have access to these tools:
- search(query: str) -> str       — search the web
- calculator(expr: str) -> float  — evaluate a math expression

Use this format for every response:
Thought: [what you need to do and why]
Action: tool_name(arguments)
Observation: [tool result — filled in by the system]
... (repeat Thought/Action/Observation as needed)
Thought: I now have enough information.
Final Answer: [your answer]
"""

Structured Output with Pydantic

from pydantic import BaseModel, Field
from openai import OpenAI

client = OpenAI()

class InvoiceExtraction(BaseModel):
    vendor:     str        = Field(description="Vendor company name")
    amount:     float      = Field(description="Total invoice amount in EUR")
    date:       str        = Field(description="Invoice date in YYYY-MM-DD format")
    line_items: list[str]  = Field(description="List of billed items")

response = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": f"Extract invoice data:\n\n{raw_text}"}],
    response_format=InvoiceExtraction,
)
invoice = response.choices[0].message.parsed
print(invoice.vendor, invoice.amount, invoice.line_items)

Prompt Template Best Practices

# Version prompts like code artifacts
EXTRACTION_V2 = """\
You are a {role}.
Task: {task}

Constraints:
- Output only valid JSON matching the schema below.
- If a field cannot be determined, use null.
- Do not add any explanation outside the JSON.

Schema: {output_schema}

Input:
{user_input}"""

def build_prompt(role: str, task: str, schema: str, user_input: str) -> str:
    assert len(user_input) < 8000, "Input too long for context window"
    return EXTRACTION_V2.format(
        role=role, task=task, output_schema=schema, user_input=user_input
    )

3. RAG — Parameters and Configuration

Chunking

from langchain.text_splitter import RecursiveCharacterTextSplitter, TokenTextSplitter

# RecursiveCharacterTextSplitter — recommended default
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,         # 256–1024 chars depending on doc type
    chunk_overlap=64,       # 10–15% of chunk_size
    separators=["\n\n", "\n", ". ", " ", ""],  # split priority
    add_start_index=True,   # adds positional metadata
)

# TokenTextSplitter — when billing by token
token_splitter = TokenTextSplitter(
    chunk_size=256,
    chunk_overlap=32,
    encoding_name="cl100k_base",
)

Embedding Models

Model	Dimensions	Cost	Best For
text-embedding-3-small	1536	$	General English, low cost
text-embedding-3-large	3072	$$$	Best quality, multilingual
nomic-embed-text	768	Free	Open-source, strong on code
BAAI/bge-m3	1024	Free	Multilingual, self-hosted

Retrieval Configuration

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vs = Chroma.from_documents(chunks, embeddings)

# search_type options
retriever = vs.as_retriever(
    search_type="mmr",              # "similarity" | "mmr" | "similarity_score_threshold"
    search_kwargs={
        "k": 5,                     # final number of docs returned
        "fetch_k": 20,              # mmr: initial candidate pool size
        "lambda_mult": 0.5,         # mmr: 0=max diversity, 1=max relevance
        "score_threshold": 0.7,     # similarity_score_threshold mode only
        "filter": {"dept": "hr"},   # metadata pre-filter
    }
)

# Reranking — apply after initial retrieval to improve precision
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

reranker = CrossEncoderReranker(
    model=HuggingFaceCrossEncoder(
        model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"
    ),
    top_n=3
)

RAG Evaluation Metrics

Stage	Metric	What It Measures
Retrieval	Recall@k	Were relevant docs returned?
Retrieval	MRR / NDCG	Rank quality of results
Generation	Faithfulness	Does answer stay grounded in context?
Generation	Answer relevancy	Does answer address the question?
End-to-end	Exact match / F1	Factual accuracy vs. ground truth

# pip install ragas
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

result = evaluate(
    Dataset.from_list([{
        "question":    "What is parental leave duration?",
        "answer":      "16 weeks",
        "contexts":    ["The policy allows 16 weeks of paid leave."],
        "ground_truth": "16 weeks of paid leave"
    }]),
    metrics=[faithfulness, answer_relevancy, context_recall]
)
print(result.to_pandas())

4. Fine-Tuning Hyperparameters

LoRA / QLoRA Configuration

from peft import LoraConfig

lora_config = LoraConfig(
    r=16,                   # rank: 4–64 (higher = more capacity, more params)
    lora_alpha=32,          # scaling factor — typically 2×r
    target_modules=[        # which layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",   # attention
        "gate_proj", "up_proj", "down_proj",        # MLP (Llama/Mistral)
    ],
    lora_dropout=0.05,      # 0.0–0.1
    bias="none",            # "none" | "all" | "lora_only"
    task_type="CAUSAL_LM",
)

Training Arguments

from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="./checkpoints",
    num_train_epochs=3,                  # 1–5 for most LoRA fine-tunes
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,       # effective batch = 4 × 4 = 16
    learning_rate=2e-4,                  # 1e-4 to 3e-4 for LoRA
    lr_scheduler_type="cosine",          # "cosine" | "linear" | "constant"
    warmup_ratio=0.03,
    fp16=True,                           # or bf16=True on Ampere+ GPUs
    optim="paged_adamw_8bit",            # memory-efficient optimizer for QLoRA
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    report_to="wandb",                   # "none" | "wandb" | "tensorboard"
)

Quantization Options

Method	VRAM Reduction	Quality Loss	Use Case
load_in_8bit=True (bitsandbytes)	~50%	Minimal	Inference + fine-tune
load_in_4bit=True (QLoRA)	~75%	Small	Training on consumer GPU
GPTQ (post-training)	~75%	Small	Fast inference
AWQ (post-training)	~75%	Minimal	Best quality/speed tradeoff
GGUF + llama.cpp	Configurable	Configurable	CPU and edge inference

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",          # "nf4" | "fp4"
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    device_map="auto",
)

5. Evaluation Metrics

# pip install rouge-score bert-score evaluate

# ROUGE — text overlap (summarization)
from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
scores = scorer.score(reference, prediction)
print(scores["rougeL"].fmeasure)

# BERTScore — semantic similarity
from bert_score import score as bert_score
P, R, F1 = bert_score([prediction], [reference], lang="en", model_type="deberta-xlarge-mnli")
print(f"BERTScore F1: {F1.mean():.3f}")

# Exact match + F1 for QA
def token_f1(prediction: str, reference: str) -> float:
    pred_tokens = set(prediction.lower().split())
    ref_tokens  = set(reference.lower().split())
    common = pred_tokens & ref_tokens
    if not common:
        return 0.0
    precision = len(common) / len(pred_tokens)
    recall    = len(common) / len(ref_tokens)
    return 2 * precision * recall / (precision + recall)

Metric Selection Guide

Task	Primary Metric	Secondary Metric
Classification	Accuracy, F1	Confusion matrix per class
Summarization	ROUGE-L	BERTScore F1
RAG question answering	Faithfulness (RAGAS)	Answer relevancy, Context recall
Information extraction	Exact match	Field-level F1
Code generation	pass@k	Execution accuracy
Safety / refusal	Refusal rate	False positive refusal rate

6. Inference Optimization

# vLLM — high-throughput serving
# pip install vllm
from vllm import LLM, SamplingParams

llm = LLM(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    quantization="awq",              # "gptq" | "awq" | None
    gpu_memory_utilization=0.9,
    max_model_len=8192,
    tensor_parallel_size=2,          # number of GPUs
    enable_prefix_caching=True,      # reuse KV cache for repeated prefixes
)
params = SamplingParams(temperature=0.2, max_tokens=512, top_p=0.9)
outputs = llm.generate(prompts, params)

# Speculative decoding — reduces latency 2–3× on long outputs
llm_speculative = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_model="meta-llama/Llama-3.2-1B-Instruct",
    num_speculative_tokens=5,
)

Optimization Techniques

Technique	Latency Impact	Throughput Impact	Quality Impact
Quantization INT8	↓ 20–40%	↑↑	Negligible
Quantization INT4	↓ 40–60%	↑↑↑	Small
Continuous batching	Neutral	↑↑↑	None
Speculative decoding	↓ 30–50%	Neutral	None
KV cache prefix reuse	↓ on repeat prompts	↑	None
Smaller model routing	↓↓	↑↑	Task-dependent
Context pruning	↓ proportional	↑	Minor
FlashAttention-2	↓ 20–30%	↑	None

7. Guardrails

# pip install guardrails-ai
import guardrails as gd
from guardrails.hub import ToxicLanguage, PIIDetection

# Input validation
input_guard = gd.Guard().use_many(
    ToxicLanguage(threshold=0.5, on_fail="exception"),
    PIIDetection(
        pii_entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD"],
        on_fail="fix"       # redact PII automatically
    ),
)
safe_input = input_guard.validate(user_input)

# Output schema validation with auto-reask
from pydantic import BaseModel
output_guard = gd.Guard.from_pydantic(output_class=Article, on_fail="reask")
_, validated, *_ = output_guard(
    llm_api=client.chat.completions.create,
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    num_reasks=2            # re-prompt the LLM up to 2 times on failure
)

# Prompt injection detection
INJECTION_PATTERNS = [
    "ignore previous instructions",
    "disregard your system prompt",
    "you are now",
    "pretend you are",
    "forget everything",
]
def detect_injection(text: str) -> bool:
    return any(p in text.lower() for p in INJECTION_PATTERNS)

on_fail Options

Value	Behavior
"exception"	Raise ValidationError — halt execution
"fix"	Auto-correct the value (e.g., redact PII)
"filter"	Remove the failing field from output
"refrain"	Return None for the field
"reask"	Re-prompt the LLM with error context
"noop"	Log only, no action taken

8. Observability — What to Trace

# LangSmith (LangChain ecosystem)
import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"]     = "ls__your_key"
os.environ["LANGCHAIN_PROJECT"]     = "production"

# Langfuse (open-source alternative)
from langfuse.openai import openai   # drop-in replacement
# All openai calls are now automatically traced

# Manual span creation
from langfuse import Langfuse
langfuse = Langfuse()
trace = langfuse.trace(name="rag_pipeline", user_id="user-123")
span  = trace.span(name="retrieval")
span.end(output={"docs_returned": 5, "top_score": 0.87})

Minimum Viable Trace Schema

Field	Why
prompt_version	Reproduce and compare across deploys
model + temperature	Explain non-determinism
input_tokens + output_tokens	Cost attribution
retrieval_query + retrieved_docs	Debug RAG failures
tool_name + tool_args + tool_result	Debug agent failures
parser_success + validation_error	Track schema failure rate
latency_ms + ttft_ms	SLA monitoring

9. Quick Decision Matrix

Problem	Best First Move	Key Parameter / Tool
Output format unstable	Structured output + schema	response_format, PydanticOutputParser
Knowledge missing or stale	RAG	chunk_size, k, reranking
Behavior stable but inconsistent	Fine-tuning (LoRA)	r, lora_alpha, num_train_epochs
System fails unpredictably	Tracing	LangSmith, Langfuse
Cost too high	Quantization + model routing	INT4, smaller model for easy tasks
Latency too high	Streaming + speculative decoding	stream=True, vLLM
Unsafe input / data leakage	Input guardrails	Guardrails AI, injection detection
Output violates policy	Output guardrails + reask	Schema validation, on_fail="reask"
Repetitive long prompts	LoRA compression	Distill prompt patterns into weights
Multi-step agentic failures	Explicit graph (LangGraph)	StateGraph, interrupt_before

Output format unstable	Structured output + schema	response_format, PydanticOutputParser
Knowledge missing or stale	RAG	chunk_size, k, reranking
Behavior stable but inconsistent	Fine-tuning (LoRA)	r, lora_alpha, num_train_epochs
System fails unpredictably	Tracing	LangSmith, Langfuse
Cost too high	Quantization + model routing	INT4, smaller model for easy tasks
Latency too high	Streaming + speculative decoding	stream=True, vLLM
Unsafe input / data leakage	Input guardrails	Guardrails AI, injection detection
Output violates policy	Output guardrails + reask	Schema validation, on_fail="reask"
Repetitive long prompts	LoRA compression	Distill prompt patterns into weights
Multi-step agentic failures	Explicit graph (LangGraph)	StateGraph, interrupt_before

---

Model Selection Quick Reference

Use Case	Model Tier	Examples	Why
Simple Q&A, classification	Small	gpt-4o-mini, Llama-3-8B	Low cost, fast latency
RAG, multi-step reasoning	Medium	gpt-4o, Llama-3-70B	Good quality/cost balance
Complex coding, math, analysis	Large	o1, o3, Claude Opus	Chain-of-thought reasoning
Local / private data	Open + quantized	Mistral-7B, Mixtral-8x7B GGUF	No data leaves machine
Long document processing	Long-context	Gemini 1.5 Pro (1M), Claude (200k)	Large context window
High-throughput batch	Open + vLLM	Llama-3-70B + AWQ	Cost-efficient serving

---

Token Budget Reference

Component	Typical Budget	Notes
System prompt	200–800 tokens	Keep concise — repeated every request
RAG context	1,000–6,000 tokens	Depends on chunk size × k
Conversation history	500–2,000 tokens	Compress after ~10 turns
Output	256–2,000 tokens	Set max_tokens explicitly
Total target	< 8,000 tokens	Safe for most gpt-4o deployments

# Token budget guard
from tiktoken import encoding_for_model

def check_budget(messages: list[dict], max_tokens: int = 8000) -> bool:
    enc   = encoding_for_model("gpt-4o")
    total = sum(len(enc.encode(m["content"])) + 4 for m in messages)
    return total <= max_tokens

---

RAG Configuration Reference

Parameter	Conservative	Balanced	Aggressive
chunk_size	256 tokens	512 tokens	1024 tokens
chunk_overlap	32 tokens	64 tokens	128 tokens
k (retrieved docs)	3	5	10
Embedding model	text-embedding-3-small	text-embedding-3-large	bge-large-en-v1.5
Reranker	None	MiniLM cross-encoder	Cohere Rerank v3
Hybrid weight (BM25 : dense)	0.5 : 0.5	0.3 : 0.7	0.2 : 0.8
Best for	Low latency, simple queries	Most production use cases	Complex technical domains

---

Fine-Tuning Quick Reference

Parameter	QLoRA 7B	QLoRA 13B	Full SFT 7B
VRAM required	8–12 GB	16–24 GB	80+ GB
Rank r	8–16	16–32	N/A
lora_alpha	16–32	32–64	N/A
Batch size	2–4	1–2	8–32
Grad accumulation	8–16	16–32	2–4
Learning rate	2e-4	2e-4	2e-5
Epochs	1–3	1–3	1–2
Convergence	~500 steps	~500 steps	~1000 steps

---

Evaluation Quick Reference

Framework	Best For	Install
RAGAS	RAG faithfulness, relevance, precision	pip install ragas
LangSmith	Tracing + experiment tracking	pip install langsmith
DeepEval	Comprehensive LLM unit tests	pip install deepeval
PromptFoo	Prompt regression testing	npx promptfoo
Braintrust	A/B testing, human review	pip install braintrust

# DeepEval example
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

test_case = LLMTestCase(
    input="What is the return policy?",
    actual_output="You can return within 30 days.",
    retrieval_context=["Returns are accepted within 30 days of purchase."],
)
assert_test(test_case, [AnswerRelevancyMetric(threshold=0.7), FaithfulnessMetric(threshold=0.8)])

---

Guardrail Stack Reference

Layer	Tool	What It Catches
Input filter	Regex + LlamaGuard	Jailbreaks, NSFW, PII exposure attempts
Rate limiting	Token bucket (Redis)	Abuse, cost spikes
Schema validation	Pydantic + instructor	Malformed tool calls, invalid outputs
Permission check	RBAC middleware	Unauthorized tool/data access
Output filter	Policy LLM	Policy violations, data leakage
Audit log	Structured logger	Compliance, forensics

---

Cost Estimation Reference (2025 pricing)

Model	Input $/1M	Output $/1M	Context
gpt-4o	$2.50	$10.00	128k
gpt-4o-mini	$0.15	$0.60	128k
claude-3-5-sonnet	$3.00	$15.00	200k
claude-3-haiku	$0.25	$1.25	200k
Mistral-7B (self-hosted)	$0.02	$0.02	32k
Llama-3-70B-AWQ (A10G)	~$0.05	~$0.05	8k

def monthly_cost_estimate(
    daily_requests: int,
    avg_prompt_tokens: int = 1500,
    avg_completion_tokens: int = 300,
    model: str = "gpt-4o-mini",
) -> float:
    pricing = {
        "gpt-4o":      (2.50e-6, 10.0e-6),
        "gpt-4o-mini": (0.15e-6,  0.60e-6),
        "claude-3-5-sonnet": (3.0e-6, 15.0e-6),
    }
    input_price, output_price = pricing.get(model, (2.50e-6, 10.0e-6))
    per_request = avg_prompt_tokens * input_price + avg_completion_tokens * output_price
    return round(per_request * daily_requests * 30, 2)

print(monthly_cost_estimate(10_000, model="gpt-4o-mini"))   # ~$297/month
print(monthly_cost_estimate(10_000, model="gpt-4o"))        # ~$4,950/month

---

Latency Targets Reference

Application Type	p50 target	p95 target	Key lever
Live chat assistant	< 1s TTFT	< 2s TTFT	Streaming, small model
Document Q&A	< 2s total	< 5s total	Retrieval caching, prefix cache
Batch analytics	Throughput-first	Throughput-first	Larger batches, async
Agent workflow	< 5s per step	< 15s total	Parallel tool calls, fast draft
Code completion	< 200ms TTFT	< 500ms TTFT	Speculative decoding, quantized

---

Common Antipatterns Reference

Antipattern	Why It’s Harmful	Correct Approach
Using temperature=1.0 for extraction	High variance, inconsistent schemas	Use temperature=0
Logging raw prompts/responses	PII exposure in logs	Log metadata only; redact content
Single global RAG retriever	Different tasks need different retrievers	Use task-specific retrievers with appropriate k and filters
Trusting model self-assessment	“I’m 95% confident” is meaningless	Use calibrated scoring or rejection sampling
Blind synthetic data	Errors propagate at scale	Human spot-check ≥5% of synthetic examples
Using same model for judge and generator	Self-enhancement bias	Use different model family for judging
Ignoring context ordering	Lost-in-the-middle degradation	Put critical evidence at start and end of context
Appending conversation history forever	Context overflow, diluted evidence	Compress after 10 turns; use summary memory
except Exception: pass in agent	Silent failures, hard to debug	Log all exceptions; emit to observability
Hardcoded API keys in code	Security vulnerability	Use environment variables + secret manager

---

Embedding Model Quick Reference

Model	Dim	Max tokens	Specialty	Cost
text-embedding-3-small	1536	8191	General purpose	$0.02/1M tokens
text-embedding-3-large	3072	8191	Best quality	$0.13/1M tokens
text-embedding-ada-002	1536	8191	Legacy default	$0.10/1M tokens
bge-large-en-v1.5	1024	512	MTEB top open source	Free (local)
nomic-embed-text	768	8192	Long context open	Free (local)
e5-mistral-7b-instruct	4096	32768	Best open quality	Free (local)
multilingual-e5-large	1024	512	Multilingual	Free (local)

---

Vector Database Quick Reference

DB	Deployment	ANN Index	Filtering	Best For
Chroma	Local / Docker	HNSW	Metadata	Dev, small prod
Qdrant	Docker / Cloud	HNSW	Full payload	Prod, complex filters
Pinecone	Cloud only	Proprietary	Metadata	Managed large-scale
Weaviate	Docker / Cloud	HNSW	GraphQL	Multi-modal
Milvus	Docker / K8s	IVF/HNSW	Scalar	Billion-scale
pgvector	PostgreSQL	HNSW / IVF	SQL	Existing PG stack
Redis	Cloud / self-hosted	HNSW	Hash fields	Low-latency caching

---

Observability Stack Comparison

Tool	Tracing	Evals	Cost tracking	Self-hostable	Open source
LangSmith	✅	✅	✅	❌	❌
Langfuse	✅	✅	✅	✅	✅
Phoenix (Arize)	✅	✅	❌	✅	✅
W&B Weave	✅	✅	❌	❌	❌
Helicone	✅	❌	✅	✅	✅
Braintrust	✅	✅	❌	❌	❌

---

Conclusion

This cheat sheet condenses the key parameters, thresholds, and decision rules across the main engineering layers of LLM systems. It is intended as a quick reference when tuning decoding, configuring RAG, selecting fine-tuning hyperparameters, or diagnosing production failures. The decision matrix, antipatterns reference, and model/tool comparison tables above provide the practical context needed for fast, confident engineering decisions. The best LLM engineers are distinguished not by knowing more models, but by having sharper intuitions about which lever to pull when quality, cost, or latency degrades — and this cheat sheet is designed to accelerate that intuition.

---

Model Provider Comparison (2025)

Provider	Top Model	Strengths	Weaknesses
OpenAI	GPT-4o, o3	Best general quality, tool calling	Cost, no self-hosting
Anthropic	Claude 3.5 Sonnet/Opus	Safety, long context, coding	Cost, less ecosystem
Google	Gemini 1.5 Pro/Flash	1M context, multimodal, price	Latency, reliability
Meta (open)	Llama-3.1-405B	Open weights, self-hostable	Requires hardware
Mistral (open)	Mixtral 8x22B	MoE efficiency	Smaller ecosystem
DeepSeek (open)	DeepSeek-V3	Best open quality	Chinese company concerns
Cohere	Command R+	RAG-optimized, grounding	Narrow use case

LLM Engineering Maturity Levels

Level	Capabilities	What You’ve Mastered
L1 Beginner	API calls, basic prompting	Chat completions, temperature
L2 Practitioner	RAG, structured output	Embeddings, Pydantic, basic eval
L3 Engineer	Agents, fine-tuning, evaluation	LangChain/LangGraph, LoRA, RAGAS
L4 Senior	Production systems, observability	Guardrails, tracing, CI/CD evals
L5 Expert	Distributed training, serving infra	FSDP, vLLM, custom architectures

---

Production LLM Debugging Workflow

User reports: "The assistant gave a wrong answer"

1. Find the request in trace logs (request_id)
2. Check route taken: docs_only / tool_required / sensitive / disallowed?
3. Inspect retrieval:
   - Were relevant docs retrieved? (check doc_ids + scores)
   - Was context truncated? (check token count)
4. Inspect prompt:
   - Which prompt version was used?
   - Was the relevant evidence included?
5. Inspect generation:
   - What did the model output before validation?
   - Did schema validation pass?
6. Root cause:
   - Retrieval miss → fix chunking, embedding, or hybrid weights
   - Prompt issue → update template, improve evidence ordering
   - Model hallucination → add faithfulness constraint, check decoding
   - Tool error → fix tool schema or authorization logic
7. Fix + regression test + deploy with eval gate

LLM Request Lifecycle Checklist

Request Received
  ├─ Rate Limit Check (✓ / reject)
  ├─ Input Safety / Jailbreak Detection (✓ / block)
  ├─ Request Classification (route: docs_only / tool / sensitive / disallowed)
  ├─ Retrieval (if needed): hybrid search + rerank
  ├─ Context Assembly: budget check + ordering
  ├─ LLM Invocation: model call + streaming
  ├─ Output Validation: schema + policy
  ├─ Tool Execution (if needed): authorize + execute + verify
  ├─ Observability: emit trace + cost + quality metrics
  └─ Response Returned

---

Prompt Engineering Quick Reference

Technique	When to Use	Key Pattern
Zero-shot	Simple, well-defined tasks	Clear instruction + output format
Few-shot	Tasks requiring specific format or style	3-5 high-quality examples in prompt
Chain-of-thought	Reasoning, math, multi-step analysis	“Think step by step” or step-by-step examples
Self-consistency	Critical decisions requiring high confidence	Generate N answers, take majority vote
ReAct	Tool-using agents	Interleave Thought/Action/Observation
Step-back	Complex queries requiring broad context	Rephrase to higher-level question first
Meta-prompting	Dynamic task routing	Have model identify which strategy to use

# Chain-of-thought prompt pattern
COT_PROMPT = """Solve this problem step by step.

Problem: {problem}

Let's think through this carefully:
Step 1: [identify what's given]
Step 2: [identify what we need to find]
Step 3: [apply the relevant method]
Step 4: [verify the answer]

Final Answer:"""

# Self-consistency pattern (generate 5, take majority)
def self_consistent_answer(problem: str, llm, n: int = 5) -> str:
    answers = [llm.invoke(COT_PROMPT.format(problem=problem)).content for _ in range(n)]
    # Extract final answers and take majority
    from collections import Counter
    final_answers = [a.split("Final Answer:")[-1].strip() for a in answers]
    return Counter(final_answers).most_common(1)[0][0]

---

Key Hyperparameter Cheat Sheet

LLM Call Parameters

Parameter	Default	Range	Effect
temperature	1.0	0–2	Response diversity (0=deterministic)
top_p	1.0	0–1	Nucleus sampling threshold
top_k	disabled	1–100	Top-k token sampling
max_tokens	model max	1–128K	Output length cap
presence_penalty	0	-2–2	Penalize repeated topics
frequency_penalty	0	-2–2	Penalize repeated words
seed	None	int	Reproducibility (not deterministic guarantee)

RAG Parameters

Parameter	Conservative	Balanced	Aggressive
chunk_size	256	512	1024
chunk_overlap	32	64	128
k (top_k)	3	5	10
Reranker	None	MiniLM	Cohere
BM25 weight	0.5	0.3	0.2

LoRA Fine-Tuning Parameters

Parameter	Small/Fast	Balanced	Full Quality
r (rank)	4–8	16	32–64
lora_alpha	8–16	32	64–128
learning_rate	3e-4	2e-4	1e-4
epochs	1	2	3
batch_size × grad_acc	64 effective	128	256

---

LLM Engineering Conclusion

This cheat sheet is a living reference for the full LLM engineering stack: from API parameters and token budgets to RAG configuration, fine-tuning hyperparameters, evaluation frameworks, observability tools, and production architecture patterns. The best LLM engineers develop intuition across all these layers because failures rarely isolate cleanly to one component. A slow system might be failing at retrieval, or at prompt assembly, or at model serving, or at all three. A system that gives wrong answers might have a retrieval problem, a context ordering problem, a faithfulness constraint problem, or a decoding problem. Navigating this stack with confidence is what separates practitioners who ship reliable AI from those who ship impressive prototypes.

LLM Engineering Resources

• LangChain / LangGraph \u2014 Orchestration frameworks
• LangSmith \u2014 Tracing, evaluation, dataset management
• vLLM \u2014 Production LLM serving
• RAGAS \u2014 RAG evaluation
• Unsloth \u2014 Efficient local fine-tuning
• HuggingFace PEFT \u2014 LoRA/QLoRA implementation
• DeepEval \u2014 LLM unit testing
• Outlines \u2014 Constrained generation

LLM Engineering: Key Takeaways

The most impactful engineering practices across the full LLM stack:

• Tokens > characters: Always count tokens; costs and context limits are token-based
• Hybrid retrieval baseline: BM25 + dense + reranker consistently outperforms pure vector search
• Schema first: Define Pydantic models before writing prompts for structured output tasks
• Eval is a product: Build your evaluation dataset before the first production deploy
• Trace everything: LangSmith from day one; debugging without traces is guesswork
• Guardrails are layered: Input moderation + output validation + access control; no single point of failure
• Profile before optimizing: Measure TTFT, cost, and quality by component; fix the measured bottleneck
• LoRA is the entry point: QLoRA fine-tuning on 8-12 GB VRAM is accessible and effective
• Version your prompts: A prompt change is a product change; track it like code
• Monitor quality, not just errors: A 10% faithfulness drop is as serious as a 10% error rate

Mastery of the full LLM engineering stack is a journey from API consumer to infrastructure operator. This cheat sheet is designed to accelerate that journey by making the key parameters, trade-offs, and decision rules available in a single reference. The best LLM engineers are distinguished not by knowing more models, but by having sharper intuitions about which lever to pull when quality, cost, or latency degrades � and acting on those intuitions with disciplined measurement.

---

Production Engineering Notes

Building production LLM systems requires continuous learning and adaptation. The patterns and tools in this ecosystem evolve rapidly, but certain engineering principles remain constant:

On reliability: The most reliable systems are built with defense in depth � multiple independent validation layers, graceful degradation paths, circuit breakers, and comprehensive observability. No single component should be a single point of failure.

On evaluation: Automated evaluation enables speed; human evaluation provides ground truth. Calibrate your automated judges regularly against human labels. A 10-point quality regression detected in CI costs hours to fix; the same regression discovered after deployment costs days or weeks of user trust.

On iteration: LLM systems improve through measurement, not intuition. Every prompt change, model upgrade, and retrieval configuration should be evaluated against a representative dataset before deployment. Blind experimentation is expensive; instrumented experimentation is invaluable.

On operations: Monitor cost, latency, and quality as a unified picture. A system that is fast and cheap but inaccurate is not production-grade. A system that is accurate but 10x over budget is not sustainable. The engineering goal is acceptable quality at acceptable cost within acceptable latency.