LLM Training Data, Curation, and Synthetic Data: The Hidden Layer Behind Model Quality

Many discussions about LLM performance focus on architecture and model size. That is understandable, but incomplete. In practice, one of the strongest determinants of model quality is the data pipeline: what data enters training, how it is filtered, how it is balanced, how it is labeled, and how synthetic data is used.

This article looks at the training data layer from an engineering perspective. The goal is to explain why data curation is often the hidden differentiator between average systems and high-quality ones.

1. Data Quality Is Not a Secondary Concern

A large model trained on noisy, duplicated, inconsistent, or badly labeled data will learn those weaknesses at scale. More compute does not magically fix weak data. In many cases, it amplifies it.

This is true across:

• pre-training corpora
• supervised fine-tuning datasets
• preference datasets
• domain adaptation datasets
• synthetic instruction data

2. Pre-Training Data Pipelines Shape the Base Model

For large-scale models, the pre-training pipeline often includes:

• raw collection
• deduplication
• language and domain filtering
• quality scoring
• toxicity or policy filtering
• data mixture balancing

The resulting mixture influences what the model internalizes about style, factuality, code, multilingual behavior, and domain coverage.

3. Deduplication and Contamination Matter

If large datasets are not deduplicated properly, the model can overfit repeated text patterns and benchmarks can become less meaningful. This matters not just for research claims, but for downstream product behavior.

Two different risks appear here:

• training redundancy that wastes capacity
• evaluation contamination that inflates performance estimates

Professional data pipelines treat both as real engineering problems.

4. Fine-Tuning Data Needs a Different Discipline

Supervised fine-tuning data is usually much smaller than pre-training data, but far more behaviorally important. This is where instruction style, refusal boundaries, response structure, and task conventions are taught or reinforced.

What matters most:

• consistent labels
• realistic prompts
• stable answer format
• representative edge cases
• explicit examples of acceptable abstention

Weak SFT data often explains why a model feels polished in demos but unstable in production.

5. Preference Data Is Not Just More Labels

Preference tuning datasets introduce a different problem: the system is no longer learning one answer directly. It is learning which answer is preferred.

That means preference data quality depends on:

• clear ranking criteria
• consistent annotation guidelines
• coverage of ambiguous prompts
• balanced sampling across behaviors
• separation between style preference and truth preference

If preference labels mostly reward tone or verbosity, the model may become more polished without becoming more reliable.

6. Synthetic Data Is Powerful but Dangerous

Synthetic data can accelerate dataset creation and cover long-tail tasks, especially when human labeling is expensive. It can be useful for:

• bootstrapping instruction datasets
• generating diverse task variants
• producing schema-constrained examples
• augmenting low-resource domains

But synthetic data also carries risks:

• error propagation from the generator model
• mode collapse in answer style
• hallucinated labels
• over-sanitized or unrealistic task distributions

Synthetic data should be treated as an accelerator, not as unquestioned truth.

7. A Practical Synthetic Data Workflow

A disciplined workflow often looks like this:

1. define target task schema
2. generate candidate synthetic examples
3. run filtering and validation
4. sample for human review
5. mix with trusted human data
6. evaluate downstream impact before scaling up

The point is to keep synthetic data inside a controlled pipeline rather than letting it silently dominate the training mix.

8. Data Mixture Is an Optimization Lever

Teams often think about training data in terms of raw size. A more useful framing is data mixture. How much of the dataset is:

• code
• domain documentation
• conversational style data
• refusal and safety examples
• task-specific structured examples

Shifting the mixture can often change model behavior more efficiently than scaling the dataset blindly.

9. Minimal Validation Pattern

def validate_example(example):
    if not example.get("instruction"):
        return False
    if not example.get("response"):
        return False
    if len(example["response"]) < 10:
        return False
    return True


def curate_dataset(raw_examples):
    return [e for e in raw_examples if validate_example(e)]

This is intentionally simple, but it illustrates the main point: data curation should be explicit and automated where possible.

10. What to Measure in a Data Pipeline

Useful pipeline indicators include:

• duplicate rate
• invalid example rate
• label disagreement rate
• distribution by task type
• refusal example share
• synthetic versus human data ratio
• downstream eval impact per dataset revision

Without this, data quality work becomes subjective.

11. Common Failure Modes

• using synthetic data without human spot checks
• mixing task formats inconsistently
• overrepresenting easy examples
• forgetting refusal and abstention cases
• assuming more data is always better than cleaner data

12. Pre-Training Data Pipelines in Practice

FineWeb-style Preprocessing

Leading open datasets like FineWeb (HuggingFace) and Dolma (Allen AI) use multi-stage pipelines:

from datasets import load_dataset
from datasketch import MinHash, MinHashLSH
import hashlib
import re
from typing import Generator

# Stage 1: URL-level quality filtering
def is_quality_url(url: str) -> bool:
    low_quality_tlds = {".ru", ".cn", ".tk", ".xyz"}
    spam_keywords    = ["casino", "porn", "phishing", "malware"]
    return (
        not any(url.endswith(tld) for tld in low_quality_tlds)
        and not any(k in url.lower() for k in spam_keywords)
    )

# Stage 2: Text quality scoring
def quality_score(text: str) -> float:
    score = 0.0
    # Penalize very short documents
    if len(text.split()) < 50:
        return 0.0
    # Reward documents with mostly alphabetic content
    alpha_ratio = sum(c.isalpha() for c in text) / max(len(text), 1)
    score += alpha_ratio * 0.4
    # Penalize excessive punctuation or symbols
    punct_ratio = sum(not c.isalnum() for c in text) / max(len(text), 1)
    score -= max(punct_ratio - 0.15, 0) * 0.5
    # Penalize documents with many repeated lines
    lines = text.split("\n")
    unique_ratio = len(set(lines)) / max(len(lines), 1)
    score += unique_ratio * 0.3
    return max(0.0, min(score, 1.0))

# Stage 3: Exact deduplication (SHA-256 hash)
seen_hashes: set[str] = set()

def deduplicate_exact(text: str) -> bool:
    h = hashlib.sha256(text.strip().encode()).hexdigest()
    if h in seen_hashes:
        return False   # duplicate
    seen_hashes.add(h)
    return True        # new document

# Stage 4: Near-duplicate detection with MinHash LSH
def build_minhash(text: str, num_perm: int = 128) -> MinHash:
    m = MinHash(num_perm=num_perm)
    for word in text.lower().split():
        m.update(word.encode("utf8"))
    return m

def near_dedup_pipeline(texts: list[str], threshold: float = 0.85) -> list[str]:
    lsh  = MinHashLSH(threshold=threshold, num_perm=128)
    kept = []
    for i, text in enumerate(texts):
        mh = build_minhash(text)
        if not lsh.query(mh):   # no near-duplicate in LSH
            lsh.insert(str(i), mh)
            kept.append(text)
    return kept

# Full pre-training pipeline
def run_pretrain_pipeline(raw_texts: list[str]) -> list[str]:
    pipeline = []
    for text in raw_texts:
        if quality_score(text) < 0.5:
            continue
        if not deduplicate_exact(text):
            continue
        pipeline.append(text)
    return near_dedup_pipeline(pipeline)

---

13. SFT Dataset Formatting

The format of supervised fine-tuning data is critical. Different models expect different chat templates:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

# Chat template format for Llama 3 instruction tuning
def format_sft_example(instruction: str, response: str) -> str:
    messages = [
        {"role": "system",    "content": "You are a helpful assistant."},
        {"role": "user",      "content": instruction},
        {"role": "assistant", "content": response},
    ]
    return tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=False,
    )

# Alpaca-style format (older models)
ALPACA_TEMPLATE = """### Instruction:
{instruction}

### Input:
{input}

### Response:
{response}"""

def format_alpaca(example: dict) -> str:
    return ALPACA_TEMPLATE.format(
        instruction=example["instruction"],
        input=example.get("input", ""),
        response=example["output"],
    )

# ShareGPT format (multi-turn)
def format_sharegpt(conversation: list[dict]) -> str:
    messages = []
    for turn in conversation:
        role    = "user"      if turn["from"] == "human" else "assistant"
        content = turn["value"]
        messages.append({"role": role, "content": content})
    return tokenizer.apply_chat_template(messages, tokenize=False)

---

14. Deduplication at Scale: MinHash + LSH

For large corpora (billions of documents), exact deduplication is not enough. Near-duplicate paragraphs and shuffled sentences create training redundancy:

from datasketch import MinHash, MinHashLSH
import multiprocessing as mp
from functools import partial

def compute_minhash_shingles(text: str, n: int = 5, num_perm: int = 256) -> MinHash:
    """Character n-gram shingles are more robust than word n-grams."""
    m = MinHash(num_perm=num_perm)
    text_clean = re.sub(r'\s+', ' ', text.lower()).strip()
    for i in range(len(text_clean) - n + 1):
        m.update(text_clean[i:i+n].encode("utf8"))
    return m

def parallel_minhash(texts: list[str], n_workers: int = 8) -> list[MinHash]:
    with mp.Pool(n_workers) as pool:
        return pool.map(compute_minhash_shingles, texts)

def deduplicate_corpus(texts: list[str], threshold: float = 0.80) -> list[int]:
    """Returns indices of documents to keep."""
    lsh     = MinHashLSH(threshold=threshold, num_perm=256)
    keep    = []
    hashes  = parallel_minhash(texts)

    for i, (text, mh) in enumerate(zip(texts, hashes)):
        candidates = lsh.query(mh)
        if not candidates:
            lsh.insert(str(i), mh)
            keep.append(i)
    return keep

# Typical deduplication reduces Common Crawl by 30-50%

---

15. Synthetic Data Generation Pipeline

A production-grade synthetic data pipeline requires multiple stages to maintain quality:

from openai import OpenAI
from pydantic import BaseModel, field_validator
from typing import Optional
import random

client = OpenAI()

class SyntheticExample(BaseModel):
    instruction: str
    response:    str
    quality:     float   # 0.0 to 1.0
    domain:      str

    @field_validator("instruction")
    def instruction_not_empty(cls, v):
        if len(v.strip()) < 10:
            raise ValueError("Instruction too short")
        return v.strip()

    @field_validator("response")
    def response_not_empty(cls, v):
        if len(v.strip()) < 20:
            raise ValueError("Response too short")
        return v.strip()

# Stage 1: Seed instruction generation
SEED_INSTRUCTIONS = [
    "Explain the difference between L1 and L2 regularization.",
    "Write a Python function to merge two sorted arrays.",
    "Summarize the main arguments for and against remote work.",
]

def generate_diverse_instructions(
    seeds: list[str],
    n: int = 50,
    domain: str = "general",
) -> list[str]:
    prompt = f"""
Generate {n} diverse instruction prompts for an AI assistant covering domain: {domain}.
Use these seeds as style examples:
{chr(10).join(f'- {s}' for s in seeds[:3])}

Make prompts:
- Varied in complexity (30% easy, 50% medium, 20% hard)
- Covering different task types (QA, coding, summarization, analysis)
- Realistic and useful

Return one instruction per line. No numbering.
"""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.9,
        max_tokens=2000,
    )
    lines = response.choices[0].message.content.strip().split("\n")
    return [l.strip() for l in lines if len(l.strip()) > 10]

# Stage 2: Response generation
def generate_response(instruction: str, model: str = "gpt-4o") -> str:
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system",  "content": "You are a knowledgeable and helpful assistant. Provide accurate, detailed, and well-structured responses."},
            {"role": "user",    "content": instruction},
        ],
        temperature=0.4,
        max_tokens=1024,
    )
    return response.choices[0].message.content.strip()

# Stage 3: Quality scoring with LLM judge
QUALITY_JUDGE_PROMPT = """
Rate the quality of this AI assistant response on a scale of 1 to 5.

Instruction: {instruction}
Response: {response}

Criteria:
5 - Correct, complete, well-structured, directly addresses the question
4 - Mostly correct with minor gaps
3 - Partially correct or somewhat off-topic
2 - Mostly incorrect or missing key information
1 - Wrong, harmful, or completely off-topic

Return ONLY a JSON object: score
"""

def score_example(instruction: str, response: str) -> dict:
    result = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": QUALITY_JUDGE_PROMPT.format(
                instruction=instruction, response=response
            )}
        ],
        temperature=0,
        response_format={"type": "json_object"},
    )
    import json
    return json.loads(result.choices[0].message.content)

# Stage 4: Full pipeline
def build_synthetic_dataset(
    domain:        str,
    target_size:   int  = 1000,
    quality_cutoff: float = 0.7,  # keep only 4-5 star examples
) -> list[SyntheticExample]:
    instructions = generate_diverse_instructions(SEED_INSTRUCTIONS, n=target_size * 2, domain=domain)
    dataset      = []

    for inst in instructions[:target_size * 2]:
        if len(dataset) >= target_size:
            break
        response = generate_response(inst)
        score    = score_example(inst, response)
        quality  = (score["score"] - 1) / 4.0   # normalize to 0-1

        if quality >= quality_cutoff:
            dataset.append(SyntheticExample(
                instruction=inst,
                response=response,
                quality=quality,
                domain=domain,
            ))

    return dataset

---

16. Preference Dataset Construction (DPO/RLHF)

Preference datasets contain chosen/rejected pairs. Quality of these pairs determines alignment quality:

from dataclasses import dataclass
from typing import Literal

@dataclass
class PreferencePair:
    prompt:   str
    chosen:   str
    rejected: str
    source:   Literal["human", "synthetic", "model_comparison"]

# Method 1: Human labeling (highest quality)
def create_human_preference_pair(
    prompt:    str,
    response_a: str,
    response_b: str,
    human_choice: Literal["A", "B"],
) -> PreferencePair:
    return PreferencePair(
        prompt=prompt,
        chosen=response_a   if human_choice == "A" else response_b,
        rejected=response_b if human_choice == "A" else response_a,
        source="human",
    )

# Method 2: LLM-as-judge for synthetic preference data
def create_synthetic_preference_pair(
    prompt:    str,
    response_a: str,
    response_b: str,
) -> Optional[PreferencePair]:
    judge_prompt = f"""
Compare these two responses to the same prompt. Decide which is better.

Prompt: {prompt}

Response A:
{response_a}

Response B:
{response_b}

Respond with JSON: winner
"""
    result = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": judge_prompt}],
        temperature=0,
        response_format={"type": "json_object"},
    )
    import json
    verdict = json.loads(result.choices[0].message.content)

    if verdict["confidence"] == "low":
        return None   # skip low-confidence pairs

    winner = verdict["winner"]
    return PreferencePair(
        prompt=prompt,
        chosen=response_a   if winner == "A" else response_b,
        rejected=response_b if winner == "A" else response_a,
        source="synthetic",
    )

# Format for DPO training (HuggingFace TRL format)
def format_for_dpo(pair: PreferencePair) -> dict:
    return {
        "prompt":   pair.prompt,
        "chosen":   pair.chosen,
        "rejected": pair.rejected,
    }

---

17. Data Mixture Optimization

Data mixture determines what behavior the model learns. Naive mixing often leads to degraded domain performance:

import numpy as np
from dataclasses import dataclass

@dataclass
class DataSource:
    name:   str
    size:   int       # number of tokens
    weight: float     # sampling probability
    domain: str

# Define your corpus sources
sources = [
    DataSource("web_text",       size=500_000_000_000, weight=0.45, domain="general"),
    DataSource("books",          size=100_000_000_000, weight=0.20, domain="long-form"),
    DataSource("code_github",    size=200_000_000_000, weight=0.20, domain="code"),
    DataSource("scientific",     size=50_000_000_000,  weight=0.08, domain="technical"),
    DataSource("multilingual",   size=80_000_000_000,  weight=0.05, domain="multilingual"),
    DataSource("curated_hq",     size=10_000_000_000,  weight=0.02, domain="quality"),
]

def normalize_weights(sources: list[DataSource]) -> list[DataSource]:
    total = sum(s.weight for s in sources)
    for s in sources:
        s.weight = s.weight / total
    return sources

def compute_effective_epochs(sources: list[DataSource], total_tokens: int) -> dict:
    result = {}
    for s in sources:
        tokens_from_source = total_tokens * s.weight
        result[s.name] = round(tokens_from_source / s.size, 2)
    return result

# Check how many times each source is repeated
epochs = compute_effective_epochs(sources, total_tokens=1_000_000_000_000)  # 1T tokens
for name, ep in epochs.items():
    print(f"{name:20s}: {ep:.2f}× epochs")

# web_text:    0.90× (seen < once — still learning)
# code_github: 2.25× (seen > twice — risk of memorization)
# curated_hq: 45.0× (very high repeat — could overfit format)

Key insight: Upsampling high-quality sources by 5–10× improves performance more than doubling total training tokens. But repetition > 10× risks memorization.

---

18. Evaluation Impact Tracking

Changes to data should be evaluated for downstream impact before scaling:

# Evaluation impact tracker for data pipeline changes
class DataPipelineEvaluator:
    def __init__(self, eval_tasks: list[str]):
        self.eval_tasks = eval_tasks   # e.g. ["mmlu", "hellaswag", "gsm8k"]

    def evaluate_model(self, model_path: str) -> dict:
        scores = {}
        for task in self.eval_tasks:
            # Use lm-evaluation-harness
            import subprocess
            result = subprocess.run(
                ["lm_eval", "--model", "hf",
                 "--model_args", f"pretrained={model_path}",
                 "--tasks", task, "--num_fewshot", "5",
                 "--output_path", f"results/{task}.json"],
                capture_output=True
            )
            scores[task] = self._parse_score(result.stdout, task)
        return scores

    def compare_data_revisions(
        self,
        baseline_model: str,
        candidate_model: str,
    ) -> dict:
        baseline  = self.evaluate_model(baseline_model)
        candidate = self.evaluate_model(candidate_model)

        comparison = {}
        for task in self.eval_tasks:
            delta = candidate.get(task, 0) - baseline.get(task, 0)
            comparison[task] = {
                "baseline":  baseline.get(task),
                "candidate": candidate.get(task),
                "delta":     round(delta, 4),
                "improved":  delta > 0,
            }
        return comparison

---

19. Data Pipeline Observability

import time
import json
from pathlib import Path

class DataPipelineLogger:
    def __init__(self, log_path: str):
        self.log_path = Path(log_path)

    def log_stage(
        self,
        stage:        str,
        input_count:  int,
        output_count: int,
        duration_s:   float,
        metadata:     dict = None,
    ):
        record = {
            "timestamp":    time.strftime("%Y-%m-%dT%H:%M:%SZ"),
            "stage":        stage,
            "input_count":  input_count,
            "output_count": output_count,
            "filtered_pct": round((1 - output_count / max(input_count, 1)) * 100, 2),
            "duration_s":   round(duration_s, 2),
            "throughput":   round(input_count / max(duration_s, 0.001)),
            **(metadata or {}),
        }
        with open(self.log_path, "a") as f:
            f.write(json.dumps(record) + "\n")
        print(f"[{stage}] {input_count:,} → {output_count:,} ({record['filtered_pct']}% filtered) in {record['duration_s']}s")

# Usage
logger = DataPipelineLogger("pipeline.jsonl")
t0 = time.time()
filtered = [d for d in raw_docs if quality_score(d) >= 0.5]
logger.log_stage("quality_filter", len(raw_docs), len(filtered), time.time() - t0)

---

Open Datasets for LLM Training

Pre-training corpora

Dataset	Size	Domain	License
FineWeb	15T tokens	Web (filtered CC)	ODC-By
Dolma	3T tokens	Web, books, code, papers	AI2 ImpACT
RedPajama-V2	30T tokens	Web, multi-source	Apache 2.0
C4	156B tokens	English web text	ODC-By
The Stack v2	900B tokens	Source code (600+ langs)	Various
ROOTS	1.6TB	Multilingual (59 langs)	BigScience RAIL

Fine-tuning datasets

Dataset	Size	Task	Quality
OpenHermes 2.5	1M examples	Instruction following	High (GPT-4 generated)
Alpaca	52K examples	Instruction	Medium (text-davinci-003)
Dolly 15K	15K examples	Instruction	High (human written)
ShareGPT	90K conv	Multi-turn chat	Mixed
UltraChat	1.5M conv	Long multi-turn	High (GPT-3.5)
FLAN v2	20M+ examples	Multi-task	Diverse

Preference datasets

Dataset	Size	Format	Notes
HH-RLHF	169K pairs	Chosen/rejected	Anthropic, human labels
UltraFeedback	64K prompts	Multi-model ranked	GPT-4 judge
Orca DPO Pairs	12K pairs	Chosen/rejected	Microsoft

---

Data Quality Scoring with Fasttext

# Build a fast quality classifier using fastText
# Reference: Kenlm perplexity + fastText is used by FineWeb/Dolma teams

import fasttext
import tempfile, os

def train_quality_classifier(
    high_quality_texts:  list[str],
    low_quality_texts:   list[str],
    model_path:          str = "quality_classifier.bin",
) -> fasttext.FastText._FastText:
    """Train a binary quality classifier."""
    with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f:
        for text in high_quality_texts:
            f.write(f"__label__quality {text[:500]}\n")
        for text in low_quality_texts:
            f.write(f"__label__noise {text[:500]}\n")
        train_file = f.name

    model = fasttext.train_supervised(
        train_file,
        epoch=5,
        lr=0.1,
        dim=64,
        wordNgrams=2,
        minCount=2,
    )
    model.save_model(model_path)
    os.unlink(train_file)
    return model

def score_quality(text: str, model) -> float:
    """Returns quality probability 0-1."""
    pred = model.predict(text.replace("\n", " "), k=1)
    label = pred[0][0]
    prob  = pred[1][0]
    return prob if label == "__label__quality" else 1.0 - prob

# Usage:
# classifier = train_quality_classifier(high_q_texts, low_q_texts)
# score = score_quality("This is a well-written technical explanation...", classifier)
# keep  = score > 0.6  # threshold

---

Tokenization Alignment Check

Before training, ensure your data aligns with the model’s tokenizer:

from transformers import AutoTokenizer
from datasets import Dataset
import numpy as np

def analyze_token_distribution(
    texts:        list[str],
    model_id:     str = "meta-llama/Meta-Llama-3-8B",
    max_seq_len:  int = 2048,
) -> dict:
    tokenizer = AutoTokenizer.from_pretrained(model_id)

    token_counts = []
    truncated    = 0
    for text in texts:
        ids = tokenizer.encode(text, truncation=False)
        n   = len(ids)
        token_counts.append(n)
        if n > max_seq_len:
            truncated += 1

    arr = np.array(token_counts)
    return {
        "total_examples":      len(texts),
        "total_tokens":        int(arr.sum()),
        "mean_tokens":         round(float(arr.mean()), 1),
        "median_tokens":       int(np.median(arr)),
        "p95_tokens":          int(np.percentile(arr, 95)),
        "p99_tokens":          int(np.percentile(arr, 99)),
        "max_tokens":          int(arr.max()),
        "truncation_rate":     round(truncated / len(texts), 4),
        "truncation_count":    truncated,
        "estimated_gb":        round(arr.sum() * 2 / 1e9, 3),  # 2 bytes per token
    }

---

Conclusion

Training data and curation are among the most underappreciated determinants of LLM quality. Strong models are not built only from better architectures or larger compute budgets. They are built from disciplined data pipelines that decide what the model should learn, what it should ignore, and how its behavior should be shaped over time. The pipeline described here — quality scoring, deduplication, format standardization, synthetic generation with judge verification, preference pair construction, and mixture optimization — represents the practices used by teams building state-of-the-art open and closed models. Data engineering is model engineering, and the best practitioners treat data quality with the same rigor they apply to model architecture and training recipes.

---

Data Quality Checklist

## Pre-Training Data Checklist
- [ ] Deduplication applied (exact + near-duplicate)
- [ ] Quality scores computed (perplexity, fastText classifier)
- [ ] Toxic content filtered
- [ ] Language identified and filtered
- [ ] Data mixture ratios defined and documented
- [ ] Tokenization alignment verified (no truncation > 5%)
- [ ] Sample reviewed manually (at least 500 examples)

## SFT Dataset Checklist
- [ ] Instructions diverse (task types, complexity levels)
- [ ] Responses accurate (human or verified model labels)
- [ ] Format consistent (same chat template for all examples)
- [ ] Edge cases included (refusal, ambiguous, hard cases)
- [ ] Validation split separate from training (no leakage)
- [ ] Evaluation impact measured before scaling up

## Preference Dataset Checklist
- [ ] Chosen vs. rejected pairs have meaningful quality gap
- [ ] Labels consistent (same criteria for all pairs)
- [ ] Style bias minimized (preference not just verbosity)
- [ ] Diverse domains and difficulty levels
- [ ] Human or calibrated LLM judge used for labeling
- [ ] Inter-annotator agreement measured (kappa > 0.6)

Data Scaling Laws Reference

For LLM training, the Chinchilla scaling law suggests:

\[N_{tokens} \approx 20 \times N_{params}\]

Where:

• $N_{params}$ = total model parameters
• $N_{tokens}$ = optimal training tokens

Model Size	Optimal Tokens (Chinchilla)	Common Overtraining
1B	20B tokens	100B+ (5×)
7B	140B tokens	1T+ (7×)
13B	260B tokens	1T+ (4×)
70B	1.4T tokens	2T (1.4×)

---

Data Pipeline Code: Full Example

from pathlib import Path
from datasets import Dataset, load_dataset
from transformers import AutoTokenizer
import json, hashlib, re

def run_full_sft_pipeline(
    input_jsonl:    str,
    output_dir:     str,
    model_id:       str = "meta-llama/Meta-Llama-3-8B-Instruct",
    max_seq_length: int = 2048,
    quality_thresh: float = 0.6,
) -> dict:
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    raw_data  = [json.loads(l) for l in Path(input_jsonl).read_text().splitlines()]

    # Step 1: Format
    formatted = []
    seen_hashes = set()
    for item in raw_data:
        inst = item.get("instruction", "").strip()
        resp = item.get("output", "").strip()
        if len(inst) < 10 or len(resp) < 20:
            continue
        h = hashlib.md5((inst + resp).encode()).hexdigest()
        if h in seen_hashes:
            continue
        seen_hashes.add(h)

        text = f"### Instruction:\n{inst}\n\n### Response:\n{resp}"
        tokens = tokenizer.encode(text, truncation=False)
        if len(tokens) > max_seq_length:
            continue   # skip too-long examples

        formatted.append({"instruction": inst, "output": resp, "text": text})

    # Step 2: Train/val split
    import random
    random.shuffle(formatted)
    n_val = max(1, int(len(formatted) * 0.05))
    val_data   = formatted[:n_val]
    train_data = formatted[n_val:]

    # Step 3: Save
    Path(output_dir).mkdir(exist_ok=True)
    Dataset.from_list(train_data).save_to_disk(f"{output_dir}/train")
    Dataset.from_list(val_data).save_to_disk(f"{output_dir}/val")

    return {
        "raw_count":    len(raw_data),
        "train_count":  len(train_data),
        "val_count":    len(val_data),
        "filtered_pct": round((1 - len(formatted) / len(raw_data)) * 100, 1),
    }

---

Training Data Engineering Principles

1. Quality beats quantity: 10K carefully curated examples outperform 100K noisy ones
2. Deduplication is not optional: Near-duplicate content causes memorization and wastes capacity
3. Inspect your data: Random-sample 200 examples manually; trust but verify every automated filtering step
4. Balance your mixture: Overrepresented task types will dominate behavior; track distribution carefully
5. Validate synthetic data: Never use synthetic data without human spot-checking at least 5%
6. Measure downstream impact: Data pipeline changes must be evaluated on an eval set, not just by dataset statistics
7. Separate domains in preference data: Code preferences vs. prose preferences vs. safety preferences require different raters
8. Treat refusal examples as first-class: Models without explicit refusal training will refuse inconsistently or not at all

Data engineering is model engineering. The best model architecture cannot overcome systematically poor training data. Teams that invest in data curation pipelines — deduplication, quality scoring, format standardization, and mixture optimization — build models that are reliably better than those trained on raw uncurated corpora.

---

Training Data Conclusion

The hidden differentiator behind frontier LLMs is not architecture alone—it is the disciplined data pipeline that feeds those architectures. GPT-4, Llama 3, and Claude are distinguished not just by their transformer designs but by the scale, quality, and curation of their training corpora. As a practitioner, the same principle applies at every scale: the time invested in deduplication, quality scoring, format standardization, synthetic data validation, and preference pair construction pays compound returns in model behavior. Data engineering is not a prerequisite to the real work—it is the real work.

Training Data Resources

• FineWeb Dataset — 15T token high-quality web corpus
• Dolma — Open pre-training corpus from AI2
• OpenHermes 2.5 — High-quality instruction dataset
• UltraFeedback — 64K preference dataset with GPT-4 labels
• lm-evaluation-harness — Standard benchmark suite
• DataTrove — HuggingFace’s large-scale data processing pipeline
• Argilla — Open-source annotation platform for SFT data

Training Data: Key Takeaways

The hidden differentiator between average and excellent LLMs is almost always the data pipeline:

• Clean data beats more data
• Deduplication is the single highest-ROI data operation
• Synthetic data accelerates iteration but must be validated
• SFT data quality determines behavioral consistency more than base model size
• Preference data should be collected with clear, consistent criteria
• Mixture optimization is an empirical problem: measure downstream impact, don’t guess

These principles apply whether you are training a 7B model for a niche domain or contributing to a frontier pre-training run. Data engineering is where model capability is decided.

Summary

Data quality is model quality. The pipeline described in this article � URL filtering, quality scoring, MinHash deduplication, SFT formatting, synthetic generation with judge verification, preference pair construction, and mixture optimization � represents the practices used by teams building state-of-the-art models. Start with clean, diverse, well-formatted data. Validate every synthetic example. Measure downstream impact before scaling any data pipeline decision. These habits compound into reliably better model behavior.