Lab Specification — Module FTDD-10: distilabel

python3.11 -m venv ftdd10-env && source ftdd10-env/bin/activate
pip install -q distilabel sentence-transformers transformers torch

import distilabel
print(f"distilabel: {distilabel.__version__}")
from sentence_transformers import SentenceTransformer
print("sentence-transformers: OK")

from distilabel.llms import TransformersLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import MagpieGenerator, LoadDataFromDicts
from distilabel.steps.tasks import TextGeneration

# Use a small instruct model for generation (CPU-friendly)
MODEL_ID = "Qwen/Qwen2.5-0.5B-Instruct"  # or a larger model if you have the hardware

with Pipeline(name="synthetic-sft") as pipeline:
    # Magpie generates diverse self-prompted instructions
    magpie = MagpieGenerator(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 256}),
        n_turns=1,
        num_rows=20,  # keep small for the lab; scale up in production
    )
    # Generate a response to each Magpie instruction
    response = TextGeneration(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 256}),
        input_mappings={"instruction": "instruction"},
    )
    magpie >> response

# Run the pipeline
distiset = pipeline.run()

# Convert to a dataset and inspect
ds = distiset["default"]["train"]
print(f"Generated examples: {len(ds)}")
for i, ex in enumerate(ds.select(range(3))):
    print(f"\n=== Example {i+1} ===")
    print(f"Instruction: {ex.get('instruction', ex.get('prompt', 'N/A'))[:200]}")
    print(f"Response: {str(ex.get('response', ex.get('generation', 'N/A')))[:200]}")

from distilabel.steps.tasks import UltraFeedback  # or a generic judge step

# A judge step scores responses on quality dimensions
# (Exact judge class may vary by distilabel version; UltraFeedback is a common choice)

with Pipeline(name="synthetic-sft-judged") as pipeline:
    magpie = MagpieGenerator(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 256}),
        n_turns=1,
        num_rows=20,
    )
    response = TextGeneration(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 256}),
    )
    # Judge: score the response quality
    judge = UltraFeedback(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 128}),
        aspect="overall-rating",
    )
    magpie >> response >> judge

distiset_judged = pipeline.run()
ds_judged = distiset_judged["default"]["train"]

import numpy as np

# Extract ratings (adapt column name to your judge output)
# UltraFeedback typically emits ratings in a structured field
# Here we simulate the threshold filter pattern:
print(f"Pre-filter count: {len(ds_judged)}")

# In practice, extract the numeric rating and filter:
# keep = ds_judged.filter(lambda ex: ex['ratings'] >= THRESHOLD)
# print(f"Post-filter count: {len(keep)}")

print("\nInspect judge outputs for a few examples:")
for i, ex in enumerate(ds_judged.select(range(3))):
    print(f"\n=== Judged Example {i+1} ===")
    # Print whatever rating/critique fields the judge produced
    keys = list(ex.keys())
    print(f"Fields: {keys}")

from sentence_transformers import SentenceTransformer, util

# Embed the instructions
embedder = SentenceTransformer("all-MiniLM-L6-v2")  # fast, small, CPU-friendly

instructions = [str(ex.get("instruction", ex.get("prompt", ""))) for ex in ds_judged]
embeddings = embedder.encode(instructions, convert_to_tensor=True)

# Compute pairwise cosine similarity
sim_matrix = util.cos_sim(embeddings, embeddings)

# Find near-duplicates (above threshold, excluding self)
THRESHOLD = 0.85
dupes = set()
for i in range(len(instructions)):
    for j in range(i + 1, len(instructions)):
        if sim_matrix[i][j] > THRESHOLD:
            dupes.add(j)  # mark j as a duplicate of i

print(f"Total: {len(instructions)}")
print(f"Near-duplicates removed (sim > {THRESHOLD}): {len(dupes)}")
print(f"Unique examples: {len(instructions) - len(dupes)}")

# Lab Specification — Module FTDD-10: distilabel

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FTDD-10 — distilabel
**Duration**: ~45 minutes
**Environment**: Python 3.11+. CPU works for the full lab at small scale; a GPU (or Colab T4) speeds generation. ~4GB free disk.

---

## Learning objectives

By the end of this lab you will have:

1. **Built a distilabel pipeline** that generates synthetic SFT data via Magpie-style self-prompting — felt the self-prompted-generation technique remove the seed-authoring bottleneck.
2. **Applied an LLM-as-judge filter** to score and threshold the generated candidates — felt the quality gate that sets the dataset ceiling.
3. **Run sentence-transformers dedup** to remove semantic near-duplicates — felt the distinct-signal principle from Module FT06 applied to synthetic data.
4. **Inspected the dataset at each stage** and quantified how many examples the judge and dedup removed — stated, in your own words, why the quality gate is the highest-leverage step.

---

## Phase 0 — Environment setup (5 min)

```bash
python3.11 -m venv ftdd10-env && source ftdd10-env/bin/activate
pip install -q distilabel sentence-transformers transformers torch
```

Verify the install:

```python
import distilabel
print(f"distilabel: {distilabel.__version__}")
from sentence_transformers import SentenceTransformer
print("sentence-transformers: OK")
```

This lab uses a small generation model. If you have a GPU, generation is faster; on CPU, use a 0.5B–1.5B model and a small sample count.

---

## Phase 1 — Generate synthetic SFT data via Magpie-style self-prompting (12 min)

distilabel provides a `MagpieGenerator` step that implements the self-prompting technique. Here we build a minimal pipeline that generates instructions and responses.

```python
from distilabel.llms import TransformersLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import MagpieGenerator, LoadDataFromDicts
from distilabel.steps.tasks import TextGeneration

# Use a small instruct model for generation (CPU-friendly)
MODEL_ID = "Qwen/Qwen2.5-0.5B-Instruct"  # or a larger model if you have the hardware

with Pipeline(name="synthetic-sft") as pipeline:
    # Magpie generates diverse self-prompted instructions
    magpie = MagpieGenerator(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 256}),
        n_turns=1,
        num_rows=20,  # keep small for the lab; scale up in production
    )
    # Generate a response to each Magpie instruction
    response = TextGeneration(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 256}),
        input_mappings={"instruction": "instruction"},
    )
    magpie >> response

# Run the pipeline
distiset = pipeline.run()
```

> **Note**: The exact distilabel step and class names evolve between versions. If the above import path differs in your version, consult `distilabel` docs — the pattern (MagpieGenerator → TextGeneration) is stable. The lab's teaching point is the pipeline structure, not the exact API spelling.

**Inspect the output:**

```python
# Convert to a dataset and inspect
ds = distiset["default"]["train"]
print(f"Generated examples: {len(ds)}")
for i, ex in enumerate(ds.select(range(3))):
    print(f"\n=== Example {i+1} ===")
    print(f"Instruction: {ex.get('instruction', ex.get('prompt', 'N/A'))[:200]}")
    print(f"Response: {str(ex.get('response', ex.get('generation', 'N/A')))[:200]}")
```

**Record**: the number of examples generated and a sample of 3. Note the diversity of the self-prompted instructions — no two should be identical (Magpie's sampling).

> **What just happened:** You generated a diverse instruction set WITHOUT hand-authoring a single seed. The instruct model prompted itself. This is the Magpie bottleneck-removal in action.

---

## Phase 2 — Apply the LLM-as-judge quality gate (12 min)

Add a judge step that scores each (instruction, response) pair, then filter by threshold.

```python
from distilabel.steps.tasks import UltraFeedback  # or a generic judge step

# A judge step scores responses on quality dimensions
# (Exact judge class may vary by distilabel version; UltraFeedback is a common choice)

with Pipeline(name="synthetic-sft-judged") as pipeline:
    magpie = MagpieGenerator(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 256}),
        n_turns=1,
        num_rows=20,
    )
    response = TextGeneration(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 256}),
    )
    # Judge: score the response quality
    judge = UltraFeedback(
        llm=TransformersLLM(model=MODEL_ID, generation_kwargs={"max_new_tokens": 128}),
        aspect="overall-rating",
    )
    magpie >> response >> judge

distiset_judged = pipeline.run()
ds_judged = distiset_judged["default"]["train"]
```

Apply a threshold filter (the exact score column name depends on the judge step's output schema):

```python
import numpy as np

# Extract ratings (adapt column name to your judge output)
# UltraFeedback typically emits ratings in a structured field
# Here we simulate the threshold filter pattern:
print(f"Pre-filter count: {len(ds_judged)}")

# In practice, extract the numeric rating and filter:
# keep = ds_judged.filter(lambda ex: ex['ratings'] >= THRESHOLD)
# print(f"Post-filter count: {len(keep)}")

print("\nInspect judge outputs for a few examples:")
for i, ex in enumerate(ds_judged.select(range(3))):
    print(f"\n=== Judged Example {i+1} ===")
    # Print whatever rating/critique fields the judge produced
    keys = list(ex.keys())
    print(f"Fields: {keys}")
```

**Record**: the pre-filter count and a sense of the judge's scoring. In a real run, apply a numeric threshold and record how many examples it removed. This ratio (kept/generated) is the quality-gate's effect made measurable.

> **What just happened:** An LLM scored each generated response. The threshold you apply to those scores is the single highest-leverage knob — it converts the noisy candidate pool into a curated dataset. Per the thesis, this step sets the ceiling on your trained model.

---

## Phase 3 — Dedup with sentence-transformers (8 min)

Generated data, especially from Magpie, contains semantic near-duplicates. Remove them.

```python
from sentence_transformers import SentenceTransformer, util

# Embed the instructions
embedder = SentenceTransformer("all-MiniLM-L6-v2")  # fast, small, CPU-friendly

instructions = [str(ex.get("instruction", ex.get("prompt", ""))) for ex in ds_judged]
embeddings = embedder.encode(instructions, convert_to_tensor=True)

# Compute pairwise cosine similarity
sim_matrix = util.cos_sim(embeddings, embeddings)

# Find near-duplicates (above threshold, excluding self)
THRESHOLD = 0.85
dupes = set()
for i in range(len(instructions)):
    for j in range(i + 1, len(instructions)):
        if sim_matrix[i][j] > THRESHOLD:
            dupes.add(j)  # mark j as a duplicate of i

print(f"Total: {len(instructions)}")
print(f"Near-duplicates removed (sim > {THRESHOLD}): {len(dupes)}")
print(f"Unique examples: {len(instructions) - len(dupes)}")
```

**Record**: how many near-duplicates were removed. Even in a small 20-example run, you may find a few. At scale (thousands of examples), dedup typically removes a meaningful fraction — this is wasted capacity and bias if left in.

---

## Phase 4 — The quality-gate thesis, in your own words (5 min)

No code. Write 3–5 sentences answering:

1. At each stage (Magpie generation, judge filter, dedup), how many examples did you have? Which stage removed the most?
2. If you had skipped the judge filter and trained on all generated examples, what would the thesis (FT00) predict about the result?
3. For a production pipeline generating 100K examples, which step would you spend the most effort calibrating, and why?

---

## Deliverables

Submit `ftdd10-lab-report.md`:

- [ ] Phase 1: the generated example count and 3 sample instructions (note the self-prompted diversity)
- [ ] Phase 2: the pre-filter count and the judge's scoring on samples (the quality gate)
- [ ] Phase 3: the dedup counts (total, near-duplicates removed, unique remaining)
- [ ] Phase 4: your 3–5 sentence quality-gate thesis

---

## Solution key

- **Phase 1**: the Magpie pipeline generates the requested number of examples (20 in the lab config). The instructions are diverse and self-prompted — none hand-authored. Note that with a 0.5B model, the instructions and responses are simple; with a larger model, quality rises. The teaching point is the *technique* (self-prompting), not the small-model output quality.
- **Phase 2**: the judge scores each example. In a real run, a numeric threshold removes the lowest-scoring fraction. The ratio of kept-to-generated is the quality gate's measurable effect. Even with a small sample, you should see score variation — some responses are clearly better than others.
- **Phase 3**: the sentence-transformers dedup identifies near-duplicates above the similarity threshold. In a 20-example run, 0–3 duplicates is typical. At scale, dedup is more impactful. Each removed duplicate would have wasted training capacity.
- **Phase 4**: a correct thesis names (a) the judge filter typically removes the most examples (quality varies more than duplication in raw generation); (b) skipping the judge means training on noise — per FT00, the model steers into the average of the garbage, and no algorithm recovers; (c) for a 100K production pipeline, the judge calibration is where to spend the most effort, because it sets the dataset ceiling and therefore the trained-model ceiling. Spot-check, calibrate the threshold on a held-out sample, watch for judge biases.

---

## Notes on environment

- **distilabel API evolution**: distilabel's step and class names evolve between versions. The pipeline *structure* (Magpie generate → TextGeneration → judge → filter → dedup) is stable; exact import paths may differ. Consult the distilabel docs for your version if an import fails.
- **Small models for the lab**: a 0.5B model keeps the lab runnable on CPU. The generated quality will be modest — that is expected. The lab teaches the pipeline structure and the quality-gate principle, not the output of a specific model.
- **Scaling up**: in production, use a larger generation model (7B+), scale `num_rows` to thousands+, and run generation on a GPU via vLLM (distilabel's vLLM integration). The pipeline structure is identical; only the backend and scale change.

---

## Stretch goals

1. **Build a preference pipeline.** Adapt the pipeline to generate K=2 responses per prompt (different temperatures), rank them via a judge, and emit prompt/chosen/rejected pairs. Load the result into TRL's `DPOTrainer`. This is the end-to-end DPO-data pipeline from sub-section 10.4.
2. **Add Evol-Instruct.** Insert an Evol-Instruct step after Magpie to stretch the instruction complexity, then re-run the judge. Observe how the difficulty distribution changes and whether the judge filters more evolved examples (higher variance). This validates the "evolution increases quality variance" point.
3. **Compare judge thresholds.** Run the judge with three thresholds (lenient, moderate, strict) and inspect the surviving examples at each. Find the threshold where the kept examples are consistently good without discarding too much. This is the judge-calibration exercise from sub-section 10.3, done empirically.