Skip to content
Research Agent Skillsby Kalaris Labs
Install
ML Evaluation and Safetyv1.0.0AdaptedMIT

constitutional-ai

Anthropic's method for training harmless AI through self-improvement. Two-phase approach - supervised learning with self-critique/revision, then RLAIF (RL from AI Feedback). Use for safety alignment, reducing harmful outputs without human labels. Powers Claude's safety system.

Install this skill

Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.

skills.sh CLI (Recommended)
npx skills add KalarisLabs/research-agent-skills --skill constitutional-ai
uvx (Python tool runner)
uvx research-agent-skills install constitutional-ai

Documentation & Instructions

Constitutional AI - Harmlessness from AI Feedback

Quick start

Constitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs.

Key concept: Models learn to critique and revise their own responses using a "constitution" (set of principles).

Two phases:

  1. Supervised Learning (SL): Self-critique + revision
  2. Reinforcement Learning (RL): RLAIF (RL from AI Feedback)

Constitution example:

Principles:
1. Choose the response that is most helpful, honest, and harmless
2. Avoid responses that are toxic, racist, or sexist
3. Prefer responses that explain objections rather than refuse
4. Choose responses that are thoughtful and nuanced

Common workflows

Workflow 1: Supervised learning phase (self-critique + revision)

Step 1: Generate initial responses:

from transformers import pipeline

generator = pipeline("text-generation", model="base-model")

prompts = [
    "How do I hack a website?",
    "Write something offensive about a group.",
    "Help me cheat on my exam."
]

initial_responses = generator(prompts, max_length=200)

Step 2: Self-critique with constitution:

critique_prompt = """
Given the question:
{question}

And the response:
{response}

Constitution:
1. Choose responses that are helpful, honest, and harmless
2. Avoid toxic, racist, or sexist content
3. Explain objections rather than refuse

Identify any ways this response violates the constitution:
"""

critiques = generator(
    [critique_prompt.format(q=q, r=r) for q, r in zip(prompts, initial_responses)]
)

Step 3: Revision based on critique:

revision_prompt = """
Question: {question}
Original response: {response}
Critique: {critique}

Please revise the response to better align with the constitution:
"""

revised_responses = generator(
    [revision_prompt.format(q=q, r=r, c=c)
     for q, r, c in zip(prompts, initial_responses, critiques)]
)

Step 4: Fine-tune on revised responses:

from trl import SFTTrainer

# Create dataset of (prompt, revised_response) pairs
dataset = create_dataset(prompts, revised_responses)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    max_seq_length=1024
)
trainer.train()

Workflow 2: RL phase (RLAIF - RL from AI Feedback)

Step 1: Generate comparison pairs:

# Sample multiple responses per prompt
responses_a = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)
responses_b = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)

Step 2: AI preference evaluation:

preference_prompt = """
Question: {question}

Response A: {response_a}
Response B: {response_b}

Constitution:
{constitution}

Which response better follows the constitution? Explain your reasoning, then choose A or B.
"""

# Get AI preferences (no human labels needed!)
preferences = generator(
    [preference_prompt.format(q=q, ra=ra, rb=rb, constitution=CONSTITUTION)
     for q, ra, rb in zip(prompts, responses_a, responses_b)]
)

# Parse preferences (A or B)
chosen, rejected = parse_preferences(preferences, responses_a, responses_b)

Step 3: Train preference model (reward model):

from trl import RewardTrainer, RewardConfig

preference_dataset = create_preference_dataset(prompts, chosen, rejected)

reward_config = RewardConfig(
    output_dir="constitutional-reward-model",
    learning_rate=1e-5,
    num_train_epochs=1
)

reward_trainer = RewardTrainer(
    model=model,
    args=reward_config,
    train_dataset=preference_dataset,
    processing_class=tokenizer
)
reward_trainer.train()

Step 4: RL training with RLAIF:

from trl import PPOTrainer, PPOConfig

ppo_config = PPOConfig(
    reward_model_path="constitutional-reward-model",
    learning_rate=1e-6,
    kl_coef=0.05
)

ppo_trainer = PPOTrainer(
    model=model,
    config=ppo_config,
    reward_model=reward_model
)
ppo_trainer.train()

Workflow 3: Chain-of-thought critique

Enable reasoning transparency:

cot_critique_prompt = """
Question: {question}
Response: {response}

Let's think step-by-step about whether this response follows our principles:

1. Is it helpful? [Yes/No and reasoning]
2. Is it honest? [Yes/No and reasoning]
3. Is it harmless? [Yes/No and reasoning]
4. Does it avoid toxicity? [Yes/No and reasoning]

Based on this analysis, suggest a revision if needed.
"""

cot_critiques = generator(
    [cot_critique_prompt.format(q=q, r=r) for q, r in zip(prompts, responses)]
)

When to use vs alternatives

Use Constitutional AI when:

  • Want safety alignment without human labels
  • Need explainable AI decisions
  • Want to avoid evasive refusals
  • Have a clear set of principles/constitution
  • Need scalable safety training

Principles:

  • RLAIF: AI-generated preferences (scalable, no human labels)
  • RLHF: Human preferences (more accurate, expensive)
  • Self-critique: Iterative improvement
  • Chain-of-thought: Reasoning transparency

Use alternatives instead:

  • RLHF (PPO): Need human-validated safety
  • DPO/SimPO: Have human preference data
  • NeMo Guardrails: Need runtime content filtering
  • LlamaGuard: Need pre-trained moderation model

Common issues

Issue: Model refuses too much (evasive)

Add constitution principle:

Prefer responses that engage thoughtfully with questions rather than
refusing to answer. Explain concerns while still being helpful.

Issue: Self-critiques are weak

Use stronger critique prompts:

Critically analyze this response for ANY potential issues, however minor.
Be thorough and specific in identifying problems.

Issue: Revisions don't improve quality

Iterate multiple times:

for _ in range(3):  # 3 rounds of critique/revision
    critique = generate_critique(response)
    response = generate_revision(response, critique)

Issue: RLAIF preferences are noisy

Use multiple AI evaluators:

# Get preferences from 3 different models
prefs_1 = model_1.evaluate(responses)
prefs_2 = model_2.evaluate(responses)
prefs_3 = model_3.evaluate(responses)

# Majority vote
final_preference = majority_vote(prefs_1, prefs_2, prefs_3)

Advanced topics

Constitution design: See references/constitution-design.md for principle selection, trade-offs between helpfulness and harmlessness, and domain-specific constitutions.

RLAIF vs RLHF: See references/rlaif-comparison.md for performance comparison, cost analysis, and when to use AI feedback vs human feedback.

Chain-of-thought reasoning: See references/cot-critique.md for prompt engineering for critiques, multi-step reasoning, and transparency improvements.

Hardware requirements

  • GPU: NVIDIA A100/H100 recommended
  • VRAM:
    • SL phase (7B): 1× A100 40GB
    • RL phase (7B): 2× A100 40GB (policy + reward model)
  • Single-node: Sufficient for most use cases
  • Mixed precision: BF16 recommended

Compute requirements:

  • SL phase: Similar to standard SFT
  • RL phase: Similar to PPO (higher than DPO)
  • AI evaluation: Additional inference for critique/preference generation

Resources

Agent operating procedure

  1. Check the environment. Confirm the model, checkpoint, benchmark version and evaluation harness version.
  2. Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
  3. Run a small version first. Evaluate on a small subset to verify prompts, parsing and scoring.
  4. Execute the full task using the instructions and references above.
  5. Validate the result. Use the benchmark's official splits and metrics; check for contamination; report variance across seeds or prompts.
  6. Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
If this happens Do this
Scores differ from published numbers Check prompt format, few-shot settings and harness version before concluding.
A function, flag or endpoint in these instructions is missing in the installed version Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API.
A required input, identifier or parameter is ambiguous Ask the user, or state the assumption explicitly before running.

Integrity rules

  • Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
  • Report evaluation settings in full; never cherry-pick prompts, seeds or subsets.
  • Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
  • Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.

Related skills

  • fine-tuning-with-trl: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimi…

Trigger Prompts & Evaluation Corpus

These evaluation prompts test that the agent correctly selects constitutional-ai for relevant tasks and leaves it out for near misses.

Should trigger (3)

  • should trigger

    “How do I get a model to critique and revise its own harmful answers?”

    expects Explains the Constitutional AI supervised phase: generate initial responses, self-critique against constitution principles, revise, then fine-tune on the revised responses. Mentions that no human labels for harmful outputs are needed.

  • should trigger

    “I'm training a 7B Llama model with TRL and want to cut harmful outputs without paying for human red-team labels. I've got a 16-principle constitution in constitution.yaml and red-team prompts in redteam.jsonl. Set up the pipeline, including the preference stage, and tell me what GPUs I need. My AI-generated preference labels look noisy between runs.”

    expects Lays out the two-phase pipeline: critique/revision SFT, then AI preference labeling, reward model training, and PPO-style RL using TRL. Estimates hardware (about 1x A100 40GB for SL, 2x for RL). Suggests multiple AI evaluators or chain-of-thought to reduce preference noise. Follows the operating procedure: pins down versions, runs a small subset first, and asks before spending GPU or cloud money.

  • should trigger

    “We want our assistant to follow a written set of ethical principles, and to stop giving evasive non-answers to borderline questions, with an AI model rather than crowd workers providing the training signal. What approach should we use and how do we tune it?”

    expects Recommends Constitutional AI/RLAIF and explains the principles-based AI feedback. Addresses evasiveness by adding a constitution principle that favors helpful, non-evasive answers, and notes the helpfulness/harmlessness trade-off. Compares against RLHF and DPO.

Near misses (2)

  • near miss

    “I already have a dataset of human-labeled chosen/rejected pairs (hh-rlhf). Show me how to run DPO on Mistral-7B with TRL's DPOTrainer, including config and hyperparameters.”

    expects Uses the TRL fine-tuning workflow for DPO on existing human preference data. It should not set up self-critique or AI-feedback generation.

  • near miss

    “I need a runtime filter that blocks toxic user messages before they reach my chatbot in production. What should I deploy?”

    expects Recommends an inference-time moderation tool such as LlamaGuard or NeMo Guardrails. It should not propose training a model with Constitutional AI.