phoenix-observability
Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.
Install this skill
Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.
npx skills add KalarisLabs/research-agent-skills --skill phoenix-observabilityuvx research-agent-skills install phoenix-observabilityDocumentation & Instructions
Phoenix - AI Observability Platform
Open-source AI observability and evaluation platform for LLM applications with tracing, evaluation, datasets, experiments, and real-time monitoring.
When to use Phoenix
Use Phoenix when:
- Debugging LLM application issues with detailed traces
- Running systematic evaluations on datasets
- Monitoring production LLM systems in real-time
- Building experiment pipelines for prompt/model comparison
- Self-hosted observability without vendor lock-in
Key features:
- Tracing: OpenTelemetry-based trace collection for any LLM framework
- Evaluation: LLM-as-judge evaluators for quality assessment
- Datasets: Versioned test sets for regression testing
- Experiments: Compare prompts, models, and configurations
- Playground: Interactive prompt testing with multiple models
- Open-source: Self-hosted with PostgreSQL or SQLite
Use alternatives instead:
- LangSmith: Managed platform with LangChain-first integration
- Weights & Biases: Deep learning experiment tracking focus
- Arize Cloud: Managed Phoenix with enterprise features
- MLflow: General ML lifecycle, model registry focus
Quick start
Installation
pip install arize-phoenix
# With specific backends
pip install arize-phoenix[embeddings] # Embedding analysis
pip install arize-phoenix-otel # OpenTelemetry config
pip install arize-phoenix-evals # Evaluation framework
pip install arize-phoenix-client # Lightweight REST client
Launch Phoenix server
import phoenix as px
# Launch in notebook (ThreadServer mode)
session = px.launch_app()
# View UI
session.view() # Embedded iframe
print(session.url) # http://localhost:6006
Command-line server (production)
# Start Phoenix server
phoenix serve
# With PostgreSQL
export PHOENIX_SQL_DATABASE_URL="postgresql://user:pass@host/db"
phoenix serve --port 6006
Basic tracing
from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor
# Configure OpenTelemetry with Phoenix
tracer_provider = register(
project_name="my-llm-app",
endpoint="http://localhost:6006/v1/traces"
)
# Instrument OpenAI SDK
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
# All OpenAI calls are now traced
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
Core concepts
Traces and spans
A trace represents a complete execution flow, while spans are individual operations within that trace.
from phoenix.otel import register
from opentelemetry import trace
# Setup tracing
tracer_provider = register(project_name="my-app")
tracer = trace.get_tracer(__name__)
# Create custom spans
with tracer.start_as_current_span("process_query") as span:
span.set_attribute("input.value", query)
# Child spans are automatically nested
with tracer.start_as_current_span("retrieve_context"):
context = retriever.search(query)
with tracer.start_as_current_span("generate_response"):
response = llm.generate(query, context)
span.set_attribute("output.value", response)
Projects
Projects organize related traces:
import os
os.environ["PHOENIX_PROJECT_NAME"] = "production-chatbot"
# Or per-trace
from phoenix.otel import register
tracer_provider = register(project_name="experiment-v2")
Framework instrumentation
OpenAI
from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor
tracer_provider = register()
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
LangChain
from phoenix.otel import register
from openinference.instrumentation.langchain import LangChainInstrumentor
tracer_provider = register()
LangChainInstrumentor().instrument(tracer_provider=tracer_provider)
# All LangChain operations traced
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o")
response = llm.invoke("Hello!")
LlamaIndex
from phoenix.otel import register
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
tracer_provider = register()
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)
Anthropic
from phoenix.otel import register
from openinference.instrumentation.anthropic import AnthropicInstrumentor
tracer_provider = register()
AnthropicInstrumentor().instrument(tracer_provider=tracer_provider)
Evaluation framework
Built-in evaluators
from phoenix.evals import (
OpenAIModel,
HallucinationEvaluator,
RelevanceEvaluator,
ToxicityEvaluator,
llm_classify
)
# Setup model for evaluation
eval_model = OpenAIModel(model="gpt-4o")
# Evaluate hallucination
hallucination_eval = HallucinationEvaluator(eval_model)
results = hallucination_eval.evaluate(
input="What is the capital of France?",
output="The capital of France is Paris.",
reference="Paris is the capital of France."
)
Custom evaluators
from phoenix.evals import llm_classify
# Define custom evaluation
def evaluate_helpfulness(input_text, output_text):
template = """
Evaluate if the response is helpful for the given question.
Question: {input}
Response: {output}
Is this response helpful? Answer 'helpful' or 'not_helpful'.
"""
result = llm_classify(
model=eval_model,
template=template,
input=input_text,
output=output_text,
rails=["helpful", "not_helpful"]
)
return result
Run evaluations on dataset
from phoenix import Client
from phoenix.evals import run_evals
client = Client()
# Get spans to evaluate
spans_df = client.get_spans_dataframe(
project_name="my-app",
filter_condition="span_kind == 'LLM'"
)
# Run evaluations
eval_results = run_evals(
dataframe=spans_df,
evaluators=[
HallucinationEvaluator(eval_model),
RelevanceEvaluator(eval_model)
],
provide_explanation=True
)
# Log results back to Phoenix
client.log_evaluations(eval_results)
Datasets and experiments
Create dataset
from phoenix import Client
client = Client()
# Create dataset
dataset = client.create_dataset(
name="qa-test-set",
description="QA evaluation dataset"
)
# Add examples
client.add_examples_to_dataset(
dataset_name="qa-test-set",
examples=[
{
"input": {"question": "What is Python?"},
"output": {"answer": "A programming language"}
},
{
"input": {"question": "What is ML?"},
"output": {"answer": "Machine learning"}
}
]
)
Run experiment
from phoenix import Client
from phoenix.experiments import run_experiment
client = Client()
def my_model(input_data):
"""Your model function."""
question = input_data["question"]
return {"answer": generate_answer(question)}
def accuracy_evaluator(input_data, output, expected):
"""Custom evaluator."""
return {
"score": 1.0 if expected["answer"].lower() in output["answer"].lower() else 0.0,
"label": "correct" if expected["answer"].lower() in output["answer"].lower() else "incorrect"
}
# Run experiment
results = run_experiment(
dataset_name="qa-test-set",
task=my_model,
evaluators=[accuracy_evaluator],
experiment_name="baseline-v1"
)
print(f"Average accuracy: {results.aggregate_metrics['accuracy']}")
Client API
Query traces and spans
from phoenix import Client
client = Client(endpoint="http://localhost:6006")
# Get spans as DataFrame
spans_df = client.get_spans_dataframe(
project_name="my-app",
filter_condition="span_kind == 'LLM'",
limit=1000
)
# Get specific span
span = client.get_span(span_id="abc123")
# Get trace
trace = client.get_trace(trace_id="xyz789")
Log feedback
from phoenix import Client
client = Client()
# Log user feedback
client.log_annotation(
span_id="abc123",
name="user_rating",
annotator_kind="HUMAN",
score=0.8,
label="helpful",
metadata={"comment": "Good response"}
)
Export data
# Export to pandas
df = client.get_spans_dataframe(project_name="my-app")
# Export traces
traces = client.list_traces(project_name="my-app")
Production deployment
Docker
docker run -p 6006:6006 arizephoenix/phoenix:latest
With PostgreSQL
# Set database URL
export PHOENIX_SQL_DATABASE_URL="postgresql://user:pass@host:5432/phoenix"
# Start server
phoenix serve --host 0.0.0.0 --port 6006
Environment variables
| Variable | Description | Default |
|---|---|---|
PHOENIX_PORT |
HTTP server port | 6006 |
PHOENIX_HOST |
Server bind address | 127.0.0.1 |
PHOENIX_GRPC_PORT |
gRPC/OTLP port | 4317 |
PHOENIX_SQL_DATABASE_URL |
Database connection | SQLite temp |
PHOENIX_WORKING_DIR |
Data storage directory | OS temp |
PHOENIX_ENABLE_AUTH |
Enable authentication | false |
PHOENIX_SECRET |
JWT signing secret | Required if auth enabled |
With authentication
export PHOENIX_ENABLE_AUTH=true
export PHOENIX_SECRET="your-secret-key-min-32-chars"
export PHOENIX_ADMIN_SECRET="admin-bootstrap-token"
phoenix serve
Best practices
- Use projects: Separate traces by environment (dev/staging/prod)
- Add metadata: Include user IDs, session IDs for debugging
- Evaluate regularly: Run automated evaluations in CI/CD
- Version datasets: Track test set changes over time
- Monitor costs: Track token usage via Phoenix dashboards
- Self-host: Use PostgreSQL for production deployments
Common issues
Traces not appearing:
from phoenix.otel import register
# Verify endpoint
tracer_provider = register(
project_name="my-app",
endpoint="http://localhost:6006/v1/traces" # Correct endpoint
)
# Force flush
from opentelemetry import trace
trace.get_tracer_provider().force_flush()
High memory in notebook:
# Close session when done
session = px.launch_app()
# ... do work ...
session.close()
px.close_app()
Database connection issues:
# Verify PostgreSQL connection
psql $PHOENIX_SQL_DATABASE_URL -c "SELECT 1"
# Check Phoenix logs
phoenix serve --log-level debug
References
- Advanced Usage - Custom evaluators, experiments, production setup
- Troubleshooting - Common issues, debugging, performance
Resources
- Documentation: https://docs.arize.com/phoenix
- Repository: https://github.com/Arize-ai/phoenix
- Docker Hub: https://hub.docker.com/r/arizephoenix/phoenix
- Version: 12.0.0+
- License: Apache 2.0
Related skills
langsmith-observability: LLM observability platform for tracing, evaluation, and monitoring.chroma: Open-source embedding database for AI applications.speculative-decoding: Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques.
Trigger Prompts & Evaluation Corpus
These evaluation prompts test that the agent correctly selects phoenix-observability for relevant tasks and leaves it out for near misses.
Should trigger (3)
- should trigger
“How do I get Phoenix tracing running locally for my OpenAI calls?”
expects Gives the install steps (arize-phoenix plus the OpenInference OpenAI instrumentor), launches the Phoenix server (default UI at port 6006), registers the tracer provider, and instruments OpenAI so traces show up in the UI.
- should trigger
“I'm running a RAG pipeline in LlamaIndex over our lab's protocol PDFs. The answers are sometimes wrong and I can't tell if retrieval or generation is at fault. I set up Phoenix at localhost:6006 but my project shows no traces, only an empty 'default' project. How do I fix that and then use the traces to find the failing step?”
expects Explains how to fix missing traces: register with the right project name and endpoint (OTLP/gRPC port 4317 or HTTP), call the LlamaIndex instrumentor before running queries, and check the collector endpoint and environment variables. Then shows how to inspect retrieval and LLM spans in the trace view to localize the failure.
- should trigger
“I have 200 question-answer pairs from our clinical-trial summarization model and want to systematically score hallucination and relevance with an LLM judge, compare two prompt versions, and keep the test set versioned so I can rerun it in CI. Self-hosted only, no SaaS. What setup do you recommend?”
expects Recommends self-hosted Phoenix: upload the pairs as a versioned dataset, use built-in LLM-as-judge evaluators (hallucination, relevance), run experiments comparing the two prompt versions, and automate the evaluation in CI. Notes the self-hosted PostgreSQL or SQLite option.
Near misses (2)
- near miss
“My LangChain app is already integrated with LangSmith. How do I set LANGCHAIN_TRACING_V2 and create a LangSmith dataset to run evaluations against?”
expects Directs the user to LangSmith setup (API key, LANGCHAIN_TRACING_V2, LangSmith datasets and evaluate), not Phoenix.
- near miss
“My Llama-70B inference is too slow for our real-time lab assistant. Can I use a small draft model to speed up decoding?”
expects Explains speculative decoding with a draft model (and options like Medusa or lookahead decoding), not observability tooling.