mlflow
Tracks machine learning experiments and manages model lifecycles with MLflow, covering mlflow.log_param, log_metric and log_artifact, autologging for scikit-learn, PyTorch Lightning, XGBoost and HuggingFace Transformers, the Model Registry with versions and stage transitions, run searching, and local or cloud model serving. Use when logging parameters, metrics and artifacts for training runs, comparing runs across experiments, registering and promoting model versions from Staging to Production, serving a logged model for inference, or reproducing an experiment from an MLflow project. Not for hyperparameter-sweep dashboards or general data versioning; use a dedicated tool for those.
Install this skill
Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.
npx skills add KalarisLabs/research-agent-skills --skill mlflowuvx research-agent-skills install mlflowDocumentation & Instructions
MLflow: ML Lifecycle Management Platform
When to Use This Skill
Use MLflow when you need to:
- Track ML experiments with parameters, metrics, and artifacts
- Manage model registry with versioning and stage transitions
- Deploy models to various platforms (local, cloud, serving)
- Reproduce experiments with project configurations
- Compare model versions and performance metrics
- Collaborate on ML projects with team workflows
- Integrate with any ML framework (framework-agnostic)
Installation
# Install MLflow
pip install mlflow
# Install with extras
pip install mlflow[extras] # Includes SQLAlchemy, boto3, etc.
# Start MLflow UI
mlflow ui
# Access at http://localhost:5000
Quick Start
Basic Tracking
import mlflow
# Start a run
with mlflow.start_run():
# Log parameters
mlflow.log_param("learning_rate", 0.001)
mlflow.log_param("batch_size", 32)
# Your training code
model = train_model()
# Log metrics
mlflow.log_metric("train_loss", 0.15)
mlflow.log_metric("val_accuracy", 0.92)
# Log model
mlflow.sklearn.log_model(model, "model")
Autologging (Automatic Tracking)
import mlflow
from sklearn.ensemble import RandomForestClassifier
# Enable autologging
mlflow.autolog()
# Train (automatically logged)
model = RandomForestClassifier(n_estimators=100, max_depth=5)
model.fit(X_train, y_train)
# Metrics, parameters, and model logged automatically!
Core Concepts
1. Experiments and Runs
Experiment: Logical container for related runs Run: Single execution of ML code (parameters, metrics, artifacts)
import mlflow
# Create/set experiment
mlflow.set_experiment("my-experiment")
# Start a run
with mlflow.start_run(run_name="baseline-model"):
# Log params
mlflow.log_param("model", "ResNet50")
mlflow.log_param("epochs", 10)
# Train
model = train()
# Log metrics
mlflow.log_metric("accuracy", 0.95)
# Log model
mlflow.pytorch.log_model(model, "model")
# Run ID is automatically generated
print(f"Run ID: {mlflow.active_run().info.run_id}")
2. Logging Parameters
with mlflow.start_run():
# Single parameter
mlflow.log_param("learning_rate", 0.001)
# Multiple parameters
mlflow.log_params({
"batch_size": 32,
"epochs": 50,
"optimizer": "Adam",
"dropout": 0.2
})
# Nested parameters (as dict)
config = {
"model": {
"architecture": "ResNet50",
"pretrained": True
},
"training": {
"lr": 0.001,
"weight_decay": 1e-4
}
}
# Log as JSON string or individual params
for key, value in config.items():
mlflow.log_param(key, str(value))
3. Logging Metrics
with mlflow.start_run():
# Training loop
for epoch in range(NUM_EPOCHS):
train_loss = train_epoch()
val_loss = validate()
# Log metrics at each step
mlflow.log_metric("train_loss", train_loss, step=epoch)
mlflow.log_metric("val_loss", val_loss, step=epoch)
# Log multiple metrics
mlflow.log_metrics({
"train_accuracy": train_acc,
"val_accuracy": val_acc
}, step=epoch)
# Log final metrics (no step)
mlflow.log_metric("final_accuracy", final_acc)
4. Logging Artifacts
with mlflow.start_run():
# Log file
model.save('model.pkl')
mlflow.log_artifact('model.pkl')
# Log directory
os.makedirs('plots', exist_ok=True)
plt.savefig('plots/loss_curve.png')
mlflow.log_artifacts('plots')
# Log text
with open('config.txt', 'w') as f:
f.write(str(config))
mlflow.log_artifact('config.txt')
# Log dict as JSON
mlflow.log_dict({'config': config}, 'config.json')
5. Logging Models
# PyTorch
import mlflow.pytorch
with mlflow.start_run():
model = train_pytorch_model()
mlflow.pytorch.log_model(model, "model")
# Scikit-learn
import mlflow.sklearn
with mlflow.start_run():
model = train_sklearn_model()
mlflow.sklearn.log_model(model, "model")
# Keras/TensorFlow
import mlflow.keras
with mlflow.start_run():
model = train_keras_model()
mlflow.keras.log_model(model, "model")
# HuggingFace Transformers
import mlflow.transformers
with mlflow.start_run():
mlflow.transformers.log_model(
transformers_model={
"model": model,
"tokenizer": tokenizer
},
artifact_path="model"
)
Autologging
Details, code examples and parameter tables: references/autologging.md. Read it when this step applies.
Model Registry
Details, code examples and parameter tables: references/model-registry-2.md. Read it when this step applies.
Searching Runs
Find runs programmatically.
from mlflow.tracking import MlflowClient
client = MlflowClient()
# Search all runs in experiment
experiment_id = client.get_experiment_by_name("my-experiment").experiment_id
runs = client.search_runs(
experiment_ids=[experiment_id],
filter_string="metrics.accuracy > 0.9",
order_by=["metrics.accuracy DESC"],
max_results=10
)
for run in runs:
print(f"Run ID: {run.info.run_id}")
print(f"Accuracy: {run.data.metrics['accuracy']}")
print(f"Params: {run.data.params}")
# Search with complex filters
runs = client.search_runs(
experiment_ids=[experiment_id],
filter_string="""
metrics.accuracy > 0.9 AND
params.model = 'ResNet50' AND
tags.dataset = 'ImageNet'
""",
order_by=["metrics.f1_score DESC"]
)
Integration Examples
Details, code examples and parameter tables: references/integration-examples.md. Read it when this step applies.
Best Practices
1. Organize with Experiments
# ✅ Good: Separate experiments for different tasks
mlflow.set_experiment("sentiment-analysis")
mlflow.set_experiment("image-classification")
mlflow.set_experiment("recommendation-system")
# ❌ Bad: Everything in one experiment
mlflow.set_experiment("all-models")
2. Use Descriptive Run Names
# ✅ Good: Descriptive names
with mlflow.start_run(run_name="resnet50-imagenet-lr0.001-bs32"):
train()
# ❌ Bad: No name (auto-generated UUID)
with mlflow.start_run():
train()
3. Log Comprehensive Metadata
with mlflow.start_run():
# Log hyperparameters
mlflow.log_params({
"learning_rate": 0.001,
"batch_size": 32,
"epochs": 50
})
# Log system info
mlflow.set_tags({
"dataset": "ImageNet",
"framework": "PyTorch 2.0",
"gpu": "A100",
"git_commit": get_git_commit()
})
# Log data info
mlflow.log_param("train_samples", len(train_dataset))
mlflow.log_param("val_samples", len(val_dataset))
4. Track Model Lineage
# Link runs to understand lineage
with mlflow.start_run(run_name="preprocessing"):
data = preprocess()
mlflow.log_artifact("data.csv")
preprocessing_run_id = mlflow.active_run().info.run_id
with mlflow.start_run(run_name="training"):
# Reference parent run
mlflow.set_tag("preprocessing_run_id", preprocessing_run_id)
model = train(data)
5. Use Model Registry for Deployment
# ✅ Good: Use registry for production
model_uri = "models:/my-classifier/Production"
model = mlflow.pyfunc.load_model(model_uri)
# ❌ Bad: Hard-code run IDs
model_uri = "runs:/abc123/model"
model = mlflow.pyfunc.load_model(model_uri)
Deployment
Serve Model Locally
# Serve registered model
mlflow models serve -m "models:/my-classifier/Production" -p 5001
# Serve from run
mlflow models serve -m "runs:/<RUN_ID>/model" -p 5001
# Test endpoint
curl http://127.0.0.1:5001/invocations -H 'Content-Type: application/json' -d '{
"inputs": [[1.0, 2.0, 3.0, 4.0]]
}'
Deploy to Cloud
# Deploy to AWS SageMaker
mlflow sagemaker deploy -m "models:/my-classifier/Production" --region-name us-west-2
# Deploy to Azure ML
mlflow azureml deploy -m "models:/my-classifier/Production"
Configuration
Tracking Server
# Start tracking server with backend store
mlflow server \
--backend-store-uri postgresql://user:password@localhost/mlflow \
--default-artifact-root s3://my-bucket/mlflow \
--host 0.0.0.0 \
--port 5000
Client Configuration
import mlflow
# Set tracking URI
mlflow.set_tracking_uri("http://localhost:5000")
# Or use environment variable
# export MLFLOW_TRACKING_URI=http://localhost:5000
Resources
- Documentation: https://mlflow.org/docs/latest
- GitHub: https://github.com/mlflow/mlflow
- Examples: https://github.com/mlflow/mlflow/tree/master/examples
- Community: https://mlflow.org/community
See Also
references/tracking.md- Comprehensive tracking guidereferences/model-registry.md- Model lifecycle managementreferences/deployment.md- Production deployment patterns
Agent operating procedure
- Check the environment. Confirm hardware, framework and server versions, model format, and expected load.
- Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
- Run a small version first. Serve or log a single request or run end to end before scaling.
- Execute the full task using the instructions and references above.
- Validate the result. Measure latency, throughput and output correctness against a reference; check resource usage and costs.
- Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
| If this happens | Do this |
|---|---|
| The server fails to start or OOMs | Check model size versus memory, quantization and parallelism settings. |
| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API. |
| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |
Integrity rules
- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
- Do not expose services or credentials publicly; confirm cloud costs before provisioning.
- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.
Trigger Prompts & Evaluation Corpus
These evaluation prompts test that the agent correctly selects mlflow for relevant tasks and leaves it out for near misses.
Should trigger (3)
- should trigger
“How do I log my sklearn model's accuracy to MLflow?”
expects Shows starting a run with mlflow.start_run and logging accuracy via mlflow.log_metric, optionally with log_param and mlflow.sklearn.log_model or autolog. Notes the tracking URI and experiment name being used.
- should trigger
“I'm training an XGBoost classifier on train.csv for churn prediction and have a tracking server at http://localhost:5000. I ran 12 configs with different max_depth and learning_rate, and now want to pull the run with the best val_auc, register that model as 'churn-xgb', and move it to Production. Getting 'RestException: RESOURCE_DOES_NOT_EXIST' when I call transition_model_version_stage. What should I do?”
expects Uses mlflow.search_runs ordered by metrics.val_auc DESC to find the best run, registers it with register_model using the runs:/<run_id>/model URI, and explains that the error likely comes from a wrong model name or version that was not yet created. Notes that stages are deprecated in newer MLflow versions in favor of aliases, and says to check the installed version's docs rather than inventing APIs.
- should trigger
“I fine-tuned a BERT classifier last month and can't remember which learning rate and epoch count gave my best F1. I'd like each training run automatically recorded with its settings, scores and saved weights so I can compare them side by side later and ask a teammate to rerun the exact best one on their machine. How should I set that up?”
expects Recommends MLflow-style tracking with HuggingFace Transformers autologging (or manual logging of params, metrics and artifacts), comparing runs in the UI or via search_runs, and packaging the code as an MLflow Project or logging the environment and git commit so a teammate can reproduce it. Asks about or states assumptions on the tracking server and framework versions.
Near misses (2)
- near miss
“I want to run a Bayesian hyperparameter sweep over learning rate and batch size with a W&B sweep config and see parallel-coordinates plots in a shared dashboard. How do I write the sweep YAML and launch agents?”
expects Explains the W&B sweep YAML (method: bayes, metric, parameters) and launching with wandb sweep and wandb agent. It should not use MLflow, since this is a W&B sweep workflow.
- near miss
“Write a LightningModule with a training_step, validation_step and a ModelCheckpoint callback that monitors val_loss, and a Trainer set up for 4-GPU DDP.”
expects Provides a LightningModule, ModelCheckpoint(monitor='val_loss') and Trainer(strategy='ddp', devices=4) code. It does not focus on MLflow tracking or registry.