cellxgene-census
Query the CZ CELLxGENE Census programmatically for versioned public single-cell and spatial transcriptomics data. Use when you need population-scale cell metadata, gene expression slices, Census summary counts, source H5AD URIs/downloads, embeddings, spatial Census data, or reference atlas comparisons across organisms, tissues, diseases, assays, and cell types. For analyzing your own local single-cell data use scanpy, anndata, or scvi-tools.
Install this skill
Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.
npx skills add KalarisLabs/research-agent-skills --skill cellxgene-censusuvx research-agent-skills install cellxgene-censusDocumentation & Instructions
CZ CELLxGENE Census
Overview
The CZ CELLxGENE Census provides programmatic access to a comprehensive, versioned collection of standardized single-cell and spatial transcriptomics data from CZ CELLxGENE Discover. This skill enables efficient querying and analysis of public Census releases without downloading whole datasets first.
The Census includes:
- 217+ million total cells and 125+ million unique cells in the 2025-11-08 stable LTS release
- 1,845 datasets in the 2025-11-08 stable LTS release
- Human, mouse, marmoset, rhesus macaque, and chimpanzee data in the current schema
- Standardized metadata (cell types, tissues, diseases, donors)
- Raw gene expression matrices and source H5AD lookup/download helpers
- Pre-calculated summary counts, embeddings, and spatial data
- Integration with AnnData, Scanpy, TileDB-SOMA, TileDB-SOMA-ML, and other analysis tools
When to Use This Skill
This skill should be used when:
- Querying single-cell expression data by cell type, tissue, or disease
- Exploring available single-cell datasets and metadata
- Training machine learning models on single-cell data
- Performing large-scale cross-dataset analyses
- Integrating Census data with scanpy or other analysis frameworks
- Computing statistics across millions of cells
- Accessing pre-calculated embeddings or model predictions
Installation and Setup
Install the Census API:
uv pip install "cellxgene-census==1.17.*"
For spatial workflows:
uv pip install "cellxgene-census[spatial]==1.17.*" "spatialdata[extra]>=0.2.5"
For PyTorch model training, use TileDB-SOMA-ML. The old cellxgene_census.experimental.ml loaders are deprecated:
uv pip install "cellxgene-census==1.17.*" tiledbsoma-ml
Core Workflow Patterns
Eight patterns, each with code, are in references/core_workflow_patterns.md:
- Opening the Census — always pin
census_versionso an analysis stays reproducible. - Exploring Census information — available datasets, cell counts, and summary tables.
- Querying expression data — small to medium scale into an
AnnData. - Large-scale queries — out-of-core processing when the slice will not fit in memory.
- Machine learning with PyTorch — the Census data loaders.
- Spatial Census data — accessing spatial assays.
- Integration with Scanpy — handing a Census slice to a standard Scanpy workflow.
- Multi-dataset integration — combining datasets and handling batch effects.
Key Concepts and Best Practices
Always Filter for Primary Data
Unless analyzing duplicates, always include is_primary_data == True in queries to avoid counting cells multiple times:
obs_value_filter="cell_type == 'B cell' and is_primary_data == True"
Specify Census Version for Reproducibility
Always specify the Census version in production analyses:
census = cellxgene_census.open_soma(census_version="2025-11-08")
Estimate Query Size Before Loading
For large queries, first check the number of cells to avoid memory issues:
# Get cell count
metadata = cellxgene_census.get_obs(
census, "homo_sapiens",
value_filter="tissue_general == 'brain' and is_primary_data == True",
column_names=["soma_joinid"]
)
n_cells = len(metadata)
print(f"Query will return {n_cells:,} cells")
# If too large (>100k), use out-of-core processing
Use tissue_general for Broader Groupings
The tissue_general field provides coarser categories than tissue, useful for cross-tissue analyses:
# Broader grouping
obs_value_filter="tissue_general == 'immune system'"
# Specific tissue
obs_value_filter="tissue == 'peripheral blood mononuclear cell'"
Select Only Needed Columns
Minimize data transfer by specifying only required metadata columns:
obs_column_names=["cell_type", "tissue_general", "disease"] # Not all columns
Check Dataset Presence for Gene-Specific Queries
When analyzing specific genes, verify which datasets measured them:
presence = cellxgene_census.get_presence_matrix(
census,
"homo_sapiens",
var_value_filter="feature_name in ['CD4', 'CD8A']"
)
Two-Step Workflow: Explore Then Query
First explore metadata to understand available data, then query expression:
# Step 1: Explore what's available
metadata = cellxgene_census.get_obs(
census, "homo_sapiens",
value_filter="disease == 'COVID-19' and is_primary_data == True",
column_names=["cell_type", "tissue_general"]
)
print(metadata.value_counts())
# Step 2: Query based on findings
adata = cellxgene_census.get_anndata(
census=census,
organism="Homo sapiens",
obs_value_filter="disease == 'COVID-19' and cell_type == 'T cell' and is_primary_data == True",
)
Available Metadata Fields
Cell Metadata (obs)
Key fields for filtering:
cell_type,cell_type_ontology_term_idtissue,tissue_general,tissue_ontology_term_iddisease,disease_ontology_term_idassay,assay_ontology_term_iddonor_id,sex,self_reported_ethnicitydevelopment_stage,development_stage_ontology_term_iddataset_idis_primary_data(Boolean: True = unique cell)
The current schema includes organism collections beyond human and mouse. Confirm available organisms for the selected release with list(census["census_data"].keys()).
Gene Metadata (var)
feature_id(Ensembl gene ID, e.g., "ENSG00000161798")feature_name(Gene symbol, e.g., "FOXP2")feature_typefeature_length(Gene length in base pairs)nnz,n_measured_obs(availability summaries useful for checking sparsity and coverage)
Reference Documentation
This skill includes detailed reference documentation:
references/census_schema.md
Comprehensive documentation of:
- Census data structure and organization
- All available metadata fields
- Value filter syntax and operators
- SOMA object types
- Data inclusion criteria
When to read: When you need detailed schema information, full list of metadata fields, or complex filter syntax.
references/common_patterns.md
Examples and patterns for:
- Exploratory queries (metadata only)
- Small-to-medium queries (AnnData)
- Large queries (out-of-core processing)
- PyTorch integration
- Spatial Census access patterns
- Scanpy integration workflows
- Multi-dataset integration
- Best practices and common pitfalls
When to read: When implementing specific query patterns, looking for code examples, or troubleshooting common issues.
Common Use Cases
Use Case 1: Explore Cell Types in a Tissue
with cellxgene_census.open_soma() as census:
cells = cellxgene_census.get_obs(
census, "homo_sapiens",
value_filter="tissue_general == 'lung' and is_primary_data == True",
column_names=["cell_type"]
)
print(cells["cell_type"].value_counts())
Use Case 2: Query Marker Gene Expression
with cellxgene_census.open_soma() as census:
adata = cellxgene_census.get_anndata(
census=census,
organism="Homo sapiens",
var_value_filter="feature_name in ['CD4', 'CD8A', 'CD19']",
obs_value_filter="cell_type in ['T cell', 'B cell'] and is_primary_data == True",
)
Use Case 3: Train Cell Type Classifier
import tiledbsoma as soma
from tiledbsoma_ml import ExperimentDataset, experiment_dataloader
with cellxgene_census.open_soma() as census:
experiment = census["census_data"]["homo_sapiens"]
with experiment.axis_query(
measurement_name="RNA",
obs_query=soma.AxisQuery(value_filter="is_primary_data == True"),
) as query:
dataset = ExperimentDataset(
query=query,
layer_name="raw",
obs_column_names=["cell_type"],
batch_size=128,
shuffle=True,
)
dataloader = experiment_dataloader(dataset)
for X, obs in dataloader:
labels = obs["cell_type"]
# Training logic
pass
Use Case 4: Cross-Tissue Analysis
with cellxgene_census.open_soma() as census:
adata = cellxgene_census.get_anndata(
census=census,
organism="Homo sapiens",
obs_value_filter="cell_type == 'macrophage' and tissue_general in ['lung', 'liver', 'brain'] and is_primary_data == True",
)
# Analyze macrophage differences across tissues
sc.tl.rank_genes_groups(adata, groupby="tissue_general")
Troubleshooting
Query Returns Too Many Cells
- Add more specific filters to reduce scope
- Use
tissueinstead oftissue_generalfor finer granularity - Filter by specific
dataset_idif known - Switch to out-of-core processing for large queries
Memory Errors
- Reduce query scope with more restrictive filters
- Select fewer genes with
var_value_filter - Use out-of-core processing with
axis_query() - Process data in batches
Duplicate Cells in Results
- Always include
is_primary_data == Truein filters - Check if intentionally querying across multiple datasets
Gene Not Found
- Verify gene name spelling (case-sensitive)
- Try Ensembl ID with
feature_idinstead offeature_name - Check dataset presence matrix to see if gene was measured
- Some genes may have been filtered during Census construction
Version Inconsistencies
- Always specify
census_versionexplicitly - Use same version across all analyses
- Check release notes for version-specific changes
Agent operating procedure
- Check the environment. Confirm network access, API keys (if required) and the database's current API documentation and rate limits.
- Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
- Run a small version first. Fetch a single known record and check the response format before bulk queries.
- Execute the full task using the instructions and references above.
- Validate the result. Identifiers resolve, record counts are reported, and the database version or access date is recorded.
- Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
| If this happens | Do this |
|---|---|
| HTTP 429 or 5xx errors | Respect rate limits with backoff, batch requests, and report partial results honestly. |
| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API. |
| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |
Integrity rules
- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
- Never invent accession numbers, IDs or records; report 'not found' instead.
- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.
Related skills
anndata: Data structure for annotated matrices in single-cell analysis.scanpy: Standard single-cell RNA-seq analysis pipeline.scvi-tools: Trains and applies scvi-tools probabilistic deep generative models (scVI, scANVI, totalVI, MultiVI, PeakVI, DestVI, Solo, CellAssign, MrVI…
Trigger Prompts & Evaluation Corpus
These evaluation prompts test that the agent correctly selects cellxgene-census for relevant tasks and leaves it out for near misses.
Should trigger (3)
- should trigger
“How many primary human B cells are in the latest Census LTS release, broken down by tissue?”
expects Opens the Census with a pinned census_version (e.g. the stable LTS release) and queries obs with is_primary_data == True and cell_type == 'B cell', grouping by tissue_general or tissue. Reports the counts together with the release version used, and does not download full datasets.
- should trigger
“I'm writing a methods section for a lung fibrosis paper. I need all primary Homo sapiens cells from lung with disease 'pulmonary fibrosis' from CELLxGENE, only the columns cell_type, donor_id, dataset_id and assay, plus expression for SFTPC and KRT5, loaded into an AnnData I can pass to scanpy. My laptop has 16 GB RAM and my first attempt with get_anndata crashed with a memory error.”
expects Estimates the query size first by counting cells with a metadata-only obs query. Then uses get_anndata with a pinned census_version, obs_value_filter including is_primary_data == True, tissue and disease filters, a var_value_filter on feature_name for the two genes, and column_names limited to the needed fields. If the slice is still too large, falls back to out-of-core axis_query iteration or further filtering. Notes the Census version for reproducibility.
- should trigger
“I want to train a cell type classifier on millions of public human brain single-cell profiles without downloading every dataset by hand. What's the right way to stream that data into PyTorch and keep the training set reproducible?”
expects Recommends the versioned Census as the data source, pinned with census_version. Uses TileDB-SOMA-ML (not the deprecated cellxgene_census.experimental.ml loaders) with an obs filter on is_primary_data == True and tissue_general == 'brain'. Mentions train/test splitting by donor or dataset, and checking the query size first.
Near misses (2)
- near miss
“I have my own local pbmc.h5ad from a 10x run. Please do QC filtering, normalize, run PCA and UMAP, and cluster with Leiden.”
expects Does not use the Census. Uses scanpy for the QC, normalization, PCA/UMAP and Leiden clustering workflow on the local AnnData file.
- near miss
“Train an scVI model on my three batches of local single-cell data to correct batch effects and get a latent embedding.”
expects Does not use the Census. Uses scvi-tools to set up AnnData with a batch key, train scVI, and extract the latent representation.