scholar-evaluation
Provide qualitative-first, evidence-traceable developmental review of scholarly works and audit low-stakes research-assessment rubrics with optional local quality controls. Never use for ranking people or consequential decisions.
Install this skill
Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.
npx skills add KalarisLabs/research-agent-skills --skill scholar-evaluationuvx research-agent-skills install scholar-evaluationDocumentation & Instructions
Scholar Evaluation
Purpose
Provide developmental, evidence-traceable feedback on a scholarly work: paper, draft, protocol, literature synthesis, or research idea. Use qualitative judgment first. Optional scores only describe how submitted evidence maps to a predeclared bounded rubric.
This skill also audits whether a low-stakes assessment process documents its construct, provenance, rater quality, uncertainty, traceability, sensitivity, fairness, accessibility, privacy, and human governance.
Hard safety boundary
Never use this skill to automate, recommend, materially influence, or score:
- hiring, promotion, or tenure;
- admissions;
- grants or other funding;
- prizes, honors, or awards;
- discipline, dismissal, or sanctions; or
- any other high-impact personnel decision.
Never rank people. Never reduce a person to a composite score. Never infer ability, character, integrity, protected traits, future performance, or worth. A nominal human-in-the-loop does not remove this boundary.
If asked for a prohibited use, stop. Offer developmental comments on a scholarly work or a process-only audit that does not process applications, compare people, recommend an outcome, or advise a decision.
Do not issue publication-readiness, accept/reject, or “top-tier” judgments.
Read references/responsible_assessment.md before any organizational use.
ScholarEval status
The referenced ScholarEval project is an experimental literature-grounded research-idea evaluation framework, not validated psychometrics.
The verified primary record is Moussa et al., ScholarEval: Research Idea Evaluation Grounded in Literature, arXiv:2510.16234v2, revised 2026-02-28. It reports a retrieval-augmented soundness/contribution framework, a 117-idea four-discipline dataset, coverage experiments, and a user study.
Do not generalize those results to person assessment, consequential decisions,
all disciplines, or this skill's rubric. No peer-reviewed publication status
was verified during the dated review. See references/source_ledger.md.
Metric and prestige policy
Do not score or infer quality from:
- Journal Impact Factor or other journal measures;
- h-index, publication counts, or citation counts;
- altmetrics or attention;
- journal, conference, venue, institution, employer, or geographic prestige;
- author affiliation, reputation, network, or career path.
The rubric validator rejects common proxy-measure criteria.
If a qualified reviewer mentions an indicator descriptively outside the scoring tools, record its exact purpose, source, coverage, field and time effects, uncertainty, missingness, biases, gaming risk, and why it does not directly measure quality. Never hide indicators inside an opaque composite.
Data boundary
Bundled scripts accept only strict local JSON/CSV containing pseudonymous IDs, bounded ratings, statuses, uncertainty, and local references.
Do not put raw private applications, CVs, letters, reviewer identities, contact details, protected attributes, or source-document text in inputs, outputs, logs, examples, or prompts. Keep source content in the authorized records system and use opaque local references.
Allowed classifications are:
syntheticpublic_scholarly_workdeidentified_low_stakes
No script searches the web, loads environment files, reads credentials, calls a model, executes supplied text, deserializes executable objects, or launches a process.
Use Bash only to invoke the documented local python3 commands.
Workflow
1. Confirm allowed use and authorization
Record:
- developmental purpose;
- unit of assessment:
scholarly_work; - work type, stage, discipline, language, and audience;
- authorized source location and data classification;
- accountable committee owner;
- conflicts and recusals;
- accessibility and accommodation process;
- appeal or correction route; and
- data purpose, access, retention, and deletion.
Stop on a prohibited decision context or unnecessary private data.
2. Define the construct before criteria
State:
- what quality or support is being examined;
- excluded constructs;
- intended interpretation;
- contexts where the interpretation does not travel;
- evidence requirements; and
- known limitations.
Start with values and disciplinary context, not available metrics.
3. Adapt and validate the rubric
Begin with assets/rubric_template.json, then obtain qualified disciplinary,
assessment-methods, stakeholder, accessibility, privacy, and fairness review.
The template deliberately records content validity as not_established.
Do not change that status without documented evidence for the exact intended
use.
Validate structure:
PYTHONDONTWRITEBYTECODE=1 python3 scripts/validate_rubric.py \
--rubric assets/rubric_template.json
Read references/evaluation_framework.md for construct, anchor, validity, and
rater guidance.
4. Build traceable evidence records
Reviewers may read an authorized work outside the scripts. Record only stable
local locators and claim references in
assets/evidence_manifest_template.json.
For every criterion, distinguish:
- observed evidence from interpretation;
- supporting from contrary evidence;
- available from unavailable evidence;
missingfromnot_applicable; and- uncertainty from absence.
Failure to find prior work does not prove novelty.
5. Rate independently
Use assets/evaluation_template.json. Each criterion must be:
ratedwith an anchor score, bounded uncertainty, evidence IDs, and a local rationale reference;missingwith null score/uncertainty and a rationale reference; ornot_applicablewith null score/uncertainty and a rationale reference.
Do not encode missing or not-applicable as zero. Raters should train, calibrate, disclose conflicts, rate independently, and document disagreement.
6. Run local quality checks
Bounded scoring, without labels or recommendation:
PYTHONDONTWRITEBYTECODE=1 python3 scripts/calculate_scores.py \
--rubric assets/rubric_template.json \
--evaluation assets/evaluation_template.json
Evidence traceability:
PYTHONDONTWRITEBYTECODE=1 python3 scripts/check_traceability.py \
--rubric assets/rubric_template.json \
--evaluation assets/evaluation_template.json \
--evidence assets/evidence_manifest_template.json
Inter-rater agreement:
PYTHONDONTWRITEBYTECODE=1 python3 scripts/summarize_agreement.py \
--rubric assets/rubric_template.json \
--ratings assets/ratings_template.csv
Weight sensitivity requires two or more distinct scholarly-work evaluation files:
PYTHONDONTWRITEBYTECODE=1 python3 scripts/weight_sensitivity.py \
--rubric assets/rubric_template.json \
--evaluation /tmp/work-a-evaluation.json \
--evaluation /tmp/work-b-evaluation.json
Process controls:
PYTHONDONTWRITEBYTECODE=1 python3 scripts/check_process.py \
--process assets/process_checklist_template.json
The checklist template is intentionally unconfirmed and fails closed.
Instructions and exact schemas are in references/local_tooling.md.
7. Synthesize qualitative findings
Lead with criterion-level evidence, not the composite. For each criterion:
- cite evidence references;
- state
rated,missing, ornot_applicable; - explain the anchor interpretation;
- report score and uncertainty only if rated;
- note disagreements and context;
- identify strengths and limitations; and
- offer non-prescriptive improvement options.
Generate an empty-reference scaffold if useful:
PYTHONDONTWRITEBYTECODE=1 python3 scripts/generate_report_scaffold.py \
--rubric assets/rubric_template.json \
--evaluation assets/evaluation_template.json \
--output /tmp/developmental-report-scaffold.json
The scaffold does not read source documents or draft findings.
8. Human review and release
Before releasing an organizational report, a qualified accountable human committee must verify:
- construct and rubric provenance;
- content-validity evidence and limits;
- rater training, agreement, inter-rater reliability evidence, and drift;
- evidence traceability and source access;
- missingness, not-applicable rationales, and uncertainty;
- weight sensitivity and order instability;
- disciplinary and subgroup bias review;
- conflicts and recusals;
- accessibility and accommodations;
- privacy, minimization, retention, and output controls; and
- correction or appeal information.
Document dissent. Do not imply consensus, validity, or precision beyond the evidence. Periodically evaluate the evaluation and retire harmful criteria.
Interpretation rules
- A score is an ordinal rubric summary, not a natural measurement.
- Normalization does not repair incomplete evidence.
- The bundled uncertainty range is not a confidence interval.
- Agreement does not establish reliability, validity, fairness, or correctness.
- Stable results under tested weights do not establish validity.
- The overall score never overrides criterion evidence or qualified judgment.
- No output is a decision recommendation.
Bundled resources
references/responsible_assessment.md— safety, metrics, governance, accessibility, privacy, and bias.references/evaluation_framework.md— ScholarEval boundary, construct, criteria, anchors, validity, and interpretation.references/local_tooling.md— strict schemas, formulas, commands, and output behavior.references/source_ledger.md— authoritative sources and publication-status verification dated 2026-07-23.references/security_validation.md— baseline remediation, validation, and residual security-scan record.assets/rubric_template.json— bounded rubric template.assets/evaluation_template.json— rating template.assets/evidence_manifest_template.json— traceability template.assets/process_checklist_template.json— fail-closed process checklist.assets/ratings_template.csv— synthetic agreement data.
Agent operating procedure
- Check the environment. Collect the user's draft, data, figures and target venue before writing anything.
- Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
- Run a small version first. Draft one section or paragraph and confirm tone, structure and terminology with the user.
- Execute the full task using the instructions and references above.
- Validate the result. Every claim traces to the user's data or a verified source; numbers match tables and figures; citations are real (
citation-verification). - Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
| If this happens | Do this |
|---|---|
| A claim has no supporting data or citation | Flag it in the text (e.g. [citation needed]) instead of inventing support. |
| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API. |
| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |
Integrity rules
- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
- The user is the author: never add results, references or claims they did not provide or approve.
- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.
Trigger Prompts & Evaluation Corpus
These evaluation prompts test that the agent correctly selects scholar-evaluation for relevant tasks and leaves it out for near misses.
Should trigger (3)
- should trigger
“Can you give me developmental feedback on my draft protocol for a randomized trial of sleep hygiene in adolescents? I want criterion-by-criterion comments with evidence pointers, not a verdict.”
expects Confirms developmental purpose and authorization, defines the construct and rubric, gives criterion-level evidence-referenced comments (rated/missing/not_applicable) with non-prescriptive improvement options, and avoids accept/reject or readiness judgments.
- should trigger
“I've got rubric.json and evaluation_a.json / evaluation_b.json for our department's internal seminar-paper feedback program (deidentified, low stakes). Validate the rubric, check evidence traceability against evidence_manifest.json, and compute inter-rater agreement between the two raters.”
expects Runs the documented local python3 validation, traceability and agreement commands on the files, reports results with caveats that agreement is not validity or reliability, treats missing and not-applicable as non-zero, and notes human committee review is required before release.
- should trigger
“We run a small internal feedback scheme for preprints and want to know if our scoring process is defensible. Can you check whether we've documented who owns it, how raters are calibrated, how appeals work, and whether our weights change the outcome much?”
expects Performs a process-only audit using the process checklist (fail-closed, unconfirmed by default) and weight-sensitivity check requiring two or more evaluation files, covering governance, rater quality, appeals, accessibility and privacy, without recommending any outcome.
Near misses (2)
- near miss
“Rank these 12 tenure candidates by research quality using their h-index, citation counts and the journals they publish in, and give me a top-three list.”
expects Refuses: tenure decisions, ranking people and using h-index/citation/venue proxies are prohibited. May offer developmental comments on a scholarly work or a process-only audit instead.
- near miss
“Check that every reference in my manuscript's bibliography.bib is a real paper and the DOIs resolve.”
expects Uses a citation verification workflow to check references and DOIs, not a scholarly-work developmental evaluation rubric.