Code Journal Club: SciAgent-Skills — Markdown as a Plugin System for Bioinformatics AI Agents

Architecture & Design

SciAgent-Skills isn't a library in the usual sense. It's a content-as-plugin system where the unit of distribution is a markdown file, not a Python package. That single choice shapes the rest of the design.

Observation 1: The SKILL.md unit is the abstraction. Each skill is a self-contained directory holding SKILL.md (frontmatter plus instructions the agent reads) and, for 37 of them, a references/ subfolder with deeper material (e.g. skills/biostatistics/pymc-bayesian-modeling/references/distributions_inference.md). This is a plugin manifest pattern, closer to Jekyll themes or MkDocs extensions than to a Python module hierarchy. The CLAUDE.md and AGENTS.md files at the repo root are themselves just instructions loaded into agent context. The headline count wobbles, though: the README says 199, registry.yaml lists 203, and the plugin manifest's description says 197.

Observation 2: A YAML registry plus Python validators. registry.yaml is the source of truth for which skills ship and their metadata. Three scripts sit around it: scripts/validate_registry.py (schema check), scripts/blind_knowledge_test.py (runs skills against held-out questions), and .claude/skills/sciagent-skill-creator/scripts/validate_description.py (lints new contributions). Only the first is wired into CI: .github/workflows/validate.yml runs pytest, which exercises the registry validator. The other two are manual tools. For a content repo this is a lightweight QA setup: YAML, a schema test, and a benchmark runner you run by hand.

Observation 3: A meta-skill lets the system extend itself. The sciagent-skill-creator skill teaches the agent how to write a new skill following the project's conventions, and AGENTS.md spells out the contribution steps, including the registry update. Replaced skills move to legacy/, which AGENTS.md describes as "deprecated entries kept for historical reference". The integration-templates/ folder ships rules files for Cursor and Windsurf plus a generic AGENTS.md (the file Codex CLI reads by convention), so the content isn't tied to Claude Code.

Observation 4: Tooling is deliberately thin. pixi.toml and pixi.lock manage a Python 3.12+ environment for running validators only; end users don't install anything. The README is emphatic on this: "Skills are plain markdown files read directly by your AI agent — no npx, npm install, or runtime dependencies needed." That fits a field where researchers already juggle conda environments. The skills themselves still assume you have the underlying tools (scanpy, RDKit, and so on) installed.

The honest limitation: 200-odd skills is a lot of markdown to keep current, and the 92% BixBench number (vs. a 65.3% baseline) is on BixBench-Verified-50, a 50-task subset. Real-world accuracy on niche assays (long-read, metagenomics, custom pipelines) isn't measured in the README.

Test Run

Tested on 2026-09-16 at commit fe505ca with Claude Code 2.1.273, loading the repo as the README recommends: claude --plugin-dir /path/to/SciAgent-Skills.

Setup:
A shallow clone is 9.2 MB and takes seconds. The plugin loaded with no errors. But when asked to count its sciagent-skills:* entries, the agent saw 139, not 203. Two packaging problems cause this, and both are easy to confirm on disk:

• .claude-plugin/plugin.json lists 11 category folders but leaves out skills/medical-imaging/, so all 7 imaging skills (pydicom, nnU-Net, SimpleITK, PathML…) are missing.
• Under skills/genomics-bioinformatics/, 57 skills sit one level deeper than everywhere else (single-cell/, rnaseq/, variant/, alignment/…), and the loader doesn't look that deep.

203 − 7 − 57 = 139. So the whole single-cell, RNA-seq and variant-calling catalog is silently missing. Asking for the scanpy skill directly returns Unknown skill: sciagent-skills:scanpy-scrna-seq. When we asked the agent to "use the scanpy skill" anyway, it said the skill wasn't available and wrote a generic scanpy tutorial pipeline instead. At least it didn't pretend.

Workaround: add ./skills/medical-imaging/ and the eight ./skills/genomics-bioinformatics/<subdir>/ paths to the skills array in plugin.json. With that local patch, the same query returned all 203 skills.

First run (basic example):
RDKit (loads without the patch). We asked for MW, logP and TPSA of aspirin, caffeine and ibuprofen. The generated script used Descriptors.MolWt / MolLogP / TPSA, the same calls as the skill's descriptor step. It ran in 0.4 s (72 MB peak) with RDKit 2026.3.6 and printed aspirin 180.16 / 1.31 / 63.60, caffeine 194.19 / −1.03 / 61.82, ibuprofen 206.28 / 3.07 / 37.30. The molecular weights match reference values.

Real-world performance:
scanpy (patched manifest). With the fix in place, the agent confirmed it invoked scanpy-scrna-seq and wrote a pbmc3k QC → HVG → PCA → Leiden → UMAP script in 19 s. We ran it on a Linux workstation (scanpy 1.12, Python 3.12). It ran cleanly in 38 s with 664 MB peak memory, kept 2,698 of 2,700 cells, found 6 Leiden clusters at resolution 0.5, and saved the UMAP. Two things worth knowing. The skill's default QC cutoffs (< 5,000 genes, < 20% mitochondrial) are loose enough to remove almost nothing from pbmc3k; the classic scanpy tutorial uses < 2,500 and < 5% and keeps 2,638 cells. And the agent quietly skipped the skill's regress_out step while copying the rest nearly line for line.

Verdict:
Useful as a library of well-written, copyable workflows for agent-assisted analysis. Just don't assume you're getting the catalog the README advertises: loaded as documented, nearly a third of it (including all of single-cell and RNA-seq) isn't there. Patch plugin.json first, and still check the thresholds against your own data.

Links: GitHub repo · BixBench benchmark · HyperLab web platform