Course quality

course quality scores a course against a six-category rubric. It's intended for generated courses — ones imported from markdown, a directory tree, a URL crawl, or a YouTube transcript — where no learner has interacted yet and there is no engagement data to fall back on. The score is computed from each lesson's markdown and its section breakdown alone, so it's immediately available the moment a course finishes importing.

The command outputs six per-category sub-scores (0–1) plus a single 0–100 composite with a letter grade, and surfaces the worst-offender lessons so you know what to fix first.

Run it

learnos course quality <id-or-title> [--verbose] [--publish] [--format json|text]

<course> is either a UUID or a case-insensitive title substring. Defaults to a text report; --verbose adds per-lesson stats; --format json emits a structured report for downstream tooling. Pass --publish to upsert the report so it shows up in the web UI.

Where it shows up

A computed report stays local until you pass --publish. Once published:

• Quality tab on the course detail page (/courses/[id]) renders the full report — composite score, six category bars (click to expand the notes), and a Top findings list where each entry deep-links to the offending lesson.
• Quality badge on the Discover detail page (/discover/[slug]) for published courses — a small "B · 73.8" pill in the metadata row, with a hover popover showing the six-category breakdown. Only appears once a report exists.

Recomputing and re-publishing overwrites the prior report — the badge and tab always reflect the most recent publish. There's no separate "publish quality" action in the web UI; computing and publishing are one command in the CLI.

Sample output:

Cosmos 3 - NVIDIA's World Foundation Model  (c5ab37cb-…)
─────────────────────────────────────────────────
composite score: 38.5 / 100  [F]

totals:
  1 lessons · 1 modules · 1,458 words · ~7 min reading
  0 code · 0 mermaid · 0 math · 0 links

citations             ····················  0%
  · 0.00 cites/1k words (target 3+)
  · 1/1 lessons have zero citations

structure             ██████████████······  70%
  · 1 lessons across 1 modules
  · module balance: mean 1.0, stdev 0.0 (cv 0.00)
…

How the composite works

The composite is the weighted sum of the six sub-scores, scaled to 0–100. Citations carry the most weight because uncited generated content is the failure mode this rubric is most concerned with catching.

Grade bands:

The rubric

Each category produces a score ∈ [0, 1] and a list of notes. Sub-scores inside a category use one of two mappings:

• ramp-up — 0 below a floor, 1 at a target, linear between. Used when more of something is better.
• ramp-down — 1 below a target, 0 above a ceiling, linear between. Used when less of something is better.

1. Citations (27%)

Does the course back up its claims? Are sources diverse?

Combined as density × 0.5 + coverage × 0.35 + diversity × 0.15.

Citations are counted from markdown links — [text](https://…), <https://…> autolinks, and bare http(s)://… URLs — found outside fenced code blocks. Frontmatter sources: entries don't count here; they're scored under Scaffolding so the two signals stay independent.

2. Structure (16%)

Is the course well-shaped? Balanced modules, reasonable lesson lengths.

Combined as balance × 0.4 + length × 0.3 + size × 0.3.

A lesson is a "short" outlier with <150 words (and >0), "long" with >5000 words, "empty" with 0 words.

3. Content richness (16%)

Code, diagrams, math — the multimodal payload that distinguishes a real lesson from a wall of prose.

Combined as code × 0.4 + media × 0.25 + tag × 0.2 + runnable × 0.15.

"Runnable" languages are js, javascript, ts, typescript, and py-run — the ones LearnOS lessons can actually execute in the browser. Math is counted from both $$…$$ display blocks and ```math fenced blocks. Mermaid diagrams are detected by their ```mermaid fence tag.

4. Retrieval-readiness (16%)

Is the course chunked well enough for retrieval and citation to work? Pure prose with no H2/H3 sectioning retrieves badly and breaks deep links.

Combined as density × 0.45 + coverage × 0.35 + sectioning × 0.2.

"Sections" are the heading-scoped pieces LearnOS splits each lesson into for search and citation. A lesson with no H2/H3/H4 headings collapses to a single "intro" section, which both retrieves poorly and gives chat citations nothing specific to link at.

5. Scaffolding (14%)

Pedagogical bones: frontmatter, objectives, resources, practice.

Each is a simple present / total ratio over lessons. Sections are detected by heading text (case-insensitive, H1–H4). A frontmatter prerequisites: [] array counts even if empty — the act of declaring it is the scaffolding signal.

6. Writing quality (11%)

Lexical variety, sentence rhythm, and the absence of LLM filler phrases.

Combined as ttr × 0.45 + filler × 0.35 + sentence × 0.2.

TTR (type-token ratio) is unique alpha-tokens divided by total alpha-tokens, sampled over the first 1000 tokens of each lesson so values are comparable across lesson lengths. Aggregated across the course as a word-count-weighted average.

Filler phrases flagged (case-insensitive substring match):

let's dive in, let's get started, in this lesson, in this section, by the end of this, as we explore, we'll cover, we will cover, let's explore, in conclusion, to wrap up, buckle up, deep dive, without further ado, now that we

Findings

After scoring, the report surfaces up to ~10 worst-offender bullets:

• uncited lesson — a lesson with >200 words and zero external links
• weak sectioning — a lesson with ≤1 section despite >400 words
• no frontmatter sources — a lesson with >200 words but no sources: frontmatter array

These are intentionally narrow — the bullets point at fixable lessons, not abstract scores. If a course scores 60% on citations, the findings tell you which lessons to go cite.

What this rubric doesn't measure

So you don't read the score as more comprehensive than it is:

• Link health — URLs are counted, not checked. A lesson can cite five dead links and score the same as one citing five live ones.
• Source authority — every domain counts equally. A citation of a major reference work scores the same as a citation of a random blog.
• Inter-lesson redundancy — two near-duplicate lessons both score well in isolation; the rubric doesn't compare lessons against each other.
• Title-vs-body alignment — a "Promises" lesson that's actually about async/await isn't flagged.
• Concept ladder — whether lesson N introduces terms before lesson N+1 uses them isn't checked.
• Engagement — citation-hit rate from chat, completion rate, flashcards generated, and highlight density are richer signals, but they only accrue once the course has been used. This rubric scores the artifact, not its reception.