@sdxc/highlight

Syntax highlighting that returns tokens rather than markup, with a markdown walk visitor and a stylesheet keyed to the token types.

A highlighter's real output is a sequence of runs, each labelled with what it is. Markup is one way to render that, and the wrong way when the renderer builds a component tree, so this package returns the tokens and a caller renders them however it renders anything else. A grammar is a plain value a module exports: no global registry, no import order to get right.

Installation

npm add @sdxc/highlight

The root entry has no dependencies. @sdxc/highlight/markdown walks a document from @sdxc/markdown, whose walk reports its outcome as a Result from @sdxc/result; both install alongside this package when you use that entry.

Usage

Tokenize Source

import { tokenize } from "@sdxc/highlight";

let tokens = tokenize("let x = 1;", "ts");
// [
//   { type: "keyword", value: "let" },
//   { type: "plain", value: " x " },
//   { type: "operator", value: "=" },
//   { type: "plain", value: " " },
//   { type: "number", value: "1" },
//   { type: "punctuation", value: ";" },
// ]

The language arrives as an author wrote it on a fence, so tokenize resolves aliases itself — ts, js, sh, yml, jsonc and gql land on the grammar that serves them, and text, txt and dotenv land on plain.

Render Tokens

Map them. Nothing about a token needs interpreting:

<code className={`language-${language}`}>
	{tokens.map((token, index) =>
		token.type === "plain" ? (
			token.value
		) : (
			<span key={index} className={`token ${token.type}`}>
				{token.value}
			</span>
		),
	)}
</code>

Render Markup

When a caller needs a string rather than elements:

import { highlight } from "@sdxc/highlight";

let markup = highlight('let name = "x";', "ts");
// '<span class="token keyword">let</span> name <span class="token operator">=</span> …'

Every value is escaped, plain included, so the result is safe inside a <pre><code>.

Paint A Parsed Document

import { highlight } from "@sdxc/highlight/markdown";
import { Markdown } from "@sdxc/markdown";
import { isFailure } from "@sdxc/result";

let painted = Markdown.walk(document, highlight);
if (isFailure(painted)) throw painted.error;

for (let node of painted.data.children) {
	if (node.type === "code") node.tokens; // Token[]
}

Every handler this visitor holds is synchronous, so the walk answers with a Result rather than a promise, and a painting pass inside a render path is never awaited.

API

tokenize(code: string, language: string): Token[]

Tokenizes source as a language, resolving aliases first. The tokens arrive in source order and cover the input exactly once. A language with no grammar yields one plain token holding the whole input, and empty source yields an empty list.

let tokens = tokenize("SELECT 1", "sql");
let covered = tokens.map((token) => token.value).join("") === "SELECT 1"; // always true

highlight(code: string, language: string): string

Highlights source into <span class="token …"> markup, escaping every value it writes. This is the form for callers that need a string; a caller rendering components maps tokenize instead.

highlight("<b>&</b>", "hcl"); // "&lt;b&gt;&amp;&lt;/b&gt;"

normalizeLanguage(language: string): string

Resolves what a fence wrote to the name a grammar answers to, lowercasing it and following an alias when one applies. The result is a name to display and to look up, not a promise that a grammar exists.

normalizeLanguage("SH"); // "bash"
normalizeLanguage("txt"); // "plain"
normalizeLanguage("hcl"); // "hcl" — no grammar, and still a usable class name

languages: Record<string, Grammar>

Every grammar, by the name it registers under: bash, css, diff, graphql, html, http, javascript, json, jsx, markdown, plain, python, ruby, sql, tsx, typescript, yaml. html also serves the xml, svg, rss, atom, mathml and erb aliases, since it is general markup rather than a list of known element names.

Object.hasOwn(languages, normalizeLanguage("toml")); // false

scan(code: string, grammar: Grammar): Token[]

Scans source with a grammar directly, skipping the registry and the alias table. Adjacent runs of the same type arrive merged, so the output is the same whether a grammar spells a construct as one rule or several.

import { scan } from "@sdxc/highlight";

import { ini } from "./ini.js";

let tokens = scan("[server]\nport = 8080\n", ini);

compose(...parts: Array<Record<string, Rule[]>>): Grammar

Merges grammars into one, mode by mode, so a language built on another states that as an import. The earlier part's rules are tried first, which is how a JSX tag wins over a TypeScript comparison on the same <. A part with no main of its own merges the same way, which is how a set of modes lifted off another grammar joins one.

import { compose } from "@sdxc/highlight";

export let tsx = compose(elements, typescript);

highlight (from @sdxc/highlight/markdown)

A Markdown.Visitor holding one code handler, so a document paints in the pass that walks it. The handler resolves the language the block names, tokenizes its body, and returns a copy of the node carrying both; a block that names no language, as an indented block never does, is painted as plain. The node keeps its content, attributes and position, and visitors merge, so one walk paints and rewrites at once.

import { highlight } from "@sdxc/highlight/markdown";
import { Markdown } from "@sdxc/markdown";

let result = Markdown.walk(document, {
	...highlight,
	link(node) {
		return { ...node, href: canonical(node.href) };
	},
});

@sdxc/highlight/styles.css

The selector layer: one rule per token type, each reading a custom property named --highlight-<type>. It sets colour and nothing else, and ships light and dark defaults, so loading it alone already paints.

import "@sdxc/highlight/styles.css";

Types

Token

interface Token {
	type: Token.Type;
	value: string;
}

Token.Type

The twenty kinds of run a grammar can name:

attr-name, attr-value, boolean, builtin, class-name, comment, constant, deleted, function, inserted, keyword, number, operator, plain, property, punctuation, regex, string, tag, variable.

builtin is for names the language itself provides, class-name for types and classes, property for a key of any kind, tag for a markup tag name and a CSS selector, variable for a sigil-marked name, inserted and deleted for the two sides of a diff, and plain for everything no rule claimed.

Rule

interface Rule {
	type: Token.Type;
	match: RegExp;
	push?: string;
	pop?: true;
}

match carries the sticky flag, so it matches where the scanner is rather than searching ahead. push and pop move the mode stack, and a rule uses one or the other.

Grammar

interface Grammar {
	main: Rule[];
	[mode: string]: Rule[];
}

main is where scanning starts. Every other entry is a context a rule pushes onto the stack — a template literal's interpolation, a markup tag's attributes, the body of an embedded language.

Markdown.Code["tokens"]

declare module "@sdxc/markdown" {
	namespace Markdown {
		interface Code {
			tokens?: Token[];
		}
	}
}

The field @sdxc/highlight/markdown declares and fills: the runs of the block's body, in source order, covering its content exactly once. It is optional because a document that no walk painted has none, and it is derived data — a serializer writes the fields a parser produced, so tokens survives a round trip through the document and not through the text.

Pattern: Painting Tokens Without A Stylesheet

An email client has no stylesheet to load, so the colour has to be inline. Key the palette by the type union and the compiler checks it covers every member, which is what keeps a type added upstream from silently rendering unpainted:

import type { Token } from "@sdxc/highlight";

const COLORS: Record<Token.Type, string | undefined> = {
	comment: "#6a737d",
	keyword: "#d73a49",
	string: "#032f62",
	// …
	plain: undefined,
};

function paint(tokens: Token[]) {
	return tokens.map((token) => {
		let color = COLORS[token.type];
		return color ? <span style={`color:${color};`}>{token.value}</span> : token.value;
	});
}

Pattern: Theming The Stylesheet

Load the selector layer, then declare the properties for the roles you have an opinion about. The rest keep the defaults:

import "@sdxc/highlight/styles.css";

:root {
	--highlight-comment: #8b949e;
	--highlight-keyword: #ff7b72;
	--highlight-string: #a5d6ff;
}

The stylesheet sets colour and nothing else. A block's padding, border and radius belong to the page around it, as does any weight or face a theme spends on a role — a bold keyword, an italic comment — which a consumer adds with its own rules on the same classes.

Pattern: Writing A Grammar

A grammar is a record of modes, each a list of rules tried in order at the cursor:

import type { Grammar } from "@sdxc/highlight";

export let ini: Grammar = {
	main: [
		{ type: "comment", match: /[#;][^\n]*/y },
		{ type: "tag", match: /\[[^\]\n]*\]/y },
		{ type: "property", match: /[A-Za-z_][\w.]*(?=\s*=)/y },
		{ type: "operator", match: /=/y },
		{ type: "string", match: /"(?:\\[\s\S]|[^"\\\n])*"?/y },
	],
};

Anything no rule claims accumulates into a plain run, so a grammar is complete from its first rule and grows by claiming more. Rule order is priority: a comment rule goes above the operator rule that would otherwise claim its opening /.

Nesting is a mode. A rule with push enters one and a rule with pop leaves it, which is how a template literal's ${…} returns to being a string, and how a <script> body highlights as JavaScript and then stops at </script>. A construct that nests inside itself pushes its own mode again, so the brace that closes it is the one that matched:

let interpolation: Rule[] = [
	{ type: "punctuation", match: /\}/y, pop: true },
	{ type: "punctuation", match: /\{/y, push: "interpolation" },
	...expression,
];

Keep every pattern linear — no quantifier inside another quantifier over the same characters — so that highlighting a block costs what the block is long. Reach for a mode before a lookahead: a construct that spans a region is a mode, and lookarounds decide what a single character means, like whether a / divides or opens a regular expression. And pick the nearest existing token type; a grammar that wants a twenty-first one usually wants builtin, property or constant.