@sdxc/spam

Spam scoring for user-generated submissions: local rules, reputation providers, a trainable classifier and an LLM second opinion add weighted signals, and the total becomes a ham, unsure or spam verdict with every reason attached.

Installation

npm add @sdxc/spam

Remote checks and moderator reports return @sdxc/result values, and the classifier's database store runs on remix/data-table. Both install alongside this package.

Usage

Score a submission

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";

let filter = createSpamFilter({ checks: DEFAULT_RULES });

let assessment = await filter.check({
	content: "cheap watches https://a.example https://b.example",
});
assessment.verdict; // "ham" | "unsure" | "spam"
assessment.score; // 5
assessment.signals; // [{ check: "links.count", score: 2, detail: "2 links" }, { check: "links.density", score: 3, … }]

Give the rules more to work with

Only content is required. Every other field lets more rules fire.

let assessment = await filter.check({
	content: form.get("comment"),
	format: "markdown",
	author: {
		name: form.get("name"),
		email: form.get("email"),
		ip: request.headers.get("CF-Connecting-IP"),
	},
	renderedAt: verifiedRenderTime, // signed at render, as @sdxc/honeypot does; never a plain hidden input
	languages: ["en", "es"],
});

Add reputation and a second opinion

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";
import { akismet } from "@sdxc/spam/akismet";
import { authorEmail } from "@sdxc/spam/author-email";

let filter = createSpamFilter({
	checks: [
		...DEFAULT_RULES,
		authorEmail(),
		akismet({ apiKey: AKISMET_KEY, blog: "https://example.com" }),
	],
	thresholds: { unsure: 5, spam: 10 },
	timeout: 1500,
});

Local rules run first. When they already reach the spam threshold, no request leaves the Worker.

Learn from moderators

let failures = await filter.report(submission, "ham"); // a moderator approved a held comment

Every check that learns receives the decision; failures lists the ones that could not record it.

API

@sdxc/spam

createSpamFilter({ checks, thresholds?, timeout? })

Returns a SpamFilter with two methods:

• check(submission) resolves a SpamFilter.Assessment: { verdict, score, signals, failures }. Checks run in three stages: local in order; remote in parallel, only while the score is below the spam threshold; escalation in parallel, only while the score is inside the unsure band. A check that fails, throws or overruns timeout (default 1500 ms, remote and escalation only) adds nothing and is listed in failures, so an outage never decides a verdict.

• report(submission, label) sends "spam" or "ham" to every check with a report method, in parallel, and resolves the ones that failed.

thresholds defaults to { unsure: 5, spam: 10 } (DEFAULT_THRESHOLDS). A score at or above spam is spam, at or above unsure is unsure, and anything lower is ham.

verdictFor(score, thresholds)

The mapping check uses, for re-scoring stored signals under new thresholds.

SpamCheck

The contract every check implements: name, stage ("local" | "remote" | "escalation"), check(submission, { signal, score }) and an optional report(submission, label). A local check returns Signal[]; any other returns Promise<Result<Signal[], SpamCheckError>>. signal aborts at the timeout, and score is the total earlier stages reached.

Signal

{ check, score, detail? }. check is a stable identifier such as links.count. A positive score points to spam and a negative one to ham, so a provider can vouch for an author.

Submission

{ content, format?, author?, renderedAt?, submittedAt?, languages? }. format is "text" (the default), "markdown" or "html"; author holds name, email, url, ip and userAgent; submittedAt defaults to the time of the check; languages are BCP 47 tags.

SpamCheckError

A check's failure. code is unavailable, timeout, misconfigured or invalid-response.

Rules

Each rule is a function returning a local SpamCheck; every weight and limit is an option with a documented default. DEFAULT_RULES holds all of them at their defaults, tuned together so a submission needs more than one kind of evidence to reach the spam threshold.

URL_SHORTENERS and ABUSED_TLDS are the lists linkTargets() uses by default.

@sdxc/spam/author-email

authorEmail({ disposableScore? }) scores an author.email on a disposable-mail domain (author-email.disposable, default 3), subdomains included. It ships separately because the domain list it bundles is large, so it is not in DEFAULT_RULES. An address that does not parse draws nothing.

@sdxc/spam/stop-forum-spam

stopForumSpam(options?) looks the author's IP, email and name up in StopForumSpam, a free community database of addresses reported for forum and blog spam. It sends only the fields the submission has, and nothing when it has none. Each reported field scores its weight times the reported confidence (ipScore 5, emailScore 6, usernameScore 2), reports last seen over maxAgeDays (90) ago are ignored, and the total is capped at maxScore (10). Signals are stop-forum-spam.ip, .email and .username. endpoint pins a regional mirror. The service has a daily query limit per IP and receives the author's IP, email and name.

@sdxc/spam/link-blocklist

linkBlocklist(options?) looks every linked domain and the author's website up in DNS URI blocklists over DNS-over-HTTPS, so it runs where no DNS socket exists. Zones are spamhaus, surbl and uribl; the default is ["surbl", "uribl"], and a dqsKey adds Spamhaus. A listing scores listedScore (6); a low-confidence one (URIBL grey, a Spamhaus "abused legitimate" domain) scores weakScore (2). At most maxHosts (10) domains are looked up, IP literals are skipped, and the total is capped at maxScore (12). Signals are link-blocklist.spamhaus, .surbl and .uribl. resolver routes queries through another DoH endpoint.

A blocklist's "query refused" or rate-limit answer fails as misconfigured, never as a listing. Spamhaus refuses public resolvers, so it needs a Data Query Service key, which travels in the query name. SURBL and URIBL are free for low-volume, non-commercial use and may refuse queries relayed through large public resolvers; when they do, the check reports misconfigured, and a private resolver or a commercial feed is the fix.

@sdxc/spam/akismet

akismet({ apiKey, blog }) sends each submission to Akismet's comment-check and turns the answer into a signal: akismet.spam (default 8, the unsure band), akismet.discard when Akismet marks it blatant (default 12, spam on its own), and akismet.ham when Akismet vouches for it (default -2). Akismet requires the author's IP, so a submission without author.ip sends nothing and draws nothing. commentType (default "comment"), blogLang and isTest pass through; the weights are spamScore, discardScore and hamScore. An invalid key fails as misconfigured, an HTTP or network error as unavailable. report forwards a moderator's decision to submit-spam or submit-ham, bounded by reportTimeout (default 10 s).

Akismet is free for personal, non-commercial sites and paid otherwise. Each call sends the author's IP, user agent, name, email, website and the full content.

@sdxc/spam/bayes

bayes({ store }) is a naive Bayes classifier that learns from moderators: every filter.report(submission, label) trains it. It tokenizes content words, a host: token per linked host and the author's website, and an email-domain: token, then combines the most telling tokens (Robinson's smoothed probabilities, Fisher's method) into a spam probability. At or above spamCut (0.9) it adds bayes.spam (spamScore, 6); at or below hamCut (0.2) it adds bayes.ham (hamScore, -4). Until the store holds minDocuments (20) reports of each label it adds nothing, so a new install relies on the rules alone. The signal's detail names the tokens that decided it. tokenize(submission, options?) is exported for inspecting what it learns.

Counts live behind TokenStore (read(tokens) and increment(tokens, label), both Results):

• MemoryTokenStore keeps counts in the instance, for tests.

• new DataTableTokenStore(db) stores them in a remix/data-table database, D1 and SQLite included. Create its table from SPAM_TOKENS_SCHEMA_SQL in a migration; spamTokens is the table definition. Every write is one upsert statement, so concurrent reports never lose a count.

In a multi-tenant app, build one store per tenant database, so one tenant's training never changes another's verdicts. The check runs in the local stage, since it reads the app's own database.

@sdxc/spam/workers-ai

workersAi({ ai, model?, site?, spamScore?, hamScore?, maxContentLength?, maxReasonLength? }) is an escalation check: the filter calls it only for submissions the other checks left unsure, so the model runs on the few cases rules cannot settle. Pass the Workers AI binding (env.AI) as ai, and describe the site in site ("comments on a cooking blog") so the model judges relevance for the right audience.

The check sends the content (cut to maxContentLength, default 4000 characters), the author's name and URL, and up to 20 of the signals earlier checks found, as escaped JSON inside delimiters, under a system prompt that treats everything inside as data to classify. The model answers spam, ham or unsure with a confidence and a reason: spam adds workers-ai.spam at spamScore × confidence (default 5, enough to carry an unsure 5 to 10), ham adds workers-ai.ham at -(hamScore × confidence) (default 5), and unsure adds nothing. The reason, cut to maxReasonLength (200), is the signal's detail. The default model is @cf/meta/llama-3.1-8b-instruct-fp8 (DEFAULT_WORKERS_AI_MODEL); model takes any Workers AI text model with JSON mode. ai is typed as any object with a matching run(), so a test passes a plain object.

@sdxc/spam/memory

new MemoryCheck({ name?, stage?, signals? }) is a scripted check for tests. A call answers the next queued outcome, else signals.

• signalNext(signals) / failNext(code, message?) queue the next answer.

• failReports(code) makes every later report fail.

• calls, last and reports record what the check received.

• reset() forgets calls, reports and queued outcomes.

name defaults to "memory" and stage to "remote".

Pattern: Refuse bots before scoring

A honeypot refuses what no person produces, before the body is read into a submission, so no rule, lookup or model call runs for it. @sdxc/honeypot does that, and what it lets through carries a signed render time for the timing rule.

import { Honeypot } from "@sdxc/honeypot";
import { honeypot } from "@sdxc/honeypot/middleware";
import { unwrap } from "@sdxc/result";
import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";

let trap = new Honeypot({ secret: HONEYPOT_SECRET });
let filter = createSpamFilter({ checks: DEFAULT_RULES });

router.post("/comments", {
	middleware: [honeypot(trap)],
	async handler(ctx) {
		let form = await ctx.request.formData();
		let assessment = await filter.check({
			content: String(form.get("content") ?? ""),
			renderedAt: unwrap(ctx.honeypotOutcome).renderedAt,
		});
		return saveComment(form, assessment);
	},
});

Pattern: Hold, discard or publish

The verdict is a recommendation; the app decides what each one does. Answering spam like a success teaches a bot nothing.

import type { Submission } from "@sdxc/spam";

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";

let filter = createSpamFilter({ checks: DEFAULT_RULES });

async function submitComment(submission: Submission) {
	let assessment = await filter.check(submission);
	if (assessment.verdict === "spam") return { status: "received" };
	let comment = await saveComment(submission, {
		published: assessment.verdict === "ham",
		reasons: assessment.signals,
	});
	return { status: comment.published ? "published" : "received" };
}

Pattern: Writing a rule for your site

A rule is an object with a name, a stage and a check function. Signals from it add to every other rule's.

import type { SpamCheck } from "@sdxc/spam";

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";

let competitorLinks: SpamCheck = {
	name: "competitor-links",
	stage: "local",
	check: (submission) =>
		/rival\.example/i.test(submission.content)
			? [{ check: "competitor-links", score: 4, detail: "links to rival.example" }]
			: [],
};

let filter = createSpamFilter({ checks: [...DEFAULT_RULES, competitorLinks] });

Pattern: Reweighting a rule

DEFAULT_RULES is a list of rule calls at their defaults. Build your own list to change one.

import { contactBait, createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";

let filter = createSpamFilter({
	checks: [
		...DEFAULT_RULES.filter((rule) => rule.name !== "contact-bait"),
		contactBait({ phoneScore: 0 }), // this form asks for a phone number
	],
});

Pattern: Testing an action that filters

MemoryCheck fixes the evidence, so a test exercises each verdict without a provider.

import { createSpamFilter } from "@sdxc/spam";
import { MemoryCheck } from "@sdxc/spam/memory";
import { expect, test } from "vitest";

test("holds an unsure comment for moderation", async () => {
	let check = new MemoryCheck({ signals: [{ check: "test", score: 6 }] });
	let filter = createSpamFilter({ checks: [check] });

	let assessment = await filter.check({ content: "hello" });

	expect(assessment.verdict).toBe("unsure");
	expect(check.last).toEqual({ content: "hello" });
});