# @sdxc/distill

Distill the article out of a web page: fetch under bounds, score the candidates, sanitize what
is left.

A page is not a document. It is an article wrapped in navigation, a share rail, a comment
thread and three related-post blocks, and deciding which subtree is the article is a scoring
heuristic rather than a question the markup answers. This package does that scoring, and it
treats the page it scored as what it is — untrusted markup from an origin nobody vetted — so
what comes back out is sanitized before any caller can render it.

## Installation

```bash
npm add @sdxc/distill
```

Every answer is a `Result`, so install
[`@sdxc/result`](https://www.npmjs.com/package/@sdxc/result) alongside this package to read
one.

## Usage

```typescript
import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

let article = await distill(post.url, {
	userAgent: "MyApp/1.0 (+https://myapp.example/about)",
});

if (isFailure(article)) return excerptAnd(post.url, article.error.outcome);

render(article.data.html, article.data.title, article.data.byline);
```

Honouring the origin's `robots.txt`, with the outcome retrieved (and cached for its
`lifetimeMs`) through [`@sdxc/robots`](https://www.npmjs.com/package/@sdxc/robots):

```typescript
import { distill } from "@sdxc/distill";
import { fetchRobots } from "@sdxc/robots/fetch";

let userAgent = "MyApp/1.0 (+https://myapp.example/about)";
let robots = await fetchRobots(post.url, { userAgent });
let article = await distill(post.url, { userAgent, robots });
```

Already holding the markup — from a fixture, or from a response somebody else retrieved:

```typescript
import { distillFrom } from "@sdxc/distill";

let article = distillFrom(source, "https://example.com/post");
```

## API

### `distill(url, options)`

Retrieves a page and reads the article out of it, answering
`Result<Distill.Retrieved, DistillError>`.

`options.userAgent` is required: a publisher who wants to refuse should be able to tell who
is asking, and only the caller knows what to call itself. `options.robots` takes the
`RobotsFetch.Outcome` that `@sdxc/robots/fetch`'s `fetchRobots` answered for the origin, and a
path it disallows is refused before any request goes out: a parsed file is evaluated for the
agent, an unavailable one (a 4xx) permits everything, and an unreachable one (a 5xx, a 429 or a
network failure) refuses everything. Omitting it consults nothing. `maxBytes`, `maxRedirects`, `timeoutMs`
and `signal` move the four bounds below.

### `distillFrom(source, url)`

The same scoring, sanitization and metadata over markup in hand, answering
`Result<Distill.Article, DistillEmptyError>`. `url` is what every relative URL in the
markup resolves against, so it is the address the page was actually served from.

### `addressable(url)`

Whether an address is somewhere this package is willing to go, before any request is made.

### `@sdxc/distill/retrieve`

The four bounds on their own, for any client fetching a URL somebody else chose (a
Webmention source, a link preview, a feed):

- `addressable(url)` and `isAddressableHost(hostname)`: the check made before every request.
- `follow(url, options)`: walks the redirect chain under the bounds and answers the last
  response whatever its status, as `{ response, url }` with the final URL.
- `retrieve(url, options)`: the same, answering only a 2xx; a refusing status (401, 402,
  403, 429, 451) is a `DistillRefusedError` and any other failure a `DistillLimitError`.
- `readWithin(retrieved, cap?)`: reads the body as text within the byte cap, answering
  `{ text, bytes }`.
- `release(body)`: cancels a body left unread, so the origin stops sending.
- `MAX_BYTES`, `MAX_REDIRECTS`, `TIMEOUT_MS`, `DistillLimitError` and `DistillRefusedError`.

`options` is `{ userAgent, maxRedirects?, timeoutMs?, signal? }`; the deadline covers the whole
chain and the body read that follows it.

```typescript
import { addressable, follow, readWithin } from "@sdxc/distill/retrieve";
import { isFailure } from "@sdxc/result";

let url = addressable(input);
if (isFailure(url)) return url;

let followed = await follow(url.data, { userAgent: "MyApp/1.0 (+https://myapp.example)" });
if (isFailure(followed)) return followed;
if (followed.data.response.status === 410) return gone();

let body = await readWithin(followed.data, 1_048_576);
```

### Outcomes

Three errors, each carrying an `outcome` a caller renders copy from.

| Error                 | `outcome` | What happened                                                 |
| --------------------- | --------- | ------------------------------------------------------------- |
| `DistillRefusedError` | `refused` | The site said no, or the address is one this will not ask for |
| `DistillLimitError`   | `timeout` | Time, bytes or hops ran out                                   |
| `DistillEmptyError`   | `empty`   | The page arrived carrying no article                          |

`Distill.Retrieved` adds `bytes`, and `mayCache` — `false` for a response whose
`X-Robots-Tag` carries `noarchive` for every agent or for the one asking, which is the exact
name for asking not to be kept.

## The four bounds

Exported as `MAX_BYTES`, `MAX_REDIRECTS` and `TIMEOUT_MS`, and moved per call through
`options`.

- **HTTP(S) only**, checked before the request, with it a refusal of any host that is a
  literal IP address, a loopback name or a `.local` name.
- **Five redirects**, walked manually with `redirect: "manual"`, which is what gives the chain
  a length it can exceed and makes the final URL a fact this package tracked.
- **Two megabytes**, counted off the stream and abandoned mid-body, so a response lying about
  its length is refused by the same bytes as one that is honest.
- **Eight seconds**, because somebody is waiting on this one.

The request carries no cookies, no credentials and no header naming whoever asked for it.

## Sanitization

`distill` and `distillFrom` both sanitize before they answer, so no consumer can forget to.
An allow-list decides what survives: prose, lists, tables, figures, links and images, with
`href` and `src` restricted to `http:`, `https:` and `mailto:`, every relative URL resolved
against the article's own address, and every surviving image carrying `referrerpolicy="no-referrer"`
and `loading="lazy"`. Everything else — `on*` handlers, `style`, `class`, `id`, scripts,
iframes and forms — is gone.

Sanitization alone leaves the publisher able to see an address when an image loads. Closing
that needs an image proxy, which belongs to whoever is rendering rather than here.

## Pattern: Caching What Comes Back

Nothing about a distilled article is about who asked for it, so the URL is the whole key and
one distillation serves everybody who opens the same link. Keep a failure too, for far less
time than a success: a blocked site asked once an hour is politer than one asked on every
open, and `mayCache` is a page asking not to be kept at all.

```typescript
import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

const DAY_MS = 86_400_000;

/** Whatever store you already have, keyed by the article's URL. */
interface ArticleStore {
	read(url: string): Promise<{ html: string | null } | null>;
	write(url: string, entry: { html: string | null }, options: { ttl: number }): Promise<void>;
}

async function articleFor(url: string, cache: ArticleStore): Promise<string | null> {
	let hit = await cache.read(url);
	if (hit !== null) return hit.html;

	let article = await distill(url, { userAgent: "MyApp/1.0 (+https://myapp.example/about)" });

	if (isFailure(article)) {
		// An hour, so a site that refuses today is asked again tomorrow rather than hourly.
		await cache.write(url, { html: null }, { ttl: DAY_MS / 24 });
		return null;
	}

	// A page carrying `X-Robots-Tag: noarchive` is answered, never stored.
	if (article.data.mayCache) {
		await cache.write(url, { html: article.data.html }, { ttl: DAY_MS * 7 });
	}

	return article.data.html;
}
```

## Pattern: Deciding The Article Was Worth It

A teaser, a consent interstitial and an error page are all pages that parse, score and
sanitize perfectly well while carrying nothing the reader did not already have. Comparing
`chars` against the excerpt already in hand is the one predicate that covers all three.

```typescript
import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

async function betterThanExcerpt(url: string, excerpt: string): Promise<string | null> {
	let article = await distill(url, { userAgent: "MyApp/1.0 (+https://myapp.example/about)" });
	if (isFailure(article)) return null;
	return article.data.chars > excerpt.length ? article.data.html : null;
}
```

## Versioning

Releases are dated rather than semantic. A version is the UTC date it was published, written
`YYYY.M.D`, so `2026.9.4` is the release from 4 September 2026. At most one release goes out
per day.

Those numbers say when, not what: a later date means a later release and carries no
compatibility promise. Any release may change or remove an export.

Depend on one exact date, and move it when you are ready to take the change:

```json
{
	"dependencies": {
		"@sdxc/distill": "2026.9.4"
	}
}
```

A caret or tilde range reads the date as major, minor and patch, so it accepts every later
release in the same year. An exact version keeps the upgrade yours to schedule.

## License

MIT

## Author

[Sergio Xalambrí](https://sergiodxa.com)