A skill that knows nothing: the architecture of Modern Web Guidance

Published:

The usual way to give expert knowledge to a coding agent is to write a Markdown file and trust that the agent reads it. That works until the document grows: every token of guidance is a token that isn’t available for the code, for the conversation history or for the output. Context is a scarce resource, and large skills consume it even when the task doesn’t need them.

Modern Web Guidance, the Chrome team’s project to stop agents from writing outdated patterns, inverts that pattern. Its main skill holds no knowledge: it holds instructions for going out to find it. And the search engine is an embeddings model that runs locally, with no network and no API keys.

It’s an interesting catalogue of guides, but what grabs me is the architecture underneath, because it’s replicable in any project that needs to bring domain knowledge to an agent.

The inverted pattern

The numbers in the published package explain the decision well. The modern-web-guidance distribution includes 139 Markdown guides that add up to about 975 KB of text, in the order of a quarter of a million tokens. Loading all of them isn’t an option: they don’t fit, and even if they did, filling the window with 138 irrelevant guides degrades the model’s attention on the one that does matter.

The SKILL.md that gets installed is 5.6 KB. Of that, the description block in the frontmatter is about 1,000 characters, in the order of 250 tokens, and it’s the only part permanently in context. The body of the skill loads only when the agent decides to activate it, and what it holds isn’t a guide, it’s two commands:

# Step 1: search for relevant use cases
npx -y modern-web-guidance@latest search "<query>"

# Step 2: retrieve the full guide by ID
npx -y modern-web-guidance@latest retrieve "<id>"

Right here we can already see a dependency on Node and npm.

The agent uses about 250 tokens at all times, about 1,400 once it activates the skill, and only then decides which specific guide is worth putting into context. Search returns the tokenCount of each candidate, so the agent itself can reason about the cost before asking for it.

The project also keeps a megaskill, a single document with all the content concatenated, for agents that can’t run commands. It’s the plan B, not the main solution.

Local semantic search, no network and no API keys

The search command doesn’t call any API: it runs an embeddings model inside the Node process.

The model is all-MiniLM-L6-v2, a small BERT from the sentence-transformers family that produces 384-dimension vectors. It’s converted to a TensorFlow.js Graph Model and sits inside the npm package: 22.5 MB of weights in group1-shard1of1.bin plus 576 KB of topology. Unquantized, in float32, a decision the repository justifies as maximum accuracy parity with the original model.

Two details of the conversion matter. The graph has the mean pooling and L2 normalization layers baked in, so the output tensor is already the final normalized vector and there’s no post-processing in JavaScript. And the BERT tokenizer resolves with local_files_only against a cache that also ships inside the package, so there isn’t a single request to Hugging Face at runtime.

// serving/lib/tfjs-embedder.ts (simplified)
await setBackend("cpu");
this.model = await loadGraphModel(ioHandler);
this.tokenizer = await BertTokenizer.from_pretrained(
  "Xenova/all-MiniLM-L6-v2",
  { local_files_only: true }
);

The index is precomputed at build time, not at query time. Each guide is split by headings and each fragment is embedded with a prefix that gives it context about where it lives:

const embeddingText = `${id} (${category})\nFeatures: ${featuresUsed.join(", ")}\n\n${chunk}`;
const vector = await embedder.embed(embeddingText);

The result is serialized into a 5.6 MB use-cases.vectors.gen.json.gz that ships with the package. On each search, the CLI computes the embedding of the query, walks the vectors in memory calculating cosine similarity, drops anything below 0.3 and returns five results. Since there are several fragments per guide, it groups by ID and keeps the highest similarity of each one: the fragments compete with each other, but the guide shows up only once.

There’s no vector database and no approximate index. At this volume, a loop over an array and a dot product are enough.

Why embeddings and not grep? Because the agent asks by intent and the guides are written with specification vocabulary. This query shares no meaningful word with the guide that answers it:

$ npx -y modern-web-guidance@latest search "close a popup when the user clicks outside of it"
[{"id":"light-dismiss-a-dialog","description":"Create a modal dialog that can be closed via light dismiss (i.e. clicking or tapping outside of the dialog)","category":"ui-behaviors","featuresUsed":["<dialog closedby>"],"tokenCount":1085,"similarity":0.6284},
...

Whoever writes the guide says light dismiss and <dialog closedby>; whoever asks says popup and clicks outside. A lexical search fails; the vector space gets it right at 0.63 similarity.

Switch queries to compare both methods over the same 139 guides. The results are real: the meaning column comes from running search, and the keyword column from counting occurrences in the package files.

The third case is the one I like most, because it doesn’t work out cleanly: embeddings improve precision, they don’t guarantee it. Latency sits around 1.2 to 1.5 seconds per query on CPU (MacBook Air M4), process startup included. For an operation that happens once at the start of a task, that’s a reasonable price for spending no network and no context.

The “local” part deserves a qualifier: inference never leaves the machine, but the CLI does send usage telemetry, and the query text goes in it along with the returned IDs and the latency. It’s on by default and turns off with DISABLE_TELEMETRY=1.

The guide as an evaluable unit

The second good decision in the design is that a guide isn’t a document, it’s a directory with a contract. Each use case brings together five pieces:

The expectations are verifiable and deterministic, not judgments:

- Cards with class name `.off-screen-card` must set `content-visibility: auto`.
- Cards with class name `.off-screen-card` must set `contain-intrinsic-size`.
- The `.debug-panel` must be hidden using `content-visibility: hidden`.

The piece that closes the system is calibration: the grader has to score 100% on the good demo and 0% on the negative one. If it doesn’t, the pipeline deletes it, regenerates it with the failing tests passed in as context, and retries. A grader that passes both implementations measures nothing, and without this check nobody would notice: it would keep scoring 100% in every evaluation and the guide would look perfect.

On the content side, the guides don’t repeat compatibility data by hand. They interpolate it with macros that expand at build time against web-features and MDN’s compatibility data:

{{ BASELINE_STATUS("content-visibility") }}

Which produces, with today’s data:

Baseline status for content-visibility: Newly available. It's been Baseline since 2025-09-15.
Supported by: Chrome 108 (Nov 2022), Edge 108 (Dec 2022), Firefox 130 (Sep 2024), and Safari 26 (Sep 2025).

The care about density reaches the format of that line. The code that generates it consolidates desktop and mobile versions into a single label when they match, and splits them out only when they diverge, justifying it with an analysis over the compatibility data. It looks like a formatting detail, but it’s a context engineering decision.

Calibrating the content with evaluations

With reliable graders, the question “is this guide worth anything?” stops being an opinion. The harness runs each task twice, unguided and guided, and compares the rate of assertions that pass.

The project uses two metrics to decide. Opportunity is 100% minus the unguided rate: if the model already solves the task well on its own, there’s nothing to gain and the guide is redundant. Uplift is the difference between both rates, and it’s what gets optimized. Guides shed everything models already know, because that content takes up context without changing the result.

The repository publishes aggregate results over 130 tasks and more than a thousand assertions. Across different models, unguided agents land around 54-57% of assertions passed, and guided ones between 82% and 89%. The jump is around 30 percentage points and stays fairly stable across agents. Read it with the usual caution: it’s a benchmark designed by the people who write the guides, over tasks they pick themselves. What matters isn’t the figure, it’s that the closed loop exists.

What we can reuse in our own project

The architecture replicates with small pieces and none of them needs infrastructure.

We can separate the index from the content, leaving in the skill only the instructions for searching. We can precompute embeddings at build time and ship them as a compressed artifact, because a few dozen or a few hundred documents don’t justify a vector database. We can split by headings and prefix each fragment with its identity, which is what keeps similarity from depending on the fragment repeating its own context. And we can demand of every document a test that proves the guide changes the result.

That last part is the most demanding and the one that pays off most. Writing an agent’s documentation without measuring its effect is writing blind, and the natural bias is to always add more text. With a grader in front of us, removing content becomes a defensible decision.

Conclusion

I love how Modern Web Guidance approaches context: it treats it as a budget. Keeping the skill present costs about 250 fixed tokens, retrieval is delegated to a semantic search engine that runs on the machine of whoever is programming, and only then does it spend context on the specific guide that’s needed.

If we’re writing skills for our teams, the pattern of searching first and retrieving after with local embeddings is cheaper to set up than it looks and scales much better than Markdown that grows out of control. The code is available in modern-web-guidance-src and a real guide, the navigation-drawer one, shows the final format that reaches the agent.

Glossary

Terms that show up in the article and are worth having clear:

Next Post
The CrUX 28-day rolling window