- 1 npm package
- 48 pages audited
- 100/100 on every page
- 36 agent actions
The portfolio nobody could read
This site is a video game. You scroll to fly through six worlds, collect six Infinity Stones (one per skill area), climb a hex skill tree, and at the end a snap brings the whole thing together. I love it. Crawlers didn't.
It's a React single-page app, so the HTML the server sends is basically this:
<body>
<div id="root"></div>
<script type="module" src="/assets/index.js"></script>
</body>Everything a person sees is drawn by JavaScript after the page loads. Google's crawler eventually runs JavaScript, but slowly and not always. Link previews on WhatsApp, Slack and LinkedIn don't run it at all, so they showed a bare URL. And when someone asked ChatGPT or Claude about me, the assistants fetching my site got an empty div. A portfolio is supposed to be found. Mine was invisible.
First attempt: my own prerender
The first fix was a build script, prerender.mjs. After vite build, it renders every route on the server and writes real HTML: the page content, the right <title> and description, Open Graph tags, JSON-LD and the page's CSS. Plus a sitemap.xml and a robots.txt.
That fixed Google and the link previews. But when I looked at what AI assistants actually need, it was a different list:
- Clean text, not markup. An agent doesn't want my pixel HUD and hex grids. It wants the content as markdown.
- A map of the site.
llms.txtis the emerging way to say "here is what's on this site and where to read it". - A clear policy. Which AI crawlers may train on my content, and which may use it to answer searches. Those are different questions.
- Things to do, not just read. "Contact Anubhav" should be an action an agent can take, not a form it has to guess at.
None of that is SEO, and every site that wants to be readable by agents needs the same pieces. So instead of growing my script, I built a package.
web-for-agents, feature by feature
web-for-agents is SEO and AI-agent readiness for React + Vite. Here is what it does, and how this site uses each part.
Head tags and typed JSON-LD. <Seo> manages the title, description, canonical URL, Open Graph and Twitter tags. Schema builders (person, article, breadcrumbList, softwareApplication, faqPage and more) produce typed JSON-LD that links nodes by @id, and the audit names any required rich-result field that is missing. Every post on this Dev Log is an Article with a breadcrumb.
Prerendering. A Vite plugin crawls from /, renders each route to static HTML with the full <head>, and the client bundle still boots on top.
A markdown copy of every page. Each page gets a .md twin, converted from its HTML. That works well for normal pages. Mine aren't normal: a hex skill board converted to markdown is a wall of noise. So the package has a markdown.transform hook, and this site writes each page's markdown from the same content files the page renders from. The skills page becomes clean lists of skills per area, not a grid of cells.
llms.txt. An index of every page linking to its markdown, plus llms-full.txt with everything in one file. llms.intro and llms.sections let me put who I am and how to hire me at the top, before the page list.
A required AI policy. robots.ai.training and robots.ai.search must both be set, or the build fails. I made it required on purpose: "allow AI training?" is a decision every site owner should make out loud, not inherit from a default. This portfolio allows both, because it wants to be found and quoted.
Agent actions, agents.json and WebMCP. You mark what an agent can do: AgentForm turns a form into an action whose schema comes from its fields, and agentLinkProps marks plain links. Every action is listed in agents.json, and in browsers that support WebMCP it is registered as a tool. The contact form on this site is an action called contact-anubhav.
Sitemap and robots.txt. Generated from the discovered routes, with lastmod dates taken from each HTML file.
The audit CLI. npx @anubhav-choubey/web-for-agents audit <url> scores any site from 0 to 100 on what agents need, with a Fix: line for every miss. --sitemap audits a whole site.
The dev overlay. While the dev server runs, a "WFA" button (or Alt+Shift+A) shows what an agent sees on the current page: head tags, JSON-LD, the markdown and the actions. It never ships in a production build.
The Agent Skill. npx web-for-agents skill copies a skill into your project, so a coding agent like Claude Code gets the integration checklist and a required build-and-audit loop.
The generate command. This site still uses its own prerender, because the game needed full control over the render. web-for-agents generate dist runs after any prerender and adds the markdown, llms.txt, agents.json, sitemap and robots files to the HTML that's already there.
The security model. Prerendering runs your app's code at build time, so it runs in a separate, locked-down Node process with a scrubbed environment. During the build, pages can only load the build's own files unless you allow other origins. Agent input is validated strictly against each action's schema, agents can never reach password, hidden, file or read-only fields, and destructive actions ask the user first.
The result
I ran the package's own audit against the live site:
$ npx @anubhav-choubey/web-for-agents audit https://anubhavchoubey.com --sitemap
web-for-agents audit: https://anubhavchoubey.com/ (48 pages from the sitemap)
PATH SCORE FAILING
/ 100 -
/about 100 -
/blog 100 -
/blog/mcp-auth-for-people-and-agents 100 -
...
/skills 100 -
/trophies 100 -
Pages: 48 audited, min 100, average 100And the single-page report for the home page:
Score: 100/100
PASS Content without JavaScript 25/25 1785 characters of text without JavaScript
PASS Title 8/8 Title is 56 characters
PASS Meta description 8/8 Meta description is 156 characters
PASS Open Graph 7/7 og:title, og:description and og:image are present
PASS Canonical URL 5/5 Canonical is https://anubhavchoubey.com/
PASS JSON-LD 12/12 1 JSON-LD block parses with an @type
PASS robots.txt 10/10 /robots.txt allows this page for all crawlers
PASS Sitemap 8/8 Sitemap https://anubhavchoubey.com/sitemap.xml lists this page
PASS llms.txt 7/7 /llms.txt found at https://anubhavchoubey.com/llms.txt
PASS Markdown copy 6/6 Markdown copy found at https://anubhavchoubey.com/index.md
PASS Agent map 4/4 Agent map found at https://anubhavchoubey.com/agents.json (48 pages, 36 actions (27 page-specific, 9 site-wide))The same game, the same scroll, the same snap. But now a crawler gets real HTML, an assistant gets clean markdown and a map, and an agent can send me a message without guessing.
What I learned
- Build for your own weird case first. The
markdown.transformhook and thegeneratecommand both exist because this site didn't fit the happy path. Those were the features other people needed too. - Make the important decision mandatory. A required AI policy annoys people for ten seconds and saves them from a default they never chose.
- Ship the audit with the tool. A score with a
Fix:line on every miss turns "is my site agent-ready?" from an opinion into a checklist.
Takeaway
A site can be a game for people and a clean, structured document for machines at the same time. The trick is to generate the machine side from the same content, not to maintain it by hand.
The package is on npm and GitHub, and you can see it running on this site.
Prefer Medium? This post is also there: read it on Medium.