• axidort
    link
    fedilink
    English
    arrow-up
    1
    ·
    9 hours ago

    Worth separating what these two things are for, because “didn’t move the needle” often comes from expecting them to do something they weren’t designed to do.

    llms.txt is a hint file. As far as I know, none of the major crawlers or assistants have documented that they read it, so it is not a ranking or retrieval lever. Where it does help is when a developer points a coding agent or IDE at your docs by hand: a clean index of markdown pages is cheaper to ingest than a rendered docs site.

    An MCP server matters when an agent has to act or fetch live state: query an API, create a record, check an order. It does little for “will the model recommend or describe my product correctly”, because that mostly comes from what is in the training data and what gets retrieved from your public pages.

    So the stuff that tends to matter more, which fits the article’s title:

    • Docs that are accurate, versioned and have working copy-paste examples. Agents are only as good as the examples they can find.
    • Plain server-rendered HTML that does not hide content behind JS, plus a sitemap and sensible canonical URLs.
    • Not blocking the bots you want, including at the CDN/WAF layer. This is easy to get wrong without noticing.
    • Clear, specific pages for the questions people actually ask (install, upgrade, common errors), instead of one big marketing page.

    On measurement: pick a fixed set of 20-30 real tasks or questions, run them against a couple of agents before and after each change, and score pass/fail by hand. Server logs show whether AI user agents fetch you at all, and referral traffic from assistants is a rough signal. One caveat: one maintainer’s single-package experience is an anecdote, so test it for your own case rather than treating it as a rule in either direction.