面向制作公司的 llms.txt 与 AI 爬虫检查: 用 Claude Code 做发布前审查

网站制作公司和客户网站运营团队团队听到 llms.txt 与 AI 爬虫发布前审查时，常常先追文件、结构化数据或新名词。访问者真正需要的是答案、价格、负责人、依据和下一步。

依据来自 llms.txt 提案、Google robots.txt、Google crawler 文档、OpenAI 文档和 Cloudflare AI Crawl Control。 Related internal reading: professional-services inquiry flow and agency checklist.

要点

先看咨询点击、表单完成、重复问题、审查时间，以及发布前发现的风险项。
Claude Code ko review table banane दें: missing facts, risky claims, unclear CTA, URL boundary, owner approval.
Human owner price, legal scope, client policy, private URL aur final publication approve kare.
Start with one page, one checklist, and one clear consultation path.

失败从哪里开始

失败通常从先讨好 AI 系统开始，而不是先帮助读者。薄 FAQ 或新文件无法修复缺少范围、证据和 CTA 的页面。

Claude Code 负责的范围和人工确认的范围

Claude Code 可以阅读页面、整理问题、比较元数据、列 URL，并生成审查表。人工确认价格、法律表达、隐私、客户方针和发布决定。

For monetized work, send team-level workflows to training and consultation.

3 个 Use case

Use case 1: Review the main page

Input: page copy, FAQ notes, price notes, owner notes.
Output: missing facts, risky claims, unclear CTA, approval questions.
Human check: price, scope, credentials, privacy, client policy.

Use case 2: Compare metadata and visible text

Input: title, description, structured data, robots or llms notes, visible page.
Output: hidden claims, old dates, missing source links, unsafe promises.
Human check: each claim is true, current, and safe to publish.

Use case 3: Turn the workflow into a paid review

Input: page URL, sitemap, form fields, client approval sheet.
Output: screenshots, go/no-go list, fixes, next-month review items.
Human check: scope, price, owner, delivery limit, and follow-up plan.

可复制的提示词

あなたは制作会社の公開前レビュー担当です。クライアントサイトのrobots.txt、llms.txt、sitemap、重要ページを読み、AIクローラーとAI検索向けの境界を整理してください。

入力:
- robots.txt:
- llms.txt:
- sitemap URL:
- 公開してよいページ:
- 学習や要約に使われたくないページ:
- 問い合わせフォームURL:
- 管理画面や資料URL:

出力:
1. robots.txtで止めるもの
2. llms.txtに載せるもの
3. llms.txtに載せないもの
4. Search向けに残すべきページ
5. クライアント承認が必要な判断
6. 公開前にブラウザで見るURL一覧

禁止:
- llms.txtをSEO順位の魔法として説明しない
- robots.txtだけで機密情報を守れると言わない
- 管理画面、見積書、顧客資料のURLをサンプルに混ぜない

可运行的检查代码

const sitePolicy = {
  robotsTxt: [
    "User-agent: GPTBot",
    "Disallow: /private/",
    "User-agent: Google-Extended",
    "Disallow: /drafts/",
    "User-agent: *",
    "Disallow: /admin/",
    "Sitemap: https://example.com/sitemap-index.xml"
  ].join("\n"),
  llmsTxt: [
    "# Example Clinic Site",
    "",
    "## Services",
    "- https://example.com/services/",
    "- https://example.com/pricing/",
    "",
    "## Support",
    "- https://example.com/faq/",
    "",
    "## Do not summarize",
    "- Private estimates and customer records are not public sources."
  ].join("\n"),
  sensitivePaths: ["/admin/", "/private/", "/drafts/", "/customers/"]
};

function auditCrawlerBoundary(policy) {
  const errors = [];
  for (const path of policy.sensitivePaths) {
    if (!policy.robotsTxt.includes(`Disallow: ${path}`)) {
      errors.push(`robots.txtに ${path} のDisallowがありません`);
    }
  }
  if (!policy.robotsTxt.includes("Sitemap:")) {
    errors.push("Sitemap行がありません");
  }
  if (!policy.llmsTxt.includes("## Services")) {
    errors.push("llms.txtに公開サービスの節がありません");
  }
  if (policy.llmsTxt.includes("/admin/") || policy.llmsTxt.includes("/customers/")) {
    errors.push("llms.txtに非公開URLが混ざっています");
  }
  return { ok: errors.length === 0, errors };
}

console.log(auditCrawlerBoundary(sitePolicy));

Pitfall: 卖概念而不是卖审查

The risky pattern is selling llms.txt 与 AI 爬虫发布前审查 as a shortcut. The better offer is a visible review: what was checked, what changed, what needs approval, and what metric will be watched.

Another risk is sending private information into the workflow. Use sanitized examples and public pages. Remove client names, private estimates, customer records, draft URLs, and internal documents unless they are necessary and approved.

常见问题

Does this guarantee more search traffic?

不能保证流量。它让页面更清楚，也更容易审查。

What should be measured first?

先看咨询点击、表单完成、重复问题、审查时间，以及发布前发现的风险项。

Who approves the final page?

The business owner or client decision maker. Claude Code prepares the table; humans make the decision.

本文实际确认的内容

I checked the primary documentation, narrowed the workflow to reviewable artifacts, and included runnable Node.js code that catches common gaps before publication.