Blocking AI Scrapers: Protecting Your Content from Unauthorised Training Data


estimated read time: 6 minutes

Blocking AI Scrapers: Protecting Your Content from Unauthorised Training Data

This week I discovered something fascinating (and mildly infuriating) in my blog analytics: 68% of my "traffic" wasn't human readers — it was AI scrapers. OpenAI's GPTBot, Meta's scraper, ChatGPT-User, and a parade of other bots were harvesting my technical writing to train models that would then regurgitate my work without attribution.

Here's how I figured it out, why it matters, and how I'm blocking them.

The Discovery

I was looking at Loki logs for my blog's post_read events this week (2026-05-26 onwards) and noticed something odd:

That direct traffic percentage was suspiciously high. Most blogs see 30-50% direct at best. So I dug into the user agents:

LOKI="http://loki.example.com"
curl -G "$LOKI/loki/api/v1/query_range" \
  -H "X-Scope-OrgID: fake" \
  --data-urlencode 'query={namespace="production"} |= "post_read"' \
  | python3 -c "import json, sys; ..."

The Results

Known AI scrapers: 47% of traffic

Suspected bot farms: 21% of traffic

Actual human readers: 32% of traffic

So my "trending posts"? The view counts were inflated by roughly 3x due to bot traffic.

Why This Matters

The Plagiaris… I Mean "Training Data" Problem

When OpenAI's GPTBot scrapes your blog, it's harvesting your content for model training. Later, someone asks ChatGPT "how do I set up Longhorn on k3s?" and it regurgitates a summary of your guide — without attribution, without a link, without credit.

You spent hours writing that guide. OpenAI makes billions from a model trained on your work. You get nothing.

It's Not Just OpenAI

The "But It's Public!" Argument

Yes, my blog is public. That doesn't make it public domain. Copyright exists. Just because I publish something openly doesn't mean companies have the right to commercialise derivatives without permission.

If I write a book and put it in a public library, that doesn't mean anyone can photocopy it, repackage it, and sell it.

The Solution: Blocking AI Scrapers

I'm implementing three layers of defence:

1. robots.txt (Polite Request)

Most scrapers honour robots.txt. Some don't. But it's the baseline:

# Blocking AI Scrapers
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow legitimate search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Note the distinction: Googlebot (search indexing) is allowed. Google-Extended (AI training) is blocked.

2. HTML Meta Tags

Next.js supports robots meta tags in the layout:

export const metadata: Metadata = {
  // ...
  robots: {
    index: true,
    follow: true,
  },
  other: {
    robots: "noai, noimageai",
  },
};

This tells crawlers "index me for search, but don't train AI models on me."

3. Traefik Middleware (HTTP Header)

For defence in depth, I'm adding an X-Robots-Tag HTTP header via Traefik middleware in my k3s cluster:

Helm chart: infra/templates/middleware.yaml

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: block-ai-scrapers
spec:
headers:
customResponseHeaders:
X-Robots-Tag: "noai, noimageai"

Reference it in the Ingress: infra/values.yaml

ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.middlewares: "production-blog-block-ai-scrapers@kubernetescrd"

This sends the header on every response, even for dynamically-generated pages where meta tags might be missed.

4. (Optional) User-Agent Filtering

If you want to be more aggressive, you can block scrapers at the ingress level:

# In Traefik IngressRoute or nginx config

if ($http_user_agent ~\* (GPTBot|ChatGPT-User|ClaudeBot)) {
return 403;
}

I'm not doing this yet — I want to monitor whether the polite blocks work first.

Testing the Implementation

After deploying, you can verify the headers:

curl -I https://blog.foulkes.cloud

# Should see:

# X-Robots-Tag: noai, noimageai

And check that robots.txt is accessible:

curl https://blog.foulkes.cloud/robots.txt

The Helm Chart Changes

For those running Next.js on k3s via Helm (like I am), here's the full changeset:

Branch: feat/block-ai-scrapers

Files changed:

Full diff: github.com/dfoulkes/personal-blog/pull/XXX (TODO: update with PR number)

Will This Actually Work?

For well-behaved scrapers: Yes. OpenAI, Anthropic, Google, and others have all stated they honour robots.txt and noai directives.

For bad actors: No. Some scrapers ignore robots.txt entirely. If they become a problem, you need user-agent filtering or a WAF (Web Application Firewall).

Can they still scrape my content? Technically yes — if they ignore robots.txt and fake their user-agent, there's nothing stopping them short of IP blocking or CAPTCHAs (which hurt legitimate users).

But making it harder and explicitly stating you don't consent strengthens your legal position if it ever comes to that.

The Bigger Picture

This isn't just about my blog. It's about the web as a commons.

If every technical blog gets scraped, summarised, and regurgitated by AI chatbots — who will write the blogs? Why would I spend hours documenting how to configure Longhorn backups to MinIO if ChatGPT just steals it and serves it up without sending readers my way?

AI companies are training on the outputs of human expertise and then replacing the humans. That's a death spiral for the open web.

So block the scrapers. Make them ask permission. Force them to compensate creators. Or at least make them acknowledge that "training data" isn't some abstract digital resource — it's someone's work.

Resources

Update Log

If you found this useful, consider starring the personal-blog repo or sharing this post. With humans. Not bots.

Related Articles