ChatGPT Search, Perplexity and Gemini crawl your site every day. Most sites are either blocked or unreadable. We configure robots.txt per crawler, deploy llms.txt, write citation-ready JSON-LD, and restructure content for RAG retrieval.
GPTBot, OAI-SearchBot, anthropic-ai, PerplexityBot, YouBot — each has a different purpose. Managing a training scraper and a live search crawler under the same rule wastes crawl budget and creates legal exposure. We write a separate directive set for each.
User-agent: GPTBot Allow: /blog/ Allow: /hizmetler/ Disallow: /admin/ User-agent: OAI-SearchBot Allow: /
llms.txt gives AI agents a machine-readable map of your site — which pages exist, what they mean, and which sources can be cited. Think of it as robots.txt rebuilt for the LLM era.
RAG pipelines run entity extraction. Without correct Organization, Service, FAQ and Review schemas, AI cannot recognize you as a named entity — but it recognizes your competitor. We write intent-matched schema for every page type.
LLMs read in chunks. Long paragraphs get lost; short, citable, semantically dense blocks get retrieved. We restructure service descriptions, FAQs, and social proof into retrievable units.
# Canlı arama — izin ver User-agent: OAI-SearchBot Allow: / # Eğitim verisi — kısıtlı User-agent: GPTBot Allow: /blog/ Disallow: / # Anthropic User-agent: anthropic-ai Allow: /
# RankMate > GEO ve AI görünürlük ajansı > URL: https://rankmate.agency/ ## Hizmetler > GEO + SEO Büyüme paketi > URL: /hizmetler/geo-seo/ > WhatsApp AI Satış Asistanı > URL: /hizmetler/whatsapp-ai/
{
"@type": "ProfessionalService",
"name": "RankMate",
"knowsAbout": [
"GEO", "AI SEO",
"LLM Visibility"
],
"areaServed": "TR"
}Technical score is a weighted average of bot access rules, llms.txt quality, schema coverage, page speed, and RAG content density.
If server logs show no GPTBot or OAI-SearchBot requests, they're either getting 403s or robots.txt is blocking them completely. We detect this in 5 minutes with a free audit.
Not yet a W3C standard, but Anthropic, OpenAI, and Perplexity all actively read it. A site without one is like a city without a map — AI doesn't know where to look.
Yes. We append only AI crawler directives and deploy llms.txt without touching your existing SEO rules. We test on staging first.
No. Schema is necessary but not sufficient. Incorrect or duplicate markup actively hurts you. Without llms.txt and RAG-ready content on top, AI still won't recognize you as an entity.