# ============================================================
# robots.txt for www.757king.com
# Last updated: July 9, 2026
# Strategy: LLMO / maximize AI citation + search visibility
# ============================================================
#
# STRUCTURE NOTE (important, unchanged and correct):
# In robots.txt, a crawler obeys ONLY the single most-specific
# user-agent group that matches it, and ignores all other groups.
# Rules are NOT inherited or merged across groups.
#
# So we deliberately do NOT give Googlebot / Bingbot / GPTBot /
# ClaudeBot / PerplexityBot / OAI-SearchBot their own groups — that
# would make them IGNORE the Disallows below. They fall back to
# "User-agent: *" and correctly pick up the path/parameter Disallows.
#
# Only bots we treat DIFFERENTLY (full blocks, or data-use toggles)
# get their own group below.
# ============================================================
# ------------------------------------------------------------
# DEFAULT — applies to Google, Bing, DuckDuckGo, Yandex, Apple,
# GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, ChatGPT-User,
# Perplexity-User, Google AI Overviews crawlers, social preview
# bots, Zillow/Realtor bots, and everything else not named below.
#
# All AI SEARCH + CITATION engines land here and are fully allowed.
# Any future/unknown AI crawler also lands here and is allowed by
# default — this block is intentionally future-proof.
# ------------------------------------------------------------
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /staging/
Disallow: /widget/
Disallow: /survey/
Disallow: /appointment/
Disallow: /_next/
Disallow: /*?preview=
Disallow: /*?token=
Disallow: /*?ghl_
Disallow: /search/
Disallow: /find-a-home/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?page=
Disallow: /*?price_min=
Disallow: /*?price_max=
# NOTES on the default block:
# - No Crawl-delay: Google ignores it; Bing respects it and it SLOWS
# indexing. No reason to throttle legitimate crawlers.
# - No /wp-admin/ or /wp-login.php: site runs on GHL, not WordPress.
# Listing WP paths gives competitors false tech-stack intel.
# - VERIFY /find-a-home/: this blocks the faceted IDX search. Your IDX
# listing DETAIL pages live on the IDX subdomain (757king.idxbroker.com),
# which has its OWN robots.txt — so this block should only affect the
# on-site search landing, not individual listings. Confirm in Search
# Console that no listing detail URLs you want indexed sit under
# /find-a-home/ on the main domain.
# - Social preview bots inherit these Disallows, so a shared /find-a-home/
# or /search/ URL won't generate a rich preview. Community + seller +
# military + listing-detail pages preview fine.
# ------------------------------------------------------------
# CANONICAL HOST
# (Host: is honored only by Yandex. Enforce canonical host for
# Google/Bing with 301 redirects + .)
# ------------------------------------------------------------
Host: https://www.757king.com
# ============================================================
# AI DATA-USE / GROUNDING TOGGLES
# ------------------------------------------------------------
# These tokens control DATA USE + GROUNDING, not core search.
# For a citation-first (LLMO) strategy the aim is MAXIMUM allowed
# presence: let the answer engines both learn about and ground on
# your content. Flip to Disallow only if you later want to opt out
# of a specific model's data use.
# ============================================================
# Google-Extended — governs Gemini training AND Vertex/Gemini grounding.
# *** CHANGED TO ALLOW (July 2026) ***
# Allowing this lets Gemini ground on (and cite) your pages. Blocking
# it was costing you Gemini/Google-AI citations with zero upside for a
# get-cited goal. Google Search + AI Overviews were never affected either
# way. This is the single most important AI-citation change in this file.
User-agent: Google-Extended
Allow: /
# CCBot (Common Crawl) — open dataset a large number of AI models and
# smaller answer engines learn from.
# *** CHANGED TO ALLOW (July 2026) ***
# Training presence is how a model comes to "know" John King exists when
# a user asks it (unprompted) for a Hampton Roads / military Realtor.
# This is a data-use choice — flip back to Disallow to opt out.
User-agent: CCBot
Allow: /
# Applebot-Extended — Apple Intelligence TRAINING control. (Applebot,
# the Siri/Spotlight crawler, inherits the * rules and stays allowed.)
# Kept ALLOW to feed Apple Intelligence answers.
User-agent: Applebot-Extended
Allow: /
# Bytespider (ByteDance/TikTok) — aggressive TRAINING-ONLY scraper that
# does NOT cite you in any answer engine. No citation upside.
# Kept BLOCKED.
User-agent: Bytespider
Disallow: /
# ============================================================
# FULL BLOCKS
# ============================================================
# ------------------------------------------------------------
# OTHER TRAINING-ONLY / NON-CITING SCRAPERS — BLOCKED
# ------------------------------------------------------------
User-agent: Diffbot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Timpibot
Disallow: /
# ------------------------------------------------------------
# SEO SPY TOOLS — BLOCKED
# Trade-off: this also blocks YOUR OWN audits (Screaming Frog, Ahrefs
# Site Audit, Semrush, etc.). Competitors can still reconstruct your
# backlink profile from other sites that link to you, so this mostly
# blinds your own tooling. If you run your own audits, temporarily
# remove the relevant line (e.g. AhrefsSiteAudit / SemrushBot-SA) or
# verify site ownership in that tool to bypass robots.txt.
# ------------------------------------------------------------
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: SemrushBot-SI
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: AhrefsSiteAudit
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: MajesticSEO
Disallow: /
User-agent: MozBot
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: SEOkicks
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: ScreamingFrogSEOSpider
Disallow: /
User-agent: serpstatbot
Disallow: /
User-agent: seoscannerbot
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: LinkpadBot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: SiteAuditBot
Disallow: /
User-agent: ContentKingBot
Disallow: /
User-agent: Seekport
Disallow: /
User-agent: ZoominfoBot
Disallow: /
User-agent: BomboraBot
Disallow: /
# ------------------------------------------------------------
# CONTENT SCRAPERS, HOSTILE CRAWLERS & UNWANTED SEARCH — BLOCKED
# ------------------------------------------------------------
User-agent: HTTrack
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: SiteSucker
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: NinjaBot
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: ChinaClaw
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: Sogou
Disallow: /
# ------------------------------------------------------------
# SITEMAPS
# ------------------------------------------------------------
Sitemap: https://www.757king.com/sitemap.xml
# ------------------------------------------------------------
# NOTES
# ------------------------------------------------------------
# - Disallow blocks CRAWLING, not INDEXING. A disallowed URL can still
# appear in results if linked elsewhere. For anything that must stay
# out of the index, use or auth,
# not just Disallow.
# - Subdomains (cma.757king.com, 757king.idxbroker.com, etc.) need their
# own robots.txt OR noindex meta tags. GHL may not support subdomain
# robots.txt — use page-level noindex in GHL settings where needed.
# - Pair this file with a curated /llms.txt (a markdown map of your key
# pages for LLMs) for maximum AI discoverability.
# - After deploying, validate with the Google Search Console robots.txt
# tester and re-check that /find-a-home/ behaves the way you intend.