# ============================================================ # robots.txt for www.757king.com # Last updated: July 9, 2026 # Strategy: LLMO / maximize AI citation + search visibility # ============================================================ # # STRUCTURE NOTE (important, unchanged and correct): # In robots.txt, a crawler obeys ONLY the single most-specific # user-agent group that matches it, and ignores all other groups. # Rules are NOT inherited or merged across groups. # # So we deliberately do NOT give Googlebot / Bingbot / GPTBot / # ClaudeBot / PerplexityBot / OAI-SearchBot their own groups — that # would make them IGNORE the Disallows below. They fall back to # "User-agent: *" and correctly pick up the path/parameter Disallows. # # Only bots we treat DIFFERENTLY (full blocks, or data-use toggles) # get their own group below. # ============================================================ # ------------------------------------------------------------ # DEFAULT — applies to Google, Bing, DuckDuckGo, Yandex, Apple, # GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, ChatGPT-User, # Perplexity-User, Google AI Overviews crawlers, social preview # bots, Zillow/Realtor bots, and everything else not named below. # # All AI SEARCH + CITATION engines land here and are fully allowed. # Any future/unknown AI crawler also lands here and is allowed by # default — this block is intentionally future-proof. # ------------------------------------------------------------ User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Disallow: /staging/ Disallow: /widget/ Disallow: /survey/ Disallow: /appointment/ Disallow: /_next/ Disallow: /*?preview= Disallow: /*?token= Disallow: /*?ghl_ Disallow: /search/ Disallow: /find-a-home/ Disallow: /*?sort= Disallow: /*?filter= Disallow: /*?page= Disallow: /*?price_min= Disallow: /*?price_max= # NOTES on the default block: # - No Crawl-delay: Google ignores it; Bing respects it and it SLOWS # indexing. No reason to throttle legitimate crawlers. # - No /wp-admin/ or /wp-login.php: site runs on GHL, not WordPress. # Listing WP paths gives competitors false tech-stack intel. # - VERIFY /find-a-home/: this blocks the faceted IDX search. Your IDX # listing DETAIL pages live on the IDX subdomain (757king.idxbroker.com), # which has its OWN robots.txt — so this block should only affect the # on-site search landing, not individual listings. Confirm in Search # Console that no listing detail URLs you want indexed sit under # /find-a-home/ on the main domain. # - Social preview bots inherit these Disallows, so a shared /find-a-home/ # or /search/ URL won't generate a rich preview. Community + seller + # military + listing-detail pages preview fine. # ------------------------------------------------------------ # CANONICAL HOST # (Host: is honored only by Yandex. Enforce canonical host for # Google/Bing with 301 redirects + .) # ------------------------------------------------------------ Host: https://www.757king.com # ============================================================ # AI DATA-USE / GROUNDING TOGGLES # ------------------------------------------------------------ # These tokens control DATA USE + GROUNDING, not core search. # For a citation-first (LLMO) strategy the aim is MAXIMUM allowed # presence: let the answer engines both learn about and ground on # your content. Flip to Disallow only if you later want to opt out # of a specific model's data use. # ============================================================ # Google-Extended — governs Gemini training AND Vertex/Gemini grounding. # *** CHANGED TO ALLOW (July 2026) *** # Allowing this lets Gemini ground on (and cite) your pages. Blocking # it was costing you Gemini/Google-AI citations with zero upside for a # get-cited goal. Google Search + AI Overviews were never affected either # way. This is the single most important AI-citation change in this file. User-agent: Google-Extended Allow: / # CCBot (Common Crawl) — open dataset a large number of AI models and # smaller answer engines learn from. # *** CHANGED TO ALLOW (July 2026) *** # Training presence is how a model comes to "know" John King exists when # a user asks it (unprompted) for a Hampton Roads / military Realtor. # This is a data-use choice — flip back to Disallow to opt out. User-agent: CCBot Allow: / # Applebot-Extended — Apple Intelligence TRAINING control. (Applebot, # the Siri/Spotlight crawler, inherits the * rules and stays allowed.) # Kept ALLOW to feed Apple Intelligence answers. User-agent: Applebot-Extended Allow: / # Bytespider (ByteDance/TikTok) — aggressive TRAINING-ONLY scraper that # does NOT cite you in any answer engine. No citation upside. # Kept BLOCKED. User-agent: Bytespider Disallow: / # ============================================================ # FULL BLOCKS # ============================================================ # ------------------------------------------------------------ # OTHER TRAINING-ONLY / NON-CITING SCRAPERS — BLOCKED # ------------------------------------------------------------ User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Timpibot Disallow: / # ------------------------------------------------------------ # SEO SPY TOOLS — BLOCKED # Trade-off: this also blocks YOUR OWN audits (Screaming Frog, Ahrefs # Site Audit, Semrush, etc.). Competitors can still reconstruct your # backlink profile from other sites that link to you, so this mostly # blinds your own tooling. If you run your own audits, temporarily # remove the relevant line (e.g. AhrefsSiteAudit / SemrushBot-SA) or # verify site ownership in that tool to bypass robots.txt. # ------------------------------------------------------------ User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: SemrushBot-BA Disallow: / User-agent: SemrushBot-SI Disallow: / User-agent: AhrefsBot Disallow: / User-agent: AhrefsSiteAudit Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: MajesticSEO Disallow: / User-agent: MozBot Disallow: / User-agent: rogerbot Disallow: / User-agent: spbot Disallow: / User-agent: SEOkicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: ScreamingFrogSEOSpider Disallow: / User-agent: serpstatbot Disallow: / User-agent: seoscannerbot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: LinkpadBot Disallow: / User-agent: Exabot Disallow: / User-agent: SiteAuditBot Disallow: / User-agent: ContentKingBot Disallow: / User-agent: Seekport Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: BomboraBot Disallow: / # ------------------------------------------------------------ # CONTENT SCRAPERS, HOSTILE CRAWLERS & UNWANTED SEARCH — BLOCKED # ------------------------------------------------------------ User-agent: HTTrack Disallow: / User-agent: WebCopier Disallow: / User-agent: WebReaper Disallow: / User-agent: SiteSucker Disallow: / User-agent: Offline Explorer Disallow: / User-agent: NinjaBot Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: WebBandit Disallow: / User-agent: EmailWolf Disallow: / User-agent: ExtractorPro Disallow: / User-agent: ChinaClaw Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Zeus Disallow: / User-agent: PetalBot Disallow: / User-agent: Sogou Disallow: / # ------------------------------------------------------------ # SITEMAPS # ------------------------------------------------------------ Sitemap: https://www.757king.com/sitemap.xml # ------------------------------------------------------------ # NOTES # ------------------------------------------------------------ # - Disallow blocks CRAWLING, not INDEXING. A disallowed URL can still # appear in results if linked elsewhere. For anything that must stay # out of the index, use or auth, # not just Disallow. # - Subdomains (cma.757king.com, 757king.idxbroker.com, etc.) need their # own robots.txt OR noindex meta tags. GHL may not support subdomain # robots.txt — use page-level noindex in GHL settings where needed. # - Pair this file with a curated /llms.txt (a markdown map of your key # pages for LLMs) for maximum AI discoverability. # - After deploying, validate with the Google Search Console robots.txt # tester and re-check that /find-a-home/ behaves the way you intend.