Careers Site SEO and GEO: The Technical Setup Guide for AI Search

Twenty-four technical checks that affect how search engines and AI tools find, read and use a careers site, from hygiene to expert level. Built on Google, Microsoft, OpenAI, Anthropic, Apple and Perplexity documentation, with the new AI training opt-out explained.

By James Robbins 42 min read
Person working on a laptop displaying Google Search Console's website indexing report.
The technical decisions behind a careers site increasingly determine where its jobs and employer information can be found.

Key points

  • Tests of ChatGPT, Claude and Perplexity found they could not read content that only appears through JavaScript. Job titles, descriptions and pay belong in the HTML the server sends.
  • Providers use different mixes of crawlers, usage tokens and product settings. A rule aimed at one leaves the others untouched, and user-triggered fetchers may ignore robots.txt.
  • A shared robots.txt token for opting out of AI training, announced on 30 October, takes effect at Microsoft from 1 February 2027 and at Google in early 2027. Once Google implements it, a site that uses it without an explicit Google-Extended rule will also lose Gemini grounding.
  • Google says a site must be included in the Search generative AI control in Search Console to be eligible for AI Overviews and AI Mode, and a careers property can inherit an exclusion from its parent.
  • GA4 now has a native AI Assistant channel. Clicks from Google's AI Overviews and AI Mode stay in Organic Search, and neither console reports clicks from AI answers.
  • Google says Search ignores llms.txt. Ownership, rendering, markup, expiry and measurement matter far more.

On 30 October, Google, Microsoft and OpenAI said they would honour a single robots.txt line that tells AI developers not to train models on a website's content. Anthropic, whose engineer drafted the proposal, has started implementing it. TechRadar's report on 8 November called it "a request rather than a lock". Most coverage treated it as a story about news publishers and copyright. For employer brands, it is a reminder that a handful of configuration files, settings and templates now shape whether search engines and AI tools can find, read and use a careers site at all.

EBN's guide to generative engine optimisation (GEO) for employer branding set out the strategic case: candidates increasingly use AI tools alongside careers sites and reviews, and AI answers can only draw on evidence about an employer that is easy to find and check. This guide covers the plumbing underneath. It is written for teams building or rebuilding a careers site, and for employer brand leads who need to brief IT, the applicant tracking system (ATS) vendor and the agency with confidence.

None of this guarantees a citation (Google says as much about its own requirements, and the same goes for every AI product). What the checks do is remove the technical reasons a careers site gets left out. Every technical claim below comes from the companies' own documentation, checked in November 2026, unless it is marked as a test, a quote or EBN's own view.

Start here

Five questions cover the basics.

  • Who owns it? Name the owner of every careers domain, the ATS configuration, the firewall and the search consoles (check 1).
  • Can a crawler reach and read the jobs? Ordinary links to every job, and the job content in the HTML the server sends (checks 2 to 4).
  • Is anything blocking AI crawlers? The robots.txt file as actually served, the firewall and the Search Console AI setting (checks 6 and 8).
  • Do closed jobs disappear? Expired requisitions removed or marked as expired the same day (check 10).
  • Can the team see the results? The AI reports in both search consoles, and AI referrals followed through to completed applications (checks 12 to 14).

The guide runs in four levels: Level 1, hygiene (checks 1 to 8), Level 2, standard (9 to 14), Level 3, advanced (15 to 20) and Level 4, expert (21 to 24), followed by what to skip and a checklist with owners.

How AI tools reach a careers page

AI companies now separate three uses of web content:

  • Training: content changes what a future model learns. The effect is slow and diffuse, and it applies to models trained after the content was collected.
  • Grounding, or retrieval: an AI assistant fetches pages while answering a question, then summarises, quotes or links them. This is where a candidate sees a careers page cited.
  • Search indexing: the classic index of links, which also feeds AI answers in Google and Bing.

A fourth category sits alongside: fetchers triggered by a person, such as a candidate pasting a job link into ChatGPT. Providers handle these uses with different combinations of crawlers, usage tokens and product settings, and a rule aimed at one leaves the others untouched.

CompanyCrawler or tokenWhat it doesEffect of blocking it in robots.txt
GoogleGooglebotGoogle Search, including AI Overviews and AI ModeGoogle stops crawling the content. URLs can still be indexed from links; noindex keeps pages out
GoogleGoogle-Extended (token, no crawler)Gemini model training, plus grounding in Gemini Apps and Vertex AINo effect on Search inclusion or ranking
MicrosoftBingbotBing search, Copilot and Microsoft's grounding servicesBing stops crawling the content, with the same indexing caveat. Microsoft's AI preferences leave Bing search eligibility unchanged
OpenAIGPTBotCrawls content that may be used to train OpenAI's foundation modelsSignals that content should not be used for training
OpenAIOAI-SearchBotSurfaces websites in ChatGPT's search featuresSite not shown in ChatGPT search answers, though it can still appear as navigational links
OpenAIChatGPT-UserFetches pages for actions users take in ChatGPT; not automatic crawlingrobots.txt rules may not apply
AnthropicClaudeBotCollects content that may contribute to model trainingFuture content excluded from training
AnthropicClaude-SearchBotIndexes content to improve search results in ClaudeMay reduce visibility in Claude's search results
AnthropicClaude-UserFetches pages when a user asks Claude a questionMay reduce visibility for user-directed searches
PerplexityPerplexityBotSurfaces and links websites in Perplexity results; not used to train foundation modelsReduces visibility in Perplexity
PerplexityPerplexity-UserVisits pages to answer a user's questionGenerally ignores robots.txt
AppleApplebotSearch in Spotlight, Siri and Safari; data may help train Apple's modelsApple stops crawling the content for its search features
AppleApplebot-Extended (token, no crawler)Controls whether Applebot data is used for trainingPages can still appear in Apple search results

Source: crawler documentation from Google, OpenAI, Anthropic, Perplexity and Apple, and Microsoft's 30 October 2026 announcement. Checked November 2026.

Two details in that table catch teams out. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt at all, because a person asked for the page. And Google-Extended and Applebot-Extended are tokens with no crawler of their own: they tell Google and Apple how to use what Googlebot and Applebot have already collected.

Level 1: Hygiene. What every careers site needs now

1. Name the owners

The settings in this guide are spread across robots.txt, page templates, HTTP headers, firewall dashboards, two search consoles and the ATS vendor's platform. The employer brand team rarely owns any of them. Before anything else, write down who can change each one. The checklist at the end of this guide includes a likely owner for every check, to be confirmed inside each organisation.

2. Make every job page crawlable and indexable

Open /robots.txt on the main domain and on every domain or subdomain that carries jobs, including ATS-hosted pages, and check that job and careers paths are not disallowed. Then check the pages themselves for a noindex robots meta tag. Google's JobPosting guidelines require that job seekers can see the details "without the need to login", and suggest running a live job through the URL Inspection tool in Search Console to confirm Googlebot can reach it.

A job page can serve perfect HTML and still go undiscovered if the only way to reach it is a search form, a filter or a "load more" button that runs JavaScript. Google's guidance on crawlable links says it can reliably follow only links written as an <a> element with an href attribute. Careers hubs, team pages, location pages and paginated job lists should link to each job that way, and the pagination itself should use ordinary links.

4. Put the content in the HTML

Most AI systems read the raw HTML a server sends. A December 2024 study by Vercel and MERJ found that "none of the major AI crawlers currently render JavaScript", naming OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot and Meta's crawler. Gemini, through Googlebot, and Applebot were the exceptions. The data came mainly from Next.js sites, including two UK job boards, Resume Library and CV Library.

Newer tests tell much the same story for the big US assistants. In June 2026, Andre Alpar gave 12 assistants each a unique URL whose real value only loaded through JavaScript, then checked his server logs, Search Engine World reported. ChatGPT, Claude, Gemini, Perplexity and Meta AI fetched only the HTML and reported a decoy value. Copilot fetched the script without running it. Five assistants, including DeepSeek and Mistral, ran it and used the real value. "Not one major American assistant ran the JavaScript and used the result," he wrote. In October, Glenn Gabe asked five AI products for the first and last paragraphs of JavaScript-rendered pages. ChatGPT, Claude and Perplexity failed. Gemini and Copilot read most of the test pages, which Gabe attributes to Google's and Bing's search systems.

And it is changing. In September, one site operator, Hisashi Space, reported log patterns suggesting that OpenAI's crawlers had started rendering pages on his Next.js sites, with requests checked against OpenAI's published IP ranges. He noted that OpenAI had announced nothing, and OpenAI, Anthropic and Perplexity do not say in their crawler documentation whether their crawlers run JavaScript. Browser-based agents, such as ChatGPT's agent or Claude in Chrome, load pages the way a person's browser does, and no public test has yet measured what they see on careers sites.

Analysis: whatever the crawlers do next, the safe design has not changed: the job title, description, location, pay, employer name and the evergreen pages on culture, benefits and hiring process should be in the HTML the server sends. Careers sites built as single-page applications, including some ATS-hosted job boards, can serve an almost empty page to anything that does not run JavaScript.

A quick screening test saves the page as the server sends it, reports the final status code and address after any redirects, and then searches for the job title:

curl -sL -o page.html -w "%{http_code} %{url_effective}\n" https://careers.example.com/jobs/12345
grep -c "Senior Data Engineer" page.html

A zero means look closer. It can also mean a redirect, an error, a bot challenge or a title split across tags. And finding the title is only a start; open page.html and check the description, location and pay are there too.

5. Give every job one URL and clean sitemaps

Google's JobPosting documentation says a sitemap for job postings "must include the canonical page for each job posting", should use "as accurate a time as possible" for lastmod, and should not include "search results pages, list pages, or other dynamic pages". That rule applies to the job sitemap. Useful team, location and role-family pages belong in the site's main sitemap.

When the same job appears on the careers site and on an ATS domain, the team should choose which version is canonical, mark the other with a rel="canonical" link to it, and make internal links and sitemaps point to the chosen version. Google's documentation treats a canonical tag as a strong signal, so consistency across those signals matters. Translations are a separate case, covered in check 20: each language version keeps its own canonical URL.

6. Check the security layer and the robots.txt actually served

A robots.txt file that welcomes AI crawlers means little if a firewall turns them away. Cloudflare, which sits in front of a large share of corporate websites, announced in July 2025 that new customers would block AI crawlers by default, Search Engine Roundtable reported. Its current documentation, updated on 2 November 2026, says that for new domains, search, agent and training crawlers are allowed by default. A careers domain's settings therefore depend on when it was set up and what IT has changed since.

In Cloudflare, the setting lives under Security, then Settings, filtered by bot traffic, in "AI Crawler and Agent access policies". Search, agents and training can be set separately, with one catch: Cloudflare says "Training actions also apply to mixed-purpose crawlers used for both Search and Training". When Bot Preference Sync is on and a policy blocks something, Cloudflare's managed robots.txt prepends its own rules to the site's file, including named groups that disallow crawlers such as GPTBot, ClaudeBot and Google-Extended.

The practical test is to open /robots.txt in a browser and read the file as served, including any Cloudflare-managed block at the top. The file the web team maintains may not be the whole story. Other content delivery networks and bot-protection tools have similar controls, and a rising count of 403 errors from AI crawlers in server logs is the usual sign of a block.

7. Verify the site with Google and Bing

In Google Search Console, a domain property verified through DNS covers every subdomain, including a careers subdomain. Bing Webmaster Tools can import sites already verified in Search Console. Submit the careers sitemaps to both. Without verification, the console reports and settings described in this guide are unavailable, along with Google's Indexing API.

8. Leave the AI switches on

Three settings can quietly remove a careers site from AI answers:

  • Search Console's Search generative AI control. Google's AI optimisation guide says a site must be included in Search generative AI features in Search Console to be eligible for them. The control reached all websites by 31 August, and Google says it "will not be used as a ranking signal". Include is the default, but Google's help page adds that "a property inherits its Search generative AI control from its closest parent". A careers property can therefore inherit an exclusion set on the main domain. Check the effective setting on every relevant property.
  • Snippet controls. Google's robots meta tag documentation says nosnippet also stops content being used as direct input for AI Overviews and AI Mode, Search Engine Journal reported. Careers templates should not carry nosnippet or max-snippet:0.
  • HTTP headers. Robots rules can also arrive in an X-Robots-Tag header, invisible in the page source. A quick check:
curl -sIL https://careers.example.com/jobs/12345 | grep -i x-robots-tag

Level 2: Standard. Machine-readable, accurate and measured

9. Mark up every job with JobPosting, to Google's rules

Google's JobPosting documentation lists five required properties: datePosted, description, hiringOrganization, jobLocation and title. It lists seven more as recommended, several of which become required in particular cases:

PropertyWhen it applies
validThroughRequired for postings that have an expiry date. Leave it out when the job has none.
jobLocationRequired for jobs with a physical workplace. Can be left out for a fully remote role when applicantLocationRequirements is given.
jobLocationTypeSet to TELECOMMUTE, and required, when the job is 100% remote. The description must say so.
applicantLocationRequirementsFor fully remote roles, the preferred way to name at least one eligible country. Without it, Google uses the country in jobLocation.
directApplyTrue when the candidate can apply through a short process on the site, or when the posting gives direct contact details for the hiring company or its representative. Google is still developing its use.
baseSalaryThe actual salary from the employer, never an estimate.

Source: Google's JobPosting structured data documentation, checked November 2026.

The rules that trip employers up most:

  • One job per page. The markup "must only be used on pages that contain a single job posting", and never on job listing or search results pages.
  • Markup matches the page. It "must appear on the same page as the job description that job seekers can read in their browser."
  • Clean titles. The title property should not include "job codes, addresses, dates, salaries, or company names".
  • Real salaries. baseSalary is "the actual base salary for the job, as provided by the employer (not an estimate)". Ranges use minValue and maxValue, and unitText must be HOUR, DAY, WEEK, MONTH or YEAR.
  • Remote means fully remote. Google says not to mark up jobs that allow occasional home working as remote. Hybrid roles keep a physical jobLocation.

A complete example for a single role:

<script type="application/ld+json">
{
  "@context": "https://schema.org/",
  "@type": "JobPosting",
  "title": "Senior Data Engineer",
  "description": "<p>The full job description, exactly as shown on the page.</p>",
  "identifier": { "@type": "PropertyValue", "name": "Example Co", "value": "REQ-1234" },
  "datePosted": "2026-11-09",
  "validThrough": "2026-12-31T23:59",
  "employmentType": "FULL_TIME",
  "hiringOrganization": {
    "@type": "Organization",
    "name": "Example Co",
    "sameAs": "https://www.example.com",
    "logo": "https://www.example.com/logo.png"
  },
  "jobLocation": {
    "@type": "Place",
    "address": {
      "@type": "PostalAddress",
      "streetAddress": "1 Example Street",
      "addressLocality": "Singapore",
      "postalCode": "018956",
      "addressCountry": "SG"
    }
  },
  "baseSalary": {
    "@type": "MonetaryAmount",
    "currency": "SGD",
    "value": { "@type": "QuantitativeValue", "minValue": 9000, "maxValue": 12000, "unitText": "MONTH" }
  },
  "directApply": true
}
</script>

Test each template with Google's Rich Results Test before launch. Structured data matters beyond Google's job search features. Fabrice Canel of Microsoft confirmed at SMX Munich that schema markup helps Microsoft's large language models understand content, Search Engine Land reported. Google's position is narrower: its AI features documentation says no special schema is needed for AI Overviews or AI Mode, and lists "making sure your structured data matches the visible text on the page" as good practice.

10. Expire closed jobs properly

Google says "jobs that are no longer open for applications must be expired" in one of three ways: set validThrough to a date in the past, remove the page so it returns a 404 or 410 status code, or remove the JobPosting markup. A job filled early should be removed. Failing to expire jobs "may result in a manual action", Google's term for a penalty applied by its reviewers. EBN's advice: expire a closed requisition the day it closes. (Google sets no deadline.)

The Vercel and MERJ data shows why tidy URLs matter beyond Google: in that study's sample, ChatGPT's crawler spent 34.82% of its fetches on 404 pages and Claude's 34.16%, against 8.22% for Googlebot. Many were requests for outdated asset files, so this is no measure of expired jobs. The point stands: AI crawlers burn a lot of effort on dead addresses, and current sitemaps and accurate redirects cut the waste.

Analysis: the strongest pattern for AI visibility pairs short-lived job pages with long-lived evergreen pages. Individual requisitions should expire cleanly. Pages for teams, locations, role families, pay philosophy, benefits and the hiring process should stay at stable URLs, so search engines and AI tools have something accurate to use after any single job closes.

11. Describe the employer as an entity

Organization markup on the corporate or careers home page tells search engines which company the careers site belongs to. Google's Organization documentation covers properties such as name, url, logo and sameAs, which links to the employer's official profiles elsewhere. The same names, logo and facts should appear on the careers site, LinkedIn and review platforms. Inconsistent details make it harder for search engines and AI tools to identify the employer and reconcile what different sources say about it.

Two markup traps are worth knowing:

  • EmployerAggregateRating is off-limits to most careers sites. Google's rules say "users must be able to post their own ratings on your site and your site must host those user ratings." Copying a Glassdoor or Indeed score into careers-site markup breaks those rules, and violations may lead to a manual action.
  • Estimated salary markup is retired. On 12 June 2025, Google announced it was phasing out Estimated Salary results along with six other structured data features, saying the displays were "no longer providing significant additional value for users". It removed the type from Search Console reporting and the Rich Results Test in September 2025. Google says the change does not affect ranking, or use of the markup outside Google Search. For Google, pay belongs in baseSalary on each job.

12. Measure AI referrals in analytics

Since 13 May 2026, Google Analytics 4 has included an AI Assistant channel in its default channel group. Google assigns the medium ai-assistant "when the referrer matches a recognized AI Assistant", naming ChatGPT, Gemini and Claude as examples. Google's channel definitions say the channel "excludes Google's AI Overviews and AI Mode", which stay in Organic Search.

The sequence for most teams: open the AI Assistant channel first, check which sources it captures for the careers site, and add a custom channel group only where the team needs extra sources, history from before May or a different definition. GA4 applies full matching to regular expressions by default, so a pattern needs wildcards to match part of a source name. Google's own custom channel example uses .*chatgpt.*. A starting point, to be tested against the property's actual session sources:

.*(chatgpt|openai|perplexity|claude|copilot|gemini).*

Some AI apps send no referrer, and attribution can break without warning. Adobe's Cedric Huesler reported that a ChatGPT bug stripped tracking tags from outbound links for about a week in October 2026, and that the referrer was still missing on most ChatGPT clicks after the fix, Relevant Audience reported. Part of AI-driven traffic will always land in Direct.

13. Follow the candidate into the ATS

Visits are only the first step. For a careers site, the useful measures are apply-button clicks, application starts, completed applications and, where the ATS can report it, qualified applications by source. When the careers site sends candidates to a separate ATS domain, the journey breaks unless analytics is set up across both domains. GA4's cross-domain measurement requires both domains in the tag configuration and the ATS to allow the tag. Redirects between the two can drop the linking information, so the full journey should be tested. Without it, every application looks like a new visit from the careers site, and the AI referral that started it disappears.

14. Turn on the AI reports in both consoles

Google Search Console. Since 31 August, the Search generative AI performance reports have been available to all sites. They show impressions in AI Overviews, AI Mode and Discover's AI features by page, country, device and date. As Barry Schwartz of Search Engine Roundtable noted, they include "no click data and no query data", and sites with little AI visibility may not see the report at all.

Bing Webmaster Tools. The AI Performance report has been in public preview since February. It counts how often a site's pages are cited as sources in Copilot, Bing's AI summaries and some partner services, shows which pages are cited, and lists a sample of the "grounding queries" AI systems used to find them. In June, Microsoft added intent and topic groupings, Citation Share, "the percentage of citations attributed to your site" for a grounding query, and Compare, which overlays a previous time period. Microsoft says Citation Share "does not expose competitor domains". Bing does not report clicks from AI answers.

Neither tool covers ChatGPT, Claude or Perplexity. For ChatGPT, OpenAI's Business Hub, in beta since 10 November, gives verified businesses 28-day counts of mentions, website clicks and discovery topics, without prompts or conversations. For the rest, the prompt audits and monitoring tools described in EBN's GEO guide remain the main option.

Level 3: Advanced. Speed and precise control

15. Push job changes to Google with the Indexing API

Google's Indexing API is one of the few direct lines into its crawler, and jobs are one of only two content types allowed to use it: it "can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject". For job URLs, Google's JobPosting documentation recommends the API over sitemaps because it prompts Googlebot to crawl sooner, while still recommending a sitemap for the whole site.

A URL_UPDATED notification announces a new or changed job, including a closed job whose page stays live with a past validThrough or with its markup removed. A URL_DELETED notification applies once the page returns a 404 or 410 or carries a noindex tag. The default quota is 200 publish requests a day per project, covering both types, and the API is free. Setup needs a Google Cloud project and a service account added as an owner of the Search Console property, and every call carries an OAuth access token. Google warns that its spam policies apply to submitted content and that sites should not "circumvent our submission limits, such as by using multiple accounts". A successful call means Google got the request, nothing more. It "may" recrawl.

POST https://indexing.googleapis.com/v3/urlNotifications:publish
Content-Type: application/json
Authorization: Bearer <ACCESS_TOKEN>

{ "url": "https://careers.example.com/jobs/12345", "type": "URL_UPDATED" }

Analysis: employers that open and close more than 200 roles a day should prioritise notifications for new roles and closures, and leave minor edits to the sitemap.

16. Push changes to Bing and others with IndexNow

IndexNow does a similar job for Bing and several other search engines. The site hosts a key file, preferably at its root, then sends changed URLs, up to 10,000 in a single request. The key can also sit elsewhere on the same host, named in a keyLocation field, which helps where an ATS limits access to the root. Participating engines "agree that submitted URLs will be automatically shared with all other participating search engines". The protocol's FAQ lists endpoints for Bing, Naver, Seznam.cz, Yandex, Yep and Amazon. Google does not appear in its documentation.

POST https://api.indexnow.org/indexnow
Content-Type: application/json; charset=utf-8

{
  "host": "careers.example.com",
  "key": "a1b2c3d4e5f6a7b8",
  "keyLocation": "https://careers.example.com/a1b2c3d4e5f6a7b8.txt",
  "urlList": [
    "https://careers.example.com/jobs/12345",
    "https://careers.example.com/jobs/12346"
  ]
}

A 200 response means the submission was received, and a 202 means the key is still being validated. Neither means the page is indexed; check that in Bing Webmaster Tools.

Freshness matters for answers about live vacancies and current employment policies. "Gen AIs value fresh content in particular, partly as a reference check of their LLM training data," Canel wrote in the discussion reported by Search Engine Land, recommending IndexNow. Microsoft's October announcement adds that IndexNow may also speed up how quickly new AI preferences are picked up, without a guaranteed date.

17. Write robots.txt deliberately

The standard behind robots.txt, RFC 9309, contains a rule many teams miss. A crawler follows only the groups that name it, and falls back to the User-agent: * group only if no group matches. So a site that adds a User-agent: GPTBot group must repeat in it any Disallow lines from the wildcard group, or GPTBot will ignore them. The reverse also holds: a permissive wildcard group does not cancel a stricter named group, including one added by Cloudflare.

A robots.txt block is also narrower than many teams assume. Google's robots.txt introduction says a blocked URL can still be indexed if other pages link to it, and that keeping a page out of Search needs noindex, which a crawler can only see if it is allowed to fetch the page. Nor does robots.txt protect anything: applicant accounts and confidential material need real access controls, such as a login.

A careers-friendly file keeps it simple:

# careers.example.com/robots.txt
# Search engines, AI search and AI training crawlers
User-agent: *
Disallow: /search/
Allow: /

# Optional: keep specific paths out of AI training.
# No group for this token means no preference is expressed.
# At Google, this will also remove Gemini grounding for these paths.
# User-Agent: __usage_ai-training
# Disallow: /internal-research/

Sitemap: https://careers.example.com/sitemap.xml

Cloudflare's managed file adds a different line, a Content-Signal, with three established values: search, ai-input (content used to answer questions, which is grounding) and ai-train. Its content signals policy launched in September 2025 with a default of search=yes, ai-train=no, and Cloudflare said customers had turned on its managed file for more than 3.8 million domains. Its current documentation also shows a fourth field, use, with values such as use=reference, which Cloudflare describes as an optional extension it "is testing". Google did not commit to honouring content signals when they launched. A careers site that writes its own signals should state search=yes and ai-input=yes. Cloudflare's documentation is blunt about the limits of every robots.txt approach: "robots.txt compliance is voluntary".

18. Set training and grounding preferences deliberately

The 30 October announcement centres on a policy token, __usage_ai-training, written as its own group in robots.txt. It comes from an Internet-Draft by Kevin Kelley of Anthropic, published within the IETF's AI Preferences work. Its key rules:

  • It governs use, not access. "This policy token is not a directive for crawling behavior," the draft says. Crawlers can still visit.
  • Silence means no preference. With no group for the token, "no preference has been expressed". A wildcard User-Agent: * rule does not count.
  • Private deals win. "Private agreements MAY override preferences expressed via this mechanism."

The status is modest. The IETF lists it as an individual draft with no formal standing, due to expire on 3 May 2027 unless revised. Anthropic says it began recognising the signal on 30 October and is rolling it out "over time and globally". OpenAI says "in the coming months", according to Nieman Lab. Microsoft will honour it from 1 February 2027. Google's announcement says it aims to support the control "in 2027", and its crawler documentation narrows that to "early 2027". Alissa Cooper, a former IETF chair who now runs the Knight-Georgetown Institute, argued in Tech Policy Press that if the biggest providers deploy it, the token "may become the de facto way" of signalling training preferences, before the standards process has finished.

Each provider adds its own wrinkle:

  • Google. In Google's announcement, John Mueller, a Google Search Advocate, wrote that when a site uses the shared token to opt out of training "and Google-Extended is not explicitly specified", Google "plans to interpret that as if the publisher had disallowed Google-Extended". Google-Extended also governs grounding in Gemini Apps and on Vertex AI, Google's platform for businesses building their own AI tools. So, once Google implements the token, one line added to keep content out of training could also remove a careers site from the Gemini answers some candidates read. The same post sets out the way back: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." At Google, Gemini training and Gemini grounding share a single switch. AI Overviews and AI Mode are unaffected: they sit behind Googlebot and the Search Console control in check 8, which Google says "doesn't affect AI training".
  • Microsoft. In its announcement, Krishna Madhavan, a principal product manager at Bing, set out two preferences. AI Training can be set in robots.txt with the shared token, or page by page with a meta tag or HTTP header, and the page-level setting overrides robots.txt. AI Grounding can only be set page by page: "There is no site-wide robots.txt grounding preference." Both "do not change eligibility for traditional Bing search results", and both default to allow.
  • Anthropic. Its crawler page says that where robots.txt names ClaudeBot for a path, that rule takes priority over the general training setting. Existing provider-specific groups should be checked before anyone assumes the shared token decides the outcome.

Sites that still rely on Bing's older NoArchive or NoCache tags face a transition. To keep a grounding restriction, Microsoft requires both new page-level grounding preferences and a temporary grounding preference recorded in Bing Webmaster Tools before 1 March 2027. Those temporary settings open on 15 December 2026 and expire when Bing recrawls the page or on 30 September 2027, whichever comes first. There is no temporary training setting. And once Microsoft detects any new training or grounding preference on a site, it disregards the remaining NoArchive and NoCache tags on that site, so a partial migration can remove restrictions the team meant to keep.

Analysis: for public careers content, the case for allowing training is simple. Future models form a picture of an employer before anyone searches, and the employer's own material can only shape that picture if it is in the mix. There is no guarantee it will be used. An opt-out, meanwhile, only covers content collected after it is published; it cannot undo what earlier models learned. The case for limited opt-outs is narrower: proprietary research, client work or material licensed from others. The token supports path rules, so a whole-domain Disallow: / is rarely necessary. A site that wants to opt out of training at participating providers while staying in Gemini's answers can pair the token with an explicit Google-Extended rule, at the price of allowing Gemini training too:

# Opt out of AI training at participating providers...
User-Agent: __usage_ai-training
Disallow: /

# ...but stay eligible for Gemini grounding (this also allows Gemini training)
User-agent: Google-Extended
Allow: /

One exception sits outside robots.txt. Under OpenAI's Businesses in ChatGPT Terms (section 3), a business that supplies a product feed agrees that text and data mining opt-outs by it and its affiliates have no effect on OpenAI. A careers-site training opt-out can be set aside by another team's feed decision, so whoever owns the opt-out should know whether anyone in the group supplies one.

19. Use page-level controls for exceptions

Most careers pages should carry no AI restrictions. Where a single page or passage needs one, page-level controls are more precise than robots.txt. Google supports data-nosnippet on specific HTML elements, and its AI features documentation warns that changes to preview controls can take "several days to several months" to take effect as pages are recrawled. From February 2027, Bing will read these tags and headers:

<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">

X-Robots-Tag: Bingbot: ai-training:disallow

The second line keeps grounding in Microsoft's own experiences, including Copilot, and withholds the page from grounding services Microsoft provides to other companies, such as Web IQ and API access in Foundry. Microsoft warns that blocking grounding means "reduced brand visibility in AI experiences", and notes that "publishing a setting is not the same as activating it": the preference applies only once Bing recrawls the page.

20. Build for countries and languages

Multi-country careers sites should link language and country versions of the same page with hreflang annotations, so search engines show candidates the right version. Each version keeps its own canonical URL, the annotations should be reciprocal, and sitemaps and internal links should match. Location data in JobPosting markup must be accurate: Google lists "false location data that does not match the actual location of the job" among the practices it does not allow.

Level 4: Expert. Evidence, verification and governance

21. Read the server logs, and verify the bots

Server or CDN logs are the only place to see what crawlers actually fetch: which pages, how often and with what status code. For a server writing the common Apache or Nginx "combined" log format, this counts requests by crawler and status code:

awk -F'"' '{ split($3, a, " ");
  if (match($6, /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Applebot|Googlebot|bingbot/))
    print substr($6, RSTART, RLENGTH), a[1] }' access.log | sort | uniq -c | sort -rn

User agents can be faked, as Google's documentation points out, so serious analysis checks the requesting IP address against each company's published list:

The questions worth answering each month: are AI search crawlers reaching job pages and evergreen pages, what share of their requests hit 404s or redirects, and is anything returning 403s that should not be.

22. Mine Bing's AI data for content gaps

Bing's grounding queries are the searches AI systems ran when they cited the site. They are not what people typed, and they say nothing about who was asking. They do show what the AI went looking for. Since 13 October 2026, Bing's Topics Portfolio, in beta, groups those queries into topics and lists "Opportunity Queries", described as "the strongest queries across your whole portfolio where other sites appear and yours does not". It does not name the other sites. Microsoft says to "treat the numbers as directional signals for planning, not as a precise ledger", and topics with too little data are not shown, so smaller employers may see a short list or none.

Analysis: for most employers, the questions worth checking first cover pay, hybrid working, progression, interview stages and visa sponsorship. The expert workflow joins three sources. Grounding queries and Opportunity Queries show what AI systems look for. The prompt audits in EBN's GEO guide show what they currently say. Citation Share, compared across time periods, shows whether new pages change the site's share. The output is a content plan of evergreen pages, each answering one question with evidence, in server-rendered HTML.

23. Test the whole job lifecycle

Most failures appear between systems. A single test requisition exposes them: publish it, change its salary and location, then close it early. At each stage, check the visible page, the JobPosting markup, the sitemap entry and its lastmod, and the Indexing API and IndexNow notifications. Asking the ATS vendor or web agency to demonstrate this run is the quickest way to find out what the careers stack actually does.

24. Put AI preferences under change control

Analysis: four measures keep the settings in this guide from drifting. Keep a single register of every AI-related setting on every careers domain, with its owner. Require sign-off from the employer brand owner before any change that touches careers pages. Add a clause to ATS contracts requiring notice before changes to crawler rules. And re-run the Level 1 checks every quarter. The coming year brings several deadlines worth putting in that register:

  • 15 December 2026: temporary grounding settings open in Bing Webmaster Tools for sites using the older NoArchive or NoCache tags.
  • 1 February 2027: Microsoft starts honouring the new AI Training and AI Grounding preferences.
  • 1 March 2027: deadline to record temporary grounding preferences in Bing Webmaster Tools. After this date, Bing no longer reads NoArchive and NoCache as training or grounding preferences.
  • Early 2027: Google's planned support for the shared token.
  • 3 May 2027: the IETF draft expires unless revised.
  • 30 September 2027: Bing's temporary grounding preferences expire, if recrawling has not already replaced them.

What to skip

Google's AI optimisation guide, updated in July 2026, devotes a section to things site owners do not need to do for Google Search:

  • llms.txt. On machine-readable AI files, the guide says creating them "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Chrome's Lighthouse tool now includes an optional, unscored llms.txt check for AI agents browsing a site. Google's guidance covers Google Search only, but for a careers site the file sits at the bottom of the list.
  • Chunking and AI rewrites. "There's no requirement to break your content into tiny pieces," the guide says. Clear headings and self-contained answers help candidates and machines alike.
  • AI-specific schema. There is no special markup for Google's AI features. JobPosting and Organization, done properly, are enough.
  • Pages built to capture AI queries. Mass-producing pages "primarily to manipulate rankings or generative AI responses" falls under Google's scaled content abuse policy.

What the specialists are watching

Search specialists have been cautious about switching off AI search features. Cyrus Shepard opted his site out of Google's AI search features for 28 days, then back in, and estimated an AI search click-through rate of about 0.9%, against about 0.4% for the site overall. It is an estimate (Search Console does not separate AI clicks), and "data from a single experiment on a single site", as he put it. It is also about AI search features, which is a separate question from training.

The other caution is about the standard itself. Nate Hake, a publisher involved in the IETF working group, has warned that the work is unfinished, PPC Land reported. Cooper's concern is the order of events: four companies have committed to a draft that the standards body has not yet adopted. For employer brands, that argues for patience on the training question and urgency on everything else in this guide. Nothing about the new token needs to change on careers sites today. Ownership, rendering, markup, expiry and measurement do.

The careers site configuration checklist

LevelCheckHow to testLikely owner
1. HygieneOwners named for domains, ATS, firewall and consolesA written list, agreed with each teamEmployer brand lead
1. HygieneJob and careers paths crawlable on every domain, including ATS-hosted pagesRead /robots.txt on each domain; run a live job through URL InspectionWeb team or ATS vendor
1. HygieneEvery job reachable through ordinary links, including paginationClick through hubs and list pages with JavaScript offWeb team or ATS vendor
1. HygieneJob content in the server-sent HTMLScreening test with curl, then view source for description, location and payWeb team or ATS vendor
1. HygieneOne canonical URL per job; job sitemap with accurate lastmodCanonical tags, internal links and sitemaps agree; no list pages in the job sitemapWeb team or SEO
1. HygieneNo AI crawler blocks at the CDN or firewallRead the served robots.txt; check CDN bot settings and 403s in logsIT or security
1. HygieneDomain verified in Search Console and Bing Webmaster ToolsBoth consoles show data for the careers subdomainSEO or web owner
1. HygieneSearch generative AI control effectively set to includeCheck every property, including inherited settingsSEO or web owner
1. HygieneNo nosnippet, max-snippet:0 or blocking X-Robots-Tag on careers templatesView source and curl -sIL on a job pageWeb team
2. StandardValid JobPosting markup on every job pageRich Results Test on each template; rich result reports in Search ConsoleWeb team or ATS vendor
2. StandardClosed jobs expired the day they close (EBN target)Page returns 404 or 410, carries a past validThrough, or loses its markupRecruitment operations and ATS vendor
2. StandardOrganization markup and consistent entity dataName, logo and profiles match across careers site, LinkedIn and review sitesEmployer brand and web team
2. StandardNo borrowed review scores in EmployerAggregateRating; pay in baseSalarySearch the templates for bothWeb team
2. StandardAI Assistant channel reviewed; custom group only where neededCheck captured sources; test any regex against session sourcesAnalytics team
2. StandardApply clicks, starts and completions tracked across the ATS domainRun a test application and follow it through GA4Analytics team and ATS vendor
2. StandardAI reports reviewed monthlySearch Console generative AI report; Bing AI PerformanceSEO or employer brand
3. AdvancedIndexing API notifications for new, changed and closed jobsQuota usage in the Google Cloud consoleWeb team or ATS vendor
3. AdvancedIndexNow notifications for job changesKey file in place; check indexing in Bing, beyond the 200 or 202 responseWeb team or ATS vendor
3. Advancedrobots.txt groups written deliberatelyNamed groups repeat the disallows they need; served file matches intentSEO and IT
3. AdvancedTraining and grounding preferences set on purposeShared token, Google-Extended, ClaudeBot and Bing preferences agreeLegal, IT and employer brand
3. Advancedhreflang across country and language versionsReciprocal annotations; each version keeps its own canonicalWeb team or SEO
4. ExpertMonthly log analysis with verified crawlersRequests by crawler, page and status, checked against published IP listsIT or SEO
4. ExpertBing grounding and opportunity queries mapped to evergreen contentA quarterly content plan alongside prompt auditsEmployer brand
4. ExpertWhole job lifecycle testedOne requisition published, edited and closed early, checked at each stageATS vendor, web team and recruitment operations
4. ExpertChange control for every AI-related settingA register with owners, employer brand sign-off and an ATS contract clauseEmployer brand lead

Compiled by EBN from the sources below. The Owner column shows who usually handles each check. Who does it in practice varies by organisation.


Takeaways

What technical setup does a careers site need to appear in AI answers?

Job pages must be reachable through ordinary links, crawlable, indexable and eligible for snippets, with their content in the HTML the server sends. The careers domain should not block AI search crawlers at the firewall, and Search Console's Search generative AI control should be set to include. JobPosting and Organization markup, clean job expiry and current sitemaps keep the content accurate and fresh. None of this guarantees a citation.

Can AI crawlers read careers sites built with JavaScript?

Often not. Tests in 2024 and 2026 found that ChatGPT, Claude and Perplexity did not use content that only appears through JavaScript. Gemini and Copilot read some JavaScript pages through Google's and Bing's search systems, and some other assistants ran the scripts. Behaviour varies and is changing, so careers sites should server-render job titles, descriptions, locations and pay.

What is the __usage_ai-training token?

A robots.txt policy token from an IETF Internet-Draft published on 30 October 2026. A site adds a group for it to express a preference against AI training on some or all paths. It does not block crawling. Microsoft will honour it from 1 February 2027 and Google plans to in early 2027. Once Google implements it, using it without an explicit Google-Extended rule will also remove grounding in Gemini Apps and Vertex AI.

Should employers opt careers pages out of AI training?

Rarely. Allowing training may make an employer's own careers content available to future models, though it does not guarantee the content is used. Public careers content is written to be found and repeated. Businesses that want to protect research or client material can use path-level rules for those areas only, and private material needs real access controls.

What JobPosting properties does Google require?

Google requires datePosted, description, hiringOrganization, jobLocation and title, though jobLocation can be left out for a fully remote role that names eligible countries. validThrough is required when a job has an expiry date, and jobLocationType is required for fully remote jobs. Each page should carry one job, and the markup must match the visible description.

How can employers measure AI visibility and traffic?

Google Search Console's Search generative AI reports show impressions in AI Overviews, AI Mode and Discover by page, country, device and date, without clicks or queries. Bing Webmaster Tools' AI Performance report shows citations, cited pages, grounding queries and citation share. GA4's AI Assistant channel shows visits from tools such as ChatGPT, Gemini and Claude. OpenAI's Business Hub adds 28-day counts of ChatGPT mentions, website clicks and discovery topics for verified businesses, without prompts or any view of who asked. Following those visits into the ATS shows whether they become applications.

Does a careers site need an llms.txt file?

Not for Google Search. Google's AI optimisation guide says Google Search ignores llms.txt and similar files, and that creating one neither helps nor harms visibility. Chrome's Lighthouse tool runs an optional, unscored check for it.

SOURCES

#PublisherSourceUsed for
1IETFdraft-kelley-aipref-training-preference-signal-00 (30 October 2026)Token syntax, definitions, status and expiry
2Microsoft BingYour content, your choice: clearer preferences for training and grounding, by Krishna Madhavan (30 October 2026)Bing training and grounding preferences, syntax, dates
3GoogleGoogle's common crawlers (updated 30 October 2026)Google-Extended scope, early 2027 support, crawler verification
4GoogleVerify requests from Google crawlersForward-confirmed reverse DNS
5Google Search CentralSupporting a shared AI training control for websites, by John Mueller (30 October 2026)Google-Extended treatment of the shared token, Gemini grounding, timeline
6PPC LandGoogle and Bing back one robots.txt rule to opt out of AI training (31 October 2026)Nate Hake
7Zyppy SignalWe opted out of Google AI search, by Cyrus Shepard (5 November 2026)Estimated AI search click-through rate
8Nieman LabAnthropic, OpenAI, Google and Microsoft will make it easier for publishers to opt out (2 November 2026)OpenAI timeline
9Tech Policy PressWhat the new AI training opt-out really means, by Alissa Cooper (4 November 2026)Standards-process critique
10TechRadarGoogle and Microsoft among those signing up to OpenAI's plan for a new website crawling standard (8 November 2026)Original coverage
11OpenAIOverview of OpenAI crawlersGPTBot, OAI-SearchBot, ChatGPT-User, IP lists
12OpenAIBusinesses in ChatGPT Terms (last updated 9 November 2026)Product feeds set aside text and data mining opt-outs (section 3)
13AnthropicDoes Anthropic crawl data from the web, and how can site owners block the crawler?ClaudeBot, Claude-SearchBot, Claude-User, training signal, IP list
14PerplexityPerplexity crawlersPerplexityBot, Perplexity-User, IP lists
15AppleAbout Applebot (4 September 2026)Applebot, Applebot-Extended, verification
16Vercel and MERJThe rise of the AI crawler (17 December 2024)JavaScript rendering, 404 rates
17Search Engine WorldDo AI assistants actually render your JavaScript when grounding? by Andre Alpar (9 June 2026)Twelve-assistant JavaScript test
18GSQiCan AI search platforms render JavaScript? by Glenn Gabe (19 October 2026)Five-platform JavaScript retest
19DEV CommunityTraffic from ChatGPT jumped! Is it because its crawlers now run JavaScript? by Hisashi Space (September 2026)Operator report of OpenAI rendering
20GoogleJob posting (JobPosting) structured data for Job SearchProperties, guidelines, expiry, sitemaps
21GoogleMake your links crawlableCrawlable links and pagination
22GoogleHow to specify a canonical URLCanonical tags
23GoogleIntroduction to robots.txtLimits of robots.txt blocking
24GoogleEmployer aggregate rating (EmployerAggregateRating) structured dataRating eligibility rules
25GoogleOrganization structured dataEntity markup
26Google Search CentralSimplifying the search results page, by Henry Hsu (12 June 2025, updated 8 September 2025)Estimated Salary retirement
27GoogleIndexing API: using the API and quota (updated 16 October 2026)Permitted content, notifications, quota
28GoogleIndexing API prerequisitesService account and OAuth setup
29IndexNowIndexNow documentation and FAQProtocol, batch size, participating engines
30Search Engine LandMicrosoft Bing/Copilot use schema for its LLMs, by Barry SchwartzFabrice Canel on schema and freshness
31GoogleAI features and your websiteEligibility, preview controls, reporting
32Google Search Console HelpSearch generative AI controlDefault, inheritance, no effect on training
33GoogleAI optimization guide (updated July 2026)Search Console eligibility, llms.txt, what to skip
34Chrome for DevelopersLighthouse llms.txt audit (5 May 2026)Optional llms.txt check
35Search Engine JournalGoogle updates robots meta tag document to include AI Modenosnippet and AI features
36GoogleNew opportunities, control and insights for website owners (updated 31 August 2026)Search generative AI control
37Google Analytics HelpDefault channel group and What's new (13 May 2026)AI Assistant channel definition
38Google Analytics HelpRegular expressions and custom channel groupsFull regex matching, custom channel example
39Google Analytics HelpSet up cross-domain measurementCareers site to ATS measurement
40Google Search CentralIntroducing Search generative AI performance reports in Search Console (June 2026)AI performance reports
41Search Engine RoundtableGlobal: Google Search Console AI performance report and AI control (1 September 2026)No click or query data
42Microsoft BingIntroducing AI Performance in Bing Webmaster Tools public preview (10 February 2026)Citations, grounding queries
43Microsoft BingNew AI visibility insights in Bing Webmaster Tools (16 June 2026)Citation Share, Compare, no competitor domains
44Microsoft BingIntroducing Topics Portfolio (beta) in AI Performance (13 October 2026)Topics and Opportunity Queries
45OpenAIIntroducing Businesses in ChatGPT (10 November 2026)Business Hub insights: mentions, website clicks, discovery topics
46Search Engine RoundtableCloudflare to block AI crawlers by default (1 July 2025)July 2025 default change
47CloudflareAI Crawler and Agent access policies (updated 2 November 2026)Current defaults and dashboard location
48Cloudflarerobots.txt setting (managed robots.txt) (updated 2 November 2026)Prepended rules, content-use test
49CloudflareGiving users choice with Cloudflare's new Content Signals Policy (24 September 2025)Content-Signal values, 3.8 million domains
50Relevant AudienceChatGPT tag and referrer loss (15 and 16 October 2026)October 2026 attribution bug, Cedric Huesler
51IETFRFC 9309: Robots Exclusion Protocol (September 2022)Group matching rules
52GoogleTell Google about localized versions of your pagehreflang
53Employer Branding NewsGEO for employer branding: how AI shapes candidate research (June 2026, updated November 2026)Strategy, prompt audits, monitoring