Gated Content and AI Search: What Google and ChatGPT Read Behind a Login, and What a Soft Gate Changes

Gated content can be cited in AI search without being given away, because a crawler reads only what a server sends it, and the publisher decides what the server sends to each reader. We put that decision in the content model, where a marketing team can see it and change it.

A soft gate splits a gated page into three layers held in separate CMS fields: a preview for every visitor, an answer layer written to be read and quoted, and the full detail for members. AI search crawlers receive the answer layer. Google receives either the full member page, with paywall structured data marking the gated part, or the same answer layer every visitor sees.

Can Google and ChatGPT read content behind a login?

Google and ChatGPT read the response a server sends to their request, and a request that carries no member session receives the signed out response unless the site is built to send more.

A sign in overlay or a blur is presentation. When the detail sits in the response underneath, anyone can read it in the page source.

Search Console shows what Google received. For the URL Inspection tool, the help page says: "To see information about the request, including the HTTP request and response, and the returned HTML, click View crawled page."

Being cited on a preview is not owning the answer

Citation on a preview page alone is possible, and we have observed it. On 1 subscription site we audited in November 2026, the signed out visitor, a request identifying as Googlebot and a request identifying as GPTBot all received the same preview HTML, and those preview pages were cited widely in AI answers.

The paid detail behind the unlock prompt was not in the page at all, and the pages carried no paywall markup.

We read that as a strong start with a limit: a preview holds the topic and little of the answer, so in our view a citation earned on a preview credits the site for a question it did not answer on that page.

What is a soft gate?

Editors set what a reader receives in the content model, and the server applies it. Each layer of the gated page sits in its own CMS field. Exhibit 1 sets out the three layers. Google publishes conditions for how it receives gated content, which the next section covers.

Exhibit 1: A soft gate decides which layer each reader receives

Layer	Who receives it	What it holds	Where it lives
Preview	Every signed out visitor	The topic, the scope and the invitation to join	A public field
Answer layer	AI search crawlers; Google and every visitor under the lead-in design	A direct, attributable answer to the question the page serves	Its own field, written to be quoted
Full detail	Signed in members, and a verified Googlebot under the full access design	The complete resource people pay or register for	A member field that never enters a signed out visitor's response

Source: Dotfusion soft gate model, November 2026.

Contentful is our example here, and in our experience the approach carries to any headless CMS. Contentful's documentation says: "Each space has a content model that represents the content types you create." Three fields on the gated content type make the three layers explicit.

The answer layer is written to stand alone. It is accurate, it names its source, and it answers the question without replacing the product. See also our guide to answer engine optimization.

Is showing crawlers more than visitors cloaking?

Showing Google gated content that signed out visitors do not see is not cloaking when Google's two stated conditions are met. Google's spam policies, last updated August 28, 2026, say: "we don't consider this to be cloaking if Google can see the full content of what's behind the paywall just like any person who has access to the gated material".

The same sentence states the second condition: following Google's Flexible Sampling guidance. The paywall markup page gives the mechanics: "Set the isAccessibleForFree property to false to specify that this section is behind a paywall." The gated section is identified by a class name, and "Only use .class selectors for the cssSelector property."

Google has tied paywalls to structured data since 2017. On October 2, 2017, Google's Flexible Sampling announcement said: "Sites with paywalls are strongly encouraged to apply the new structured data to their pages, because without it, the paywall may be interpreted as a form of cloaking, and the pages would then be removed from search results."

We read Google's wording as leaving two designs inside it:

• Full access for Google. A verified Googlebot receives the member page with the gated section marked. Google's AI features page names the controls for what Search then displays: "use nosnippet, data-nosnippet, max-snippet, or noindex controls".
• Lead-in for everyone. The answer layer is published to every visitor. Google's Flexible Sampling guidance describes "lead-in, which offers a portion of an article's content without it being shown in full". Every reader receives the same page.

A third design gives Googlebot the answer layer while holding the full detail back from Google. Google's condition is written as full content, so we settle that design against the published wording before any build.

How do you serve the right layer safely?

Serving the right layer safely takes four mechanics: verify who is asking, decide on the server for each request, label the gated section, and cache by reader type. We describe each at a general level here.

1. Verify who is asking. A user agent string is a name a request gives itself. Google's verification page, last updated March 20, 2026, describes a reverse DNS lookup on the accessing IP address, or matching that address against Google's published IP ranges, for cases where others are "claiming to be from Google". OpenAI and Perplexity each point site owners to published IP ranges.
2. Decide on the server. The server chooses the layer for each request before it sends anything, so member detail never reaches a browser that has not signed in.
3. Label the gated section. The paywall markup above identifies the gated section for Google, alongside the full content condition.
4. Cache by reader type. A cache that stores one copy of a page will serve a crawler's copy to a visitor. MDN's reference on the Vary header states the principle: "Including a Vary header ensures that responses are separately cached based on the headers listed in the Vary field."

Verification lists move. On March 31, 2026, Google announced a new location for its crawler IP range files and wrote: "We will eventually phase out the old locations and redirect them to the new ones within 6 months." We did not check whether the old locations have since been retired. We give any system that reads those files a named owner.

Which AI crawlers do you allow, and which do you block?

We sort AI crawlers into three types and set the allow or block decision per type: training crawlers, search crawlers, and fetchers that visit a page when a person asks a question. OpenAI and Anthropic give robots.txt controls for training and search crawlers, and Perplexity for its search crawler. For the third, OpenAI and Perplexity say robots.txt rules may not apply or are generally ignored, and Anthropic says its bots honor robots.txt.

Cloudflare's bot categories use the same three behaviours. Search is "Crawling to build search indexes or RAG databases." Agent is "User-directed agents visiting a page on behalf of a human." Training is "Crawling to train or fine-tune models."

Exhibit 2: Two AI vendors say robots.txt may not bind fetchers acting for a person

Type	Documented agents	What the owner's page says
Training	GPTBot, ClaudeBot, Google-Extended	OpenAI provides a robots.txt tag for GPTBot. Google-Extended is a robots.txt token covering Gemini training and grounding, and Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
Search	OAI-SearchBot, PerplexityBot, Claude-SearchBot	OpenAI: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers".
Fetch for a person	ChatGPT-User, Perplexity-User, Claude-User	OpenAI: "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

Source: OpenAI, Perplexity, Anthropic and Google crawler documentation, read November 9, 2026.

A fetcher visits a page to answer the person who asked, so we treat anything in a signed out response as public and protect member detail on the server, not in robots.txt.

One change is announced and not yet in effect. Google's crawler page, updated October 30, 2026, says Google intends to support the AIPREF shared training preference signal in early 2027.

We also check licence terms before writing an answer layer. Where a page carries third party content, those terms decide what any layer holds.

What does each leader do first?

Each leader has one first action, and none of the four needs a build.

• CMO. Name the 10 gated pages that matter most to revenue and record what a signed out request receives on each.
• CDO. Add the GA4 AI Assistant channel to the trial, registration and membership reports this month.
• CIO. Ask how a request claiming to be Googlebot is verified today, and who maintains the IP range lists.
• COO. Assign one owner for crawler policy and for the licence review of third party content.

See also our enterprise content operations guide.

How do you measure whether AI citations bring members?

We measure whether AI citations bring members with 2 reads set against the site's own trial, registration and membership events: referrals from AI assistants, and visibility inside Google's AI features. The three tools below differ in scope.

• GA4. The default AI Assistant channel covers arrivals from sources like ChatGPT, Gemini and Copilot. Google's page says: "It excludes Google's AI Overviews and AI Mode." GA4 counts those under Organic Search.
• Microsoft Clarity. The AIPlatform and PaidAIPlatform channel groups count sessions arriving from AI platforms. Microsoft's page says: "AI features embedded in search engines or productivity tools aren't included."
• Search Console. The generative AI performance report, rolled out worldwide as of August 31, 2026, shows "how many times links to your site were shown to a user in a generative AI feature on Google Search". We use it as a visibility read.

See also our AEO measurement framework.

A team can test its own position in fifteen minutes:

1. Open a gated page signed out, view the source and search for a sentence from the gated text.
2. Have a developer request the page with Googlebot's and GPTBot's user agents and compare the responses.
3. Search the source for isAccessibleForFree.
4. Run URL Inspection in Search Console and read the HTML Google received.

On the second check, matching responses mean the site sends those requests the signed out page, as a site that verifies crawlers by IP address does. Differing responses mean the site treats the user agent string as a signal. A response that carries more of the page than the signed out version means the site trusts that string.

See also our AEO readiness audit.

Start with one page

A proof of concept on 1 gated page turns this strategy into something a team can see. We set the before and the after side by side, showing what search engines, AI assistants, visitors and members each receive. That is how we start this work with clients.

See also our piece on Google's Universal Commerce Protocol.

Method: every statement about a third party in this post comes from a page we opened on November 9, 2026, and every quotation was checked by script against a saved copy of that page. The audit observation is 1 site in November 2026 and is not a sample. Chris Bryce is CEO and founding partner of Dotfusion, a Contentful Silver tier partner.

What do teams ask about gated content and AI search?

Is serving Google more than visitors cloaking?

Serving Google more than visitors is not cloaking when Google can see the full gated content the way a member does and the page follows Google's Flexible Sampling guidance. Paywall structured data, with isAccessibleForFree set to false on the gated section, helps Google tell a paywall apart from cloaking.

Can ChatGPT read content behind a login?

ChatGPT reads only what the server returns to its request. OpenAI documents separate agents, including OAI-SearchBot for search and ChatGPT-User for pages visited when a person asks a question.

Do I block AI crawlers from member content?

Decide per crawler type. Robots.txt manages training crawlers such as GPTBot and search crawlers such as OAI-SearchBot. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, so we protect member content on the server, where the response is decided.

If an AI assistant can read my member content, can it give it away?

When an assistant fetches a page, it reads the response the server sends. A fetcher such as ChatGPT-User visits a page to answer the person who asked, so we treat anything in a signed out response as public. Member detail stays out of every response except a verified Googlebot's under the full access design.

What is the first step toward a soft gate?

The first step toward a soft gate is finding out what each reader receives today: a signed out visitor, Googlebot, an AI crawler and a member. The second step is proving the approach on 1 gated page, with the before and the after side by side.

Sources

1. Google Search Central: Subscription and paywalled content markup, last updated September 8, 2026.
2. Google Search Central: Spam policies for Google web search, last updated August 28, 2026.
3. Google Search Central: Flexible Sampling general guidance, last updated December 10, 2025.
4. Google Search Central Blog: Enabling more high quality content for users, October 2, 2017.
5. Google: Verifying requests from Google crawlers and fetchers, last updated March 20, 2026.
6. Google Search Central Blog: New Location for the Google Crawlers' IP Range Files, March 31, 2026.
7. Google: Common crawlers, including Google-Extended, last updated October 30, 2026.
8. Google Search Central: AI features and your website, last updated December 10, 2025.
9. Search Console Help: URL Inspection tool.
10. Search Console Help: Generative AI performance report.
11. Google Analytics Help: Default channel group.
12. OpenAI: Overview of OpenAI crawlers.
13. Perplexity: Perplexity crawlers.
14. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
15. Cloudflare: Verified bot categories.
16. Microsoft Clarity: AI channel groups, last updated September 23, 2025.
17. MDN Web Docs: Vary header, last modified November 21, 2025.
18. Contentful: Data model.

All pages opened November 9, 2026.

This post was conceived and written by Chris Bryce with our stack of AI research agents doing the heavy lifting on sources and data. If you want to talk about gated content, AEO, or content operations, talk to us. We love this stuff.