If you run a B2B SaaS or professional services company targeting North America, Europe, and Australia, your website is likely built with React, Vue, a Headless CMS, or a heavily customized WordPress. When a user visits your product page, they can see the feature overview, service scope, FAQ, pricing, and industry solutions. However, when your team views the page source, uses a curl request, or submits the page to a text extraction tool, they might only get the title, page container, navigation, and footer.
What a person can see is not the same as what a crawler system can reliably access.
I’ve noticed many B2B companies expanding overseas start by rewriting articles, adding keywords, or increasing page count when troubleshooting GEO issues. But when a website relies on JavaScript to load core information, the problem may not be content quantity, but whether the content can be read consistently. In such scenarios, I usually start by examining the gap between "user-visible content" and "crawlable content" before deciding whether to adjust the tech architecture, page templates, or content structure.
Why a Page Looks Normal but the Crawled Result is Just an Empty Shell
The initial HTML from the server doesn't contain the main body
When a user visits a webpage, the browser first requests the URL from the server. Ideally, the returned HTML includes the product positioning, service description, FAQ, and key conclusions. However, on some React, Vue, or Headless CMS sites, the server might only return the title, an empty container node, and JavaScript file paths.
The product introduction, pricing, and service scope are only written into the page after the browser downloads and executes the scripts, and calls the content APIs. A standard browser usually completes this process, but some crawler systems may not execute all scripts or wait for all asynchronous requests to finish. The final page they get is just the outer shell.
Content appears only under specific conditions
Some pages load the body text, but it only appears after certain conditions are met. For example, the user must scroll to a certain area, click a tab, expand a collapsible module, or accept cookies before the script proceeds to request the content.
Pages might also return different content based on login status, local storage, browser language, or region. Lazy-loaded modules only start fetching content once they enter the viewport. If a crawl request can’t trigger these conditions, it may only get incomplete content.
Security configurations deliver different results to different visitors
CDN, WAF, or Bot management strategies might return different pages depending on the User-Agent. A standard browser gets the full content, but automated requests might face a CAPTCHA, a challenge page, a blank response, or a 403/429 status code.
Content APIs may also have CORS restrictions, authentication requirements, rate limits, or regional rules. A team testing in their own browser doesn’t prove that requests with no cookies, from different regions, or with different User-Agents will get the same body text.
It’s important to note that an HTTP 200 status code doesn’t guarantee the body text was successfully returned. A page being in the search index doesn’t mean every paragraph can be reliably extracted. A page being accessible doesn’t mean its structure supports accurate understanding.
If your website is rarely cited by ChatGPT, Gemini, Google AI Mode, Perplexity, or Google AI Overview, accessibility could be a factor. However, you shouldn’t rely solely on citation changes to diagnose technical issues. Citation results are also influenced by query phrasing, page quality, external sources, and platform updates.
Don’t Start by Changing Content; First Identify Where the Content is Lost
In our methodology, we typically diagnose this type of issue in this order: server response, raw HTML, JavaScript & CSS resources, rendered DOM, content API, security policies, and server logs.
The key isn’t to determine which framework the site uses, but to pinpoint where the core information appears and where it disappears. Only by narrowing down the problem area can the development and GEO teams choose the appropriate solutions.
Check 1: Is the content returned directly from the server?
First, get the raw response of the core URLs and search for product names, service types, FAQ questions, or a paragraph of recognizable body text.
If the raw HTML already contains the main content, the issue might be with text parsing, page structure, or security configuration, not necessarily client-side rendering. If the raw HTML only has an empty container, you need to check the JavaScript and content API.
Check 2: Does the content load without any interaction?
Open a new incognito window, clear cookies, don’t log in, scroll, click tabs, or dismiss pop-ups. Observe if the product description, service scope, and FAQ still appear automatically.
If core information requires scrolling, clicking, or cookie consent to load, crawler systems may not be able to get this content reliably. You should prioritize adjusting the default loading behavior of this content.
Check 3: Are resources and APIs returning data stably?
Open your browser’s developer tools and inspect the JavaScript, CSS, fonts, images, and content APIs. Focus on identifying 4xx, 5xx, CORS errors, request timeouts, or unusual loading order.
Also, check if the content API requires a Token, Session, login state, specific headers, or local storage data. Sometimes the page framework loads successfully, but the API responsible for the body text doesn’t respond, leaving you with an empty page.
Check 4: Do different requests get different pages?
Request the same URL using different User-Agents, cookie-less environments, and different regional or network conditions. Then compare the status code, response body size, amount of body text, and number of resource requests.
If a normal browser gets the full page, but an automated request gets a verification page or a significantly smaller response body, you should prioritize checking your CDN, WAF, and Bot management rules.
Check 5: Once technically accessible, is the content worth citing?
Technical accessibility is just the baseline. The page also needs to clearly communicate its product positioning, service scope, target audience, geographic coverage, use cases, and differentiators.
If key conclusions only exist in images, Canvas elements, videos, or interactive components, the crawler system may still not understand them. The page also needs clear titles, paragraphs, lists, Q&A structures, and entity information, ensuring the body text isn't overwhelmed by navigation, footer, and template elements.
Five Actions for Diagnosis and Remediation: From Identifying the Problem to Restoring Content Extractability
Action 1: Cross-Reference Raw HTML, Rendered DOM, and Plaint Text Body
Generate three results for the same URL: the server’s initial raw HTML, the DOM after browser execution, and the plain text body after removing navigation, styles, and scripts.
The raw HTML can be saved using curl, the browser’s View Source feature, or a crawling tool. The rendered DOM can be inspected in the developer tools’ Elements panel or generated using a headless browser that supports JavaScript. The plain text body is used to check if the main information on the page can be extracted consecutively.
Search for the same set of identifying points across all three results, such as product names, service types, FAQ questions, and key conclusions.
If the raw HTML lacks body text but the rendered DOM has it, it means the content is primarily client-generated. If the rendered DOM has the body text, but the text extraction is still missing pieces, the problem might be with page structure, hidden attributes, or template noise. If none of the three results contain the content, you need to investigate the content API, permissions, and regional logic.
This cross-referencing prevents confusing "missing body text," "scripts not executing," and "unclear body structure" as the same issue.
Action 2: Check HTTP Responses and Crawling Control Directives
Confirm that core URLs stably return a 200 status code. Check for multiple redirects, soft 404s, login redirects, region redirects, or security verification pages.
Some pages might appear to load, but an automated request gets led to a different version. For example, a visitor might be redirected to a region selector, a cookie consent page, or a login page, resulting in a different body text than what a normal user sees.
Next, check if robots.txt incorrectly restricts pages, script directories, or API paths. Then check the meta robots tag on the page and the X-Robots-Tag in the response headers. Test desktop, mobile, and cookie-less requests separately; don’t just verify one environment.
This action is primarily used to locate issues where crawling is blocked, requests are rewritten, or core resources are inaccessible.
Action 3: Diagnose JavaScript, Resource Files, and Content APIs
In the Network panel, filter for JS, Fetch, and XHR requests. Check if the scripts and APIs the body text depends on return data stably.
During the check, note any 4xx, 5xx, CORS errors, and loading timeouts. Pause scrolling, clicking, and cookie consent actions, and observe if the body text API still makes a request automatically. Loading the page with a slower network connection can also help determine if content is missing due to insufficient wait time.
If the site uses a Headless CMS or custom content API, check if the API requires a login state, Token, Session, or specific headers. Some sites display fine in a browser because the user has previously left cookies or local storage data, but new crawl requests won’t have these.
You can also temporarily disable some third-party scripts to see if errors from ads, tracking, or chat widgets are blocking subsequent rendering. The page body might depend on multiple resources; if one critical script fails, the browser might still show the page framework, but the body text won't appear.
Action 4: Move Content That Needs to Be Understood into a Server-Outputtable Form
Product positioning, service scope, use cases, FAQ, pricing information, and key conclusions should ideally be present in the initial HTML, or accessible through a stable rendering method.
Based on your website architecture, you could choose server-side rendering (SSR), static site generation (SSG), incremental static regeneration (ISR), or pre-rendering for key pages. For sites using a Headless CMS, set up body text fields that can be output directly into the HTML. If a full site overhaul isn't feasible, start by providing stable, static content versions for your most important product, service, and FAQ pages.
Content adjustments must happen in parallel. Don’t put key conclusions only in images, Canvas elements, or videos. Don’t hide primary explanations behind pop-ups or tabs that require a click. Pages should use clear H1, H2, paragraphs, and lists.
FAQ pages can use appropriate structured data based on the actual content, but structured markup cannot replace the page's body text. Product and service pages can also supplement with accurate schemas like Organization, Product, Service, Article, or BreadcrumbList that match the real content.
The goal here isn't to cater to one specific platform, but to ensure consistent and recognizable core information is available from different access environments.
Action 5: Check CDN, WAF, and Server Logs
Request the same page with different User-Agents, cookie-less settings, and from different geographical locations. Compare the status codes, response body sizes, and amount of body text between a normal browser and an automated request.
Focus on identifying CAPTCHAs, challenge pages, 403/429 errors, blank responses, or content degradation. Then, review your CDN and WAF configurations, checking Bot management rules, rate limits, IP rules, and regional policies.
Server logs can provide further insight into what actually happened during the request. Check the request time, source, User-Agent, status code, response size, and API calls. This can tell you if the crawl request reached the server, successfully returned body text, or failed at a specific resource request stage.
For crawl requests confirmed as legitimate traffic, you can set more reasonable rules. However, it’s not recommended to disable security features entirely just to improve accessibility. The better approach is to first identify which requests are being blocked incorrectly, and then make targeted adjustments.
After Technical Fixes, Don’t Just Test if the Page Opens
A technical fix isn’t the finish line. After adjustments, you need to monitor three layers: page response, content extraction, and AI citations.
Layer 1: Page Response Monitoring
It’s recommended to log HTTP status codes, redirect count, response time, HTML response body size, and failures of key scripts and APIs. Also, compare response differences across different User-Agents, cookie states, and geographic locations.
If the status code is normal but the response body suddenly shrinks significantly, or key APIs start timing out, it could indicate the body text is missing again.
Layer 2: Content Extraction Monitoring
Set a fixed checklist for each core page. This includes whether the product positioning can be extracted, if the service scope is complete, if FAQ questions and answers are present, if the heading hierarchy is clear, and if the body text is overwhelmed by navigation and template content.
Also, compare the initial HTML with the rendered result to ensure the main information is consistent between the two. Conduct spot checks on a set of core URLs weekly, and run regression tests after any template, CMS, or front-end component changes.
Layer 3: AI Citation Monitoring
Build a query set around real user questions. For example, "What types of businesses is [a certain] B2B SaaS suitable for?", "What does [a specific] professional service include?", "What’s the difference between Solution A and Solution B?", or "What criteria should be evaluated when choosing a vendor?"
For overseas markets, observe whether ChatGPT, Gemini, Google AI Mode, Perplexity, and Google AI Overview start referencing your website’s product pages, service pages, FAQ pages, or industry solution pages.
Platform responses are influenced by query phrasing, region, time, and available sources. Therefore, monitoring should focus on trends, cited page types, and content gaps, rather than guaranteeing consistent appearance.
If the HTML is complete but text extraction is still poor, adjust the titles, paragraphs, and body structure. If the body text depends on an API, prioritize fixing API permissions and server output. If only requests from certain regions fail, check CDN and regional rules. If the page is extractable but rarely appears in related queries, supplement with problem-oriented content, comparative explanations, use cases, and factual evidence.
Observable Milestones After Fixing Such Technical Issues
The following ranges are methodological projections based on industry implementation processes, not fixed outcomes. Typically, server response and body text extraction improve first, while changes in AI mentions and page citations take longer to appear.
| Observation Dimension | Typical State Before Optimization | Reasonable State Range After Optimization | Typical Observation Period |
|---|---|---|---|
| Raw HTML Content | Mainly page framework, title, or empty containers | Key product, service, FAQ, and scenario descriptions are accessible in the initial HTML or via stable rendering results | 1—4 weeks |
| Page Response Stability | Some requests result in blank pages, resource failures, security challenges, or different versions | Core pages stably return 200; abnormal responses and content discrepancies gradually decrease | 2—8 weeks |
| Body Text Extraction Completeness | Crawl results are dominated by navigation, footer, and generic template elements | Main titles, service descriptions, FAQs, and key conclusions can be extracted consecutively | 2—8 weeks |
| AI Mentions & Page Citations | Website pages are rarely cited in highly relevant queries | Some related questions begin citing product pages, service pages, FAQ pages, or industry solution pages | 8—12 weeks or longer |
| Team Diagnostic Efficiency | Repeatedly changes content or relies on single tests when issues arise | Can narrow down problems based on server responses, HTML, resources, APIs, security policies, and logs | 4—12 weeks continuous improvement |
Weeks 1—3 are typically used for systematic checks of crawling, rendering processes, and server configuration. Weeks 4—8 are typically used to complete SSR, SSG, or pre-rendering adjustments for core templates. Weeks 8—12 or longer are needed to observe content extraction stability and AI citation changes.
If the website has complex templates, a large number of pages, or involves multi-region deployment, the timeline may extend further. Technical accessibility is a necessary condition, not a guarantee of citation results. Page content quality, query relevance, and external information consistency will also influence subsequent performance.
Rendering Issue Diagnosis Checklist for B2B Global Websites
- Use curl or a similar tool to get the raw HTML of core URLs. Search for product names, service descriptions, and FAQ text.
- Compare the browser's rendered DOM to determine if missing content is server-returned or client-generated.
- Re-test the page in a cookie-less, non-logged-in environment without scrolling or clicking.
- Check pages, JavaScript, CSS, and content APIs for 4xx, 5xx, CORS errors, or timeouts.
- Review
robots.txt,meta robots, andX-Robots-Tagto ensure pages and key resources aren't incorrectly restricted. - Request the page with different User-Agents and from different geographic locations. Compare status codes, response sizes, and body text amounts.
- Move product positioning, service scope, FAQs, and key conclusions into the initial HTML or a stably renderable body area.
- Check CDN, WAF, and server logs to confirm legitimate crawl requests aren't blocked by CAPTCHAs, rate limits, or regional policies.
- Establish weekly regression tests for core pages, recording response status, body text extraction, and AI citation page changes.
- After technical fixes, re-evaluate the content structure. Supplement with problem-oriented titles, clear answers, factual evidence, and use cases.
When your website looks fine in a browser but crawlers only get the page framework, adding more articles may not solve the problem. A better approach is to first select 5—10 core product, service, and FAQ pages. Confirm whether the content is missing during the server response, raw HTML, resource loading, rendering, content API, or security policy stage.
If you're evaluating whether to pursue GEO, start with a baseline diagnosis—either manually or using our free Audit. Our team has years of practical experience in the GEO field. We'll organize the issue list from the perspectives of technical accessibility and content comprehensibility, but we won't make fixed promises about citation results on any specific platform.
Related Questions
If a page is indexed by Google, why might AI tools still fail to retrieve the body text?
Indexing usually just means a page was discovered and processed at some point; it doesn't guarantee all body text can be reliably obtained across different crawl environments. If main content depends on JavaScript, API permissions, or user interactions, some systems might only read the page framework or incomplete text.
Do React or Vue websites necessarily need to be switched to server-side rendering?
Not necessarily. First, check if the core content is already in the raw HTML, if the rendering is stable, and if the API and security policies are normal. Only if client-side generation consistently impacts content extraction should you evaluate options like SSR, static generation, or pre-rendering.
Why does curl return empty content while the page looks complete in a browser?
By default, curl does not execute JavaScript like a browser. If the server only returns a page container and the body text is generated by scripts calling an API, curl will get very little content. This indicates a need to investigate the rendering method, but you cannot conclude the entire page is uncrawlable based on a single request.
If robots.txt doesn't block crawling, why is the page still extracted incompletely?
robots.txt is just one check point. The page could also be affected by meta robots, X-Robots-Tag, API authentication, CORS configuration, lazy loading, cookie conditions, or CDN/WAF policies. Investigation needs to combine resource requests and server logs.
After fixing JavaScript rendering, how long before I see changes in AI citations?
Technical accessibility and body text extraction improvements can usually be verified in stages within 1—8 weeks. Changes in AI citations may take 8—12 weeks or longer and are influenced by query relevance, page quality, platform updates, and other available sources. Therefore, focus on trends rather than specific counts.
How should GEO teams and development teams divide the work?
The GEO team is responsible for defining the pages, key content, and test queries, and for establishing a baseline. The development team is responsible for checking server output, resource requests, API permissions, rendering methods, and security configurations. Both teams should then collaborate on template and content adjustments based on the monitoring results.