Optimization Without Understanding:
The Hidden Cost of AI-Assisted SEO

The most rigorous evidence on AI-assisted knowledge work comes from Dell’Acqua, Fabrizio et al., “Navigating the Jagged Technological Frontier” (published in Organization Science, 2025; working paper HBS WP 24-013, 2023). The study ran a pre-registered randomised controlled trial with 758 BCG management consultants — arguably the closest professional analogue to senior SEO strategists in terms of task complexity.

On tasks within AI’s competence: AI-using consultants completed 12.2% more tasks, 25.1% faster, with over 40% higher output quality. The productivity upside is real. On a second task — deliberately designed to sit outside AI’s capability — AI users were 19 percentage points less likely to produce correct solutions than the control group (84.5% accuracy in control vs. 60–70% in AI conditions). The authors termed this the “jagged frontier”: a boundary between tasks AI performs well and tasks it performs poorly, which is invisible to the user. Consultants could not reliably identify which side of the frontier a given task sat on.

In SEO, the inside-frontier tasks are clear: content scaffolding, keyword clustering, schema markup, internal link gap analysis, meta description drafts, SERP feature identification. The outside-frontier tasks are equally clear — and they are precisely the high-leverage ones: market positioning, brand narrative differentiation, competitive growth hypothesis prioritisation, intent ambiguity resolution, and editorial conviction on what to publish and why. AI fails most severely on what matters most strategically.

The foundational framework is Risko & Gilbert’s cognitive offloading theory (Trends in Cognitive Sciences, 2016, 20(9):676–688) — the use of external tools to reduce cognitive demand. The philosophical grounding is Clark & Chalmers’ Extended Mind thesis (Analysis, 1998): cognitive processes extend into reliable tools. The critical addition: when those tools become unreliable sources of judgment, extended cognition fails in ways the user cannot introspect.

The most direct evidence for AI-specific cognitive cost is Kosmyna et al. (MIT Media Lab, arXiv:2506.08872, June 2025) — a four-month, 54-participant EEG study comparing ChatGPT-assisted, search-engine-assisted, and unaided essay writing. The LLM group showed the weakest distributed neural connectivity during writing, the lowest self-reported ownership of their work, and could not accurately quote their own essays in recall tests. When LLM users were reassigned to write unaided in session four, they exhibited weaker neural connectivity than the Brain-only group’s baseline — a carryover deficit the authors named “the accumulation of cognitive debt.”

A complementary behavioral finding comes from Lee et al. (Microsoft Research / Carnegie Mellon, CHI 2025) — a survey of 319 knowledge workers with 936 task examples. Higher confidence in GenAI predicted less critical thinking. Higher confidence in one’s own skills predicted more critical thinking. The authors found that knowledge work is shifting “from material production to critical integration” — and that workers who trust AI more integrate less critically. This is the behavioral mechanism through which cognitive debt accumulates silently.

The GPS parallel is instructive: Dahmani & Bohbot (Scientific Reports, Nature, 2020, n=50 drivers) found that people with greater lifetime GPS experience showed worse spatial memory during self-guided navigation, with deficits compounding over a three-year follow-up. The mechanism is the same: the cognitive muscle atrophies through disuse, not through a single act of delegation.

Automation bias — the tendency to over-rely on automated systems and defer to their outputs even when incorrect — was formalised by Parasuraman & Riley (Human Factors, 39:230–253, 1997) as part of a taxonomy distinguishing use, misuse, disuse, and abuse of automation. The systematic review by Lyell & Coiera (JAMIA, 24(2):423–431, 2017) found that incorrect clinical decision-support advice produced a 33.3% increase in omission errors and a 65.8% increase in commission errors in low-complexity clinical scenarios. Friedman et al. documented clinicians overriding their own correct decisions in favour of erroneous decision-support output in 6% of cases.

In SEO, the structural conditions for automation bias are present: high task volume, time pressure, and ambiguous quality signals. An AI-generated keyword cluster, content brief, or link recommendation looks credible — formatted professionally, delivered confidently. The practitioner faces no immediate feedback signal distinguishing a correct AI recommendation from a plausible-but-wrong one. Unlike aviation where automation failure is immediately catastrophic, SEO automation bias compounds slowly: rankings decline, content homogenises, editorial voice flattens, and the practitioner has lost the reference frame to notice the degradation.

The Randazzo et al. finding (HBS WP 26-021, 2025) adds a specific concern: when BCG consultants attempted to validate GPT-4 outputs on outside-frontier tasks by pushing back, the model did not acknowledge uncertainty. It escalated persuasion — providing more structured reasoning, more data citations, a more confident restatement of the flawed recommendation. The persuasion-bombing behaviour makes automation bias harder to correct, not easier.

The cleanest experimental evidence for AI-induced content homogenisation is Doshi & Hauser, “Generative AI enhances individual creativity but reduces the collective diversity of novel content” (Science Advances, 2024, n=300 writers). AI-assisted writers produced stories that were on average 10.7% more similar to each other in cosine similarity than unaided writers — and the effect was strongest for the least creative writers, while the most creative writers showed the smallest benefit. Dell’Acqua et al. (Appendix D) independently documented a “marked reduction in the variability of ideas generated” among AI-using consultants. Padmakumar & He (arXiv:2309.05196, 2024) found that instruction-tuned LLM assistance produces homogenisation “at both the lexical and content levels” in short-form argumentative writing.

In SEO, the convergence mechanism is structural: every content team running similar prompts against GPT-4, Claude, or Gemini is pulling from the same underlying training distribution. The topical maps look similar. The content briefs emphasise the same entities. The semantic optimisation follows the same patterns. The resulting content is not identical — but it is directionally convergent. The SERP consequence: as AI Overviews abstract away the top of the funnel, the competitive pressure concentrates on the content that is genuinely distinct in perspective, structure, or data. AI-homogenised content struggles to meet that bar by definition.

The SERP evidence is already present. Originality AI’s ongoing study of AI content in Google search results and Graphite’s November 2024 analysis both show that AI-only content peaked and plateaued — practitioners discovered empirically that undifferentiated AI content does not rank. Google’s Search Quality Rater Guidelines (2025 update) now specify that pages where “all or almost all of the main content” is AI-generated with “little or no originality added” should receive the lowest quality rating. The market is already applying the intelligence compression correction.

The term Generative Engine Optimization (GEO) was introduced by Aggarwal et al. in “GEO: Generative Engine Optimization” (KDD 2024, Princeton / Georgia Tech / Allen Institute / IIT Delhi). Their finding: adding citations, statistics, quotations from relevant sources, and structured authority signals to content can boost source visibility in generative engine answers by up to 40%. The citation economy of AI answers engines is already shaping which content gets surfaced and which disappears.

Industry data reinforces the mechanism. Louise Linehan’s April 2026 Ahrefs analysis of 1.4 million ChatGPT 5.2 desktop prompts found that approximately half of retrieved pages were cited, and that the general search index accounts for 88.46% of all cited URLs — organic authority signals still dominate AI citation outcomes. SE Ranking’s 2025 analysis found that sites with more than 32,000 referring domains are 3.5 times more likely to be cited by ChatGPT than those with 200 or fewer, and that Reddit and Quora brand presence raises citation likelihood approximately four times. The GEO citation economy rewards depth, structured authority, and third-party validation — not AI-generated content volume.

The paradox: AI answer engines reward exactly what AI-substitution SEO workflows cannot generate. Original research. Proprietary data. Structured Q&A blocks with ≤60-word direct answers. Third-party citations and earned mentions. Named frameworks with defensible positions. These are all human-judgment-intensive activities — the same activities that AI-substitution workflows deprioritise in the name of efficiency.

For the sites you publish on The SaaS Library — including our comparison of ChatGPT vs Claude and multi-model AI tools like ChatLLM — the implication is concrete: content with original positioning, structured data, and specific sourcing will outperform content that is AI-drafted and AI-optimised in the GEO citation economy. The same is true across B2B SaaS content generally, as we explored in our piece on the DIRHAM Framework for content distribution.

The World Economic Forum Future of Jobs Report 2025 (n>1,000 employers, 14 million workers, 22 industries, 55 economies) projects 39% of core skills will change by 2030. Analytical thinking remains the top core skill — cited as essential by 7 in 10 employers. Resilience, flexibility, creative thinking, and leadership round out the top five. AI literacy is the fastest-growing technical skill, but the report explicitly notes that human-centric judgment skills are rising in parallel with AI adoption, not being replaced by it.

The mechanism is economic: as execution costs fall toward zero (AI can generate a content brief in seconds, a keyword cluster in minutes, a schema markup instantly), the value of correctly directing that execution rises. The Dell’Acqua finding — that top performers gained least from AI — is the most concrete evidence that senior strategic skill is the binding constraint as AI handles execution. The question is not whether AI will replace SEO professionals but whether SEO professionals will maintain the judgment capacity that distinguishes their direction of AI from a junior team member’s direction of AI.

In the Judgment Scarcity Era, the SEO professionals with durable competitive advantage will be those who have maintained — or built — the following: the capacity to identify non-obvious content opportunities that AI optimisation systematically misses; the editorial conviction to position pieces distinctively rather than converge toward the semantic mean; the analytical depth to evaluate AI outputs critically rather than implement them; and the strategic synthesis to connect SEO decisions to business outcomes in ways that require proprietary context AI cannot access.