In brief, llms.txt is an open-source initiative, aimed at enabling online publishers to achieve higher content visibility to GAI, while providing more control over AI access to it.
The Story behind
The development of the llms.txt concept has in many ways been fueled by the accelerated evolution of generative AI assistants and AI agents and AI-based search algorithms, which have transformed the entire online search, hence significantly affecting publishers’ referral traffic in mostly negative ways.
These tectonic changes have forced publishers to look for new ways to improve their visibility to AI search crawlers, while restricting their access to particular, mostly copyrighted editorial content pieces, or at least keeping such access under control.
In view of this, the idea of llms.txt implementation is currently perceived in tandem with the adoption of IAB Tech Lab’s LLM Content Ingest API, designed to enable publishers to monetize LLM’s access to their content.
What’s Inside
From a technical perspective, llms.txt refers to a markdown file, located in the root folder of a publisher’s domain, which includes human- and LLM-readable information on how GAI can and should access specific content within particular digital properties.
The only required section of the file is H1, which includes the website title, whereas other sections, dedicated to key website sections and/or other essential information about it, are optional. These should be added as a markdown list with the markdown link/s in the following format: [link name](url), with : and the notes about the link, if any, added after.
To improve the llms.txt effectives, it’s recommended to use the clear and concise language, make link descriptions informative and terse, and test out the file’s readability by LLMs using one of the specifically designed tools prior to publishing it.
Perspectives of Adoption
Undoubtedly, the introduction of llms.txt itself is a positive step, marking the industry’s move in the right direction for the new digital era.
However, likewise as with many other open-source standards, its market-wide adoption largely depends on how eager the leading AI-driven agents are to embrace it — which, as of Q3 2025, remains unclear.