Local SEO Unlocked

LLMS.txt: The Secret Map for Teaching AI to Understand Your Website

• Don Phelps • Season 1 • Episode 18

Share episode

The LLMs.txt file is revolutionizing how websites interact with AI models by providing a curated list of key URLs that helps AI tools better understand website content. This emerging standard addresses a critical challenge in the age of generative AI, where language models often access information differently than traditional search engines.

• Plain text file in Markdown format that lives in a website's root directory
• Acts as a guide specifically for AI models like ChatGPT, Claude, and Gemini
• Points AI to the most important pages while keeping it away from sensitive content
• Part of the broader Generative Engine Optimization (GEO) trend
• Complements existing SEO tools like robots.txt and sitemap.xml but serves a unique purpose
• Can be implemented manually or through automated tools and plugins
• Provides brand accuracy, targeted visibility, reduced confusion, and selective disclosure
• Best practices include keeping the list focused, using clear descriptions, and testing regularly
• Benefits sites with valuable specific content like blogs, documentation, FAQs, and product pages
• Takes proactive control over how AI perceives and represents your digital presence

Consider what pages would make it onto your highlight reel for AI - which content do you want AI to understand perfectly every time?


Thanks for tuning in to Local SEO Unlocked! If you enjoyed today’s episode, don’t forget to subscribe, leave a review, and share it with others who want to master Local SEO. Stay connected with us weekly for more insights on SEO! Until next time, keep optimizing and stay ahead in local search!

Speaker 1:

The rise of AI has been well, absolutely incredible, hasn't it?

Speaker 2:

Oh, definitely.

Speaker 1:

It's really reshaping how we find information, how we interact online Completely. But you know, for anyone with a website, any online presence, it raises a big question how do we make sure these AI models actually understand our content, the stuff we put out there?

Speaker 2:

Yeah, how do you guide them, make sure they get it right?

Speaker 1:

Exactly how do you stop them misinterpreting your site?

Speaker 2:

It's a real challenge.

Speaker 1:

So today we're diving deep into something that sounds simple but is actually pretty powerful for tackling this. It's the lmstxt file.

Speaker 2:

Ah yes, the newcomer on the block.

Speaker 1:

We're going to unpack what it is, why it's becoming so important and really how it gives you, the listener, more control in this age of generative AI.

Speaker 2:

It's crucial too, because AI tools they don't work quite like the old search engines do they.

Speaker 1:

Right, that's a key point.

Speaker 2:

They often grab info more in real time, like when someone asks a question. They don't always have that deep stored index Google built over time.

Speaker 1:

So the old rules don't always apply.

Speaker 2:

Not entirely, and that's why new things like LMStxt are becoming essential. So, yeah, our mission today give you the shortcut to understanding this thing.

Speaker 1:

Okay, let's start right at the beginning. What is an LLMStxt file?

Speaker 2:

fundamentally, Okay, so at its heart it's really just a lightweight plain text document.

Speaker 1:

Plain text Simple enough.

Speaker 2:

Yeah, and it's written in Markdown, which, if you haven't used it, it's just a super simple way to format text Easy for people to read, easy for machines.

Speaker 1:

Okay, and its job.

Speaker 2:

Its main job is to help the large language models you know ChatGPT, claude Gemini. Those guys understand your website better.

Speaker 1:

By giving them a list.

Speaker 2:

Exactly A curated list of your most important pager, a kind of cheat sheet for your site.

Speaker 1:

Right Now. If you've dealt with websites, that might ring a bell. Sounds a bit like robotstxt.

Speaker 2:

You're definitely on the right track. It serves a similar kind of guiding purpose. Yeah, but, different, but different. Robotstxt is for traditional search engine crawlers, telling them where they can and can't go. Lmstxt is specifically for these AI tools that fetch content more dynamically.

Speaker 1:

And where does it live on the site.

Speaker 2:

Usually right in the root directory, so you know your dash websitecom, llmstxt and AI could potentially find it there, or you could even check yourself.

Speaker 1:

Okay, interesting. So why is this suddenly so important? Why the buzz?

Speaker 2:

Well, it gets down to how these AI models work, or rather their limitations. They're smart, but they often have a pretty short memory when they're browsing.

Speaker 1:

Short memory.

Speaker 2:

Yeah, they don't typically crawl and index your entire site and remember it all, like Google does.

Speaker 1:

They often just dip in for a specific user query okay, so they grab what they need in might miss other stuff exactly, and they can easily get tripped up by complex HTML or sites heavy on JavaScript things a human might navigate easily so without some guidance, important pages might just get ignored or misinterpreted yeah even if they look fine to us. That's the core problem this tries to solve so lmstxt steps in like a like a guide precisely.

Speaker 2:

It's like your site's concise, easy to read map specifically for these quick ai visits okay, and how does it help practically? Well, it directly points the ai to your key urls, your most valuable content.

Speaker 1:

So better understanding of site structure.

Speaker 2:

Definitely improves that and, crucially, it can also help prevent the AI from grabbing stuff you don't want it to see.

Speaker 1:

Like what.

Speaker 2:

Maybe proprietary information or old outdated pages, irrelevant sections you get to choose.

Speaker 1:

Okay, that makes sense. It's part of that bigger trend right.

Speaker 2:

Yeah.

Speaker 1:

Generative engine optimization GEO.

Speaker 2:

Exactly, it's a core piece of GEO optimizing specifically for how these generative AI models find and use information.

Speaker 1:

So boiling it down for you listening. This file gives you real power, doesn't it?

Speaker 2:

It really does.

Speaker 1:

Power to shape how AI sees your brand, your content. You get direct control over what shows up in those AI answers.

Speaker 2:

Which is becoming increasingly important as more people use AI for search and information gathering. It ensures your site acts like a good ambassador, not just a random source.

Speaker 1:

Okay, let's look under the hood. What does one actually look like? What's the structure?

Speaker 2:

It's remarkably simple actually.

Speaker 1:

Yeah.

Speaker 2:

Each item is basically just a clean URL, the web address. Right Then yeah. Each item is basically just a clean URL, the web address Right, then a descriptive title for that page.

Speaker 1:

Make it clear what it is.

Speaker 2:

Exactly, and then optionally you can add a bit more context, a short description for extra clarity for the AI.

Speaker 1:

Okay, can we see an example?

Speaker 2:

Sure, the source material had a good basic one. Let me read it out roughly it uses markdown formatting. Okay, it might start with a comment line like hashtag your-websitecom sample file, then maybe sections like hashtag tag posts.

Speaker 1:

Using the double hash for a heading.

Speaker 2:

Right and under that maybe a bullet point like hello world dot. Https dot your website dot com. Hello dash world. Welcome to WordPress. This is your first post.

Speaker 1:

So square brackets for the title, parentheses for the URL, then a colon in the description.

Speaker 2:

Exactly, and you might have another section, hashtag tag pages with something like privacy policy HTPSyour-websitecom for privacy-policy Learn how we handle your data. Super simple structure and, as we said, you, the listener, could literally try this. Go to a website you know. Add llmtxt to the end of the main address in your browser.

Speaker 1:

And if they have one.

Speaker 2:

You'll see it right there. It's becoming more common.

Speaker 1:

Okay, so how do you get one onto your site? Let's say you want to do this.

Speaker 2:

Well, there's the manual way.

Speaker 1:

Right, just type it out.

Speaker 2:

Yeah, write it using that markdown syntax. We just saw Save the file literally as llmstxt.

Speaker 1:

Got to get the name right.

Speaker 2:

Absolutely llmstxt. Then upload it to your website's root directory. Often that's a folder like var wetl on the server. Okay that sounds doable, but maybe a bit technical for some. Well, it can be. Thankfully, there are automated options popping up.

Speaker 1:

Like what.

Speaker 2:

There are specific tools like WordLift. Has an LLMStxt generator.

Speaker 1:

Okay.

Speaker 2:

And some WordPress plugins are getting in on it. Yoast SEO, for example, is starting to offer features for this.

Speaker 1:

Ah, so if you use WordPress, your SEO plugin might handle it Increasingly.

Speaker 2:

Yes, or dedicated plugins might appear, and some hosts, like Hostinger, are apparently building it into maybe a toggle in their tools or auto generation.

Speaker 1:

That's handy.

Speaker 2:

Yeah, and of course you can always use standard tools like FileZilla, you know, ftp or SFTP clients to upload the file manually, if you prefer.

Speaker 1:

But the manual way.

Speaker 2:

Yeah.

Speaker 1:

You mentioned challenges.

Speaker 2:

Oh yeah, it sounds easy, but there are pitfalls Like Well, the markdown formatting has to be exactly right, no wiggle room.

Speaker 1:

Okay, syntax errors break it.

Speaker 2:

Pretty much, and if your site content changes a lot new blog posts, updated pages you have to remember to update the file manually.

Speaker 1:

Constantly. That could updated pages.

Speaker 2:

You have to remember to update the file manually constantly. That could be a pain. It really can be. Plus, the location is critical. It must be in the root directory and not a subfolder easy mistake to make very easy and in coding matters it needs to be UTF 8.

Speaker 2:

And maybe the biggest headache there's no standard central way to check if it's valid no validator to not really a universal one yet, so you often have to test it yourself, maybe by asking an AI about your site and seeing if it seems to use the info correctly. It's not foolproof. Right, so the takeaway is For most people, honestly, using an automated tool or a plugin is probably the safer, more reliable and definitely more time efficient way to go.

Speaker 1:

Makes sense. Let the tools handle the tricky bits.

Speaker 2:

Exactly. They usually get the formatting and placement right automatically.

Speaker 1:

Okay, let's talk benefits. Why should someone listening go to the trouble, even with a tool? What's the payoff for your website?

Speaker 2:

There are some really solid advantages, big ones.

Speaker 1:

Lay them out for us.

Speaker 2:

Okay, first, brand accuracy. This file helps ensure the AI actually understands and represents your business, your brand voice, correctly. Less chance of weird summaries. Okay, accuracy, Good. What else? Visibility, but targeted visibility. It highlights the key stuff your best tutorials, your main product pages, your FAQs, the things you really want AI to find and share.

Speaker 1:

Right Cuts through the noise.

Speaker 2:

Exactly, which leads to reduced confusion. The AI gets a clear signal about what's important and relevant. Less confusion for the AI means better, more accurate answers for users asking about you.

Speaker 1:

That sounds critical.

Speaker 2:

It is. Then there's future-proofing You're basically getting your site ready for how more and more people are going to discover content through AI. It's happening now.

Speaker 1:

Being proactive.

Speaker 2:

And finally, selective disclosure. This is huge. You control what the AI sees. You can keep it away from pages that are outdated, maybe internal only or just not ready for prime time.

Speaker 1:

So real control over the AI's view of your site.

Speaker 2:

That's the essence of it Control and clarity.

Speaker 1:

Okay, and to make the most of those benefits, are there best practices, things to keep in mind when you're setting it up?

Speaker 2:

Definitely A few key things.

Speaker 1:

Go on.

Speaker 2:

First, keep the list focused. Don't just dump your entire site map in there. Pick the most important pages. Resist adding clutter.

Speaker 1:

Quality over quantity.

Speaker 2:

Exactly. Use short, clear, descriptive titles for each URL. Make it obvious what the page is about.

Speaker 1:

Like good signposting.

Speaker 2:

Perfect analogy. Also group links into logical sections using those markdown headings. Hashtag tag tutorials. Hashtag tag tutorials. Hashtag tag products. Hashtag tag about us. Helps the AI understand the relationships.

Speaker 1:

Structure matters.

Speaker 2:

It does, and be explicit about what not to include. Login pages, duplicate content pages, sensitive stuff Reinforces that selective disclosure benefit Right.

Speaker 1:

Keep it clean.

Speaker 2:

And finally test it. Don't just set it and forget it. Use tools like ChatGPT or Gemini. Ask them questions about your site and see if the answers reflect the guidance you've provided in your LMStxt.

Speaker 1:

Check that it's actually working as intended.

Speaker 2:

Yeah, it's the only way to be sure.

Speaker 1:

Okay, this is super helpful, but you mentioned robotstxt earlier and people know about sitemapxml. It could get confusing. Can we clarify the differences? Where does LMStxt fit?

Speaker 2:

Absolutely. It's easy to mix them up, but they have distinct jobs. Think of it like this, like a little table in your mind.

Speaker 1:

Okay, give us the columns.

Speaker 2:

Right LMStxt purpose guide, ai models, format markdown, content, key URLs, maybe with context audience AI tools Got it Next. Robotstxt purpose guide. Search engine crawlers, the traditional ones format plain text content. Rules all your URLs, often with extra metadata like update frequency, audience also search engines.

Speaker 1:

So lmstxt is curated. Robloxtxt is rules. Sitemapxml is exhaustive.

Speaker 2:

That's a great way to put it lmstxt is your hand-picked highlight reel for AI.

Speaker 1:

Right, and it's not yet universally adopted by every AI. Out there is it, but the big ones are starting.

Speaker 2:

That's the current state. Many leading models are beginning to honor it or are expected to. It's definitely the direction things are heading. It complements your existing SEO efforts like sitemaps and robotstxt, but doesn't directly replace them or impact, say, Google's ranking directly.

Speaker 1:

yet Okay, and you touched on this, but just to reinforce why do LLMs sometimes mess up reading websites, even if they look fine to us?

Speaker 2:

Yeah, it comes back to those core differences. Their short memory windows mean they might not see the whole picture. They might miss the overall link structure if it's not immediately obvious. Cluttered layouts or complex JavaScript can confuse them during that quick real-time fetch. They don't have that pre-built index to fall back on like a traditional search engine does.

Speaker 1:

So it's the speed and the lack of stored context basically.

Speaker 2:

Largely yes, which is why sites with lots of valuable specific content blogs, docs, FAQs, tutorials, e-commerce product details benefit most. That's the kind of stuff that needs clear signposting for a quick AI visit.

Speaker 1:

Gotcha, and quickly, any common mistakes people make when trying to implement this. Things to absolutely avoid.

Speaker 2:

Oh yeah, definitely a few common trip-ups.

Speaker 1:

Hit us with them.

Speaker 2:

Using the wrong file name is a big one. Typing llmtxt instead of llmtxt plural as is key.

Speaker 1:

Easy typo.

Speaker 2:

Very. Uploading it to the wrong directory, putting it in a subfolder instead of the main root. Ai won't find it there, location, location, location. Absolutely Including broken links or links to pages that are outdated or gone. That just misleads the AI.

Speaker 1:

Keep it current.

Speaker 2:

And messing up the markdown syntax using the wrong symbols, forgetting a parenthesis that can make the whole file unreadable to the AI.

Speaker 1:

So details matter. Get those right and you avoid the main problems.

Speaker 2:

Pretty much Attention to detail pays off here.

Speaker 1:

Okay, this has been incredibly clarifying.

Speaker 2:

Good. The core takeaway really is this LOM is txt. It might seem small, just a text file, but it's actually a really powerful strategic tool now in this age of AI search.

Speaker 1:

It's about taking control, isn't it?

Speaker 2:

It's about being proactive.

Speaker 1:

Yeah.

Speaker 2:

Taking control of how your information is understood and presented by AI.

Speaker 1:

So final thought then, for you listening, if your goal is to really shape how AI sees your content, how it surfaces information about you, implementing LMStxt now, even though it's still evolving, it, gives you a real strategic edge, doesn't it?

Speaker 2:

Absolutely. As AI use grows and it's growing fast getting this in place early means better visibility for the right things, more accurate representation of your brand or content.

Speaker 1:

Better performance overall in this new landscape.

Speaker 2:

Exactly. It positions you well for the future of search.

Speaker 1:

So the final challenge for you, listening, is think about your own site, your own content. How could this apply? What are those absolutely crucial pages on your website that you'd want any AI to understand perfectly first time, every time?

Speaker 2:

That's the question to ponder what's on your highlight reel?