Crawling and Indexing: How Search Engines Discover Your Pages

You could write the clearest, most useful page on the entire internet, and it would still draw exactly zero visitors from search — if no search engine ever found it. Long before a page can be ranked, or even judged worth keeping, it has to be discovered. And discovery is not quite something search engines do to your site; it is closer to a conversation, one where your site quietly signals which pages exist and which doors are open. This is about how that discovery works — and the small, unglamorous controls that let you steer it.

Discovery Begins with a Link

Search engines keep no master list of every page on the web. A crawler — Google's is Googlebot, Microsoft's is Bingbot — discovers new pages in just one way: by following links. It reads a page it already knows, pulls out the links on it, and queues up any unfamiliar addresses to visit next. Picture a courier who only learns new streets by reading the signposts on streets already walked.

This has a blunt consequence: a page that nothing links to — an orphan page — is almost invisible, because no path leads a crawler to it. The most reliable way to get a page found is simply to link to it from other pages you publish. And discovery is only the first of three stages — find a page, make sense of it, then rank it against others — but it is the gate: if a page is never found, nothing else can happen to it.

robots.txt — the Doorman for Crawlers

Where it lives

One of the first things many crawlers look for when they arrive is a small plain-text file at the very root of the domain, named robots.txt — for example https://example.com/robots.txt. It has to live at the root; a crawler will not look for it in a subfolder. Its reach is also narrower than most people expect: a robots.txt speaks only for the exact address it sits on. The file at https://example.com/robots.txt governs pages under https://example.com/ and nothing else. The plain http:// version of the site, the www.example.com version, and a blog.example.com subdomain each count as a separate place as far as crawlers are concerned — so every one of them needs its own robots.txt. (In technical terms, the file only covers URLs that share its protocol, host, and port.)

The syntax

The syntax is small. The file is built from groups, each starting with a User-agent line naming the crawler the rules apply to (* matches most of them), followed by Disallow and Allow rules that name paths. A standalone Sitemap line points to your sitemap and applies regardless of user agent:

A few rules govern how this is read. When two rules conflict, the most specific (longest-matching) path wins. Paths are case-sensitive, so Disallow: /Admin/ does not block /admin/. And anything not matched by a rule is allowed by default — the file is a list of exceptions, not a whitelist.

The fine print

If you're just getting started, feel free to skim this part — it's the kind of detail that only matters when something breaks. A handful of current technical facts are worth pinning down, because older tutorials get them wrong. The Robots Exclusion Protocol had been an informal convention since 1994, but it only became a formal internet standard — RFC 9309 — in 2022. Google honors exactly four fields (user-agent, allow, disallow, sitemap) and silently ignores the rest, including crawl-delay; Bing, for its part, still respects crawl-delay, a small reminder that engines do not agree on every detail. Google also stops reading the file past roughly 500 KiB, so keep it lean by blocking whole directories rather than listing individual URLs. Finally, how the server responds matters: a 404 on robots.txt is read as "no restrictions, crawl freely," but a 5xx server error makes Googlebot pause crawling the whole host until the file is reachable again. A misconfigured server that throws 500 errors on its robots.txt can therefore quietly throttle the crawling of an entire site — an oddly common, oddly invisible failure.

What a real one looks like

Textbook examples are tidy; real ones are chattier. Here is a trimmed excerpt from Wikipedia's actual robots.txt — the same file the exercise at the end invites you to open:

Three things jump out that the tidy example can't teach. First, real files are written for humans too: they open with plain-English comments where the maintainers explain themselves — here, even a polite note asking crawlers not to go too fast. Second, they rarely stop at User-agent: *; they name specific crawlers (Mediapartners-Google, grub-client) and block them one by one, often with a comment recording why — usually a bot that once misbehaved. Third, Wikipedia's file runs to hundreds of lines, most of it a long roll-call of banned bots. That you can read every word of it — their entire crawling policy, editorial asides and all — is the next point, made concrete.

The Distinction Everyone Gets Wrong

If you remember one thing from all of this, make it this: robots.txt controls crawling; it does not control indexing. Those are two different jobs, and treating them as one is the single most common — and most self-defeating — mistake in the whole topic.

Disallow vs. noindex

A Disallow line tells a crawler not to fetch a page. It says nothing about whether that URL may appear in search results. In fact, a page you disallowed can still be indexed if other pages link to it — typically showing up with no description, sometimes with a bare note that no information is available — because the engine knows the URL exists (it saw the link) but was told not to look inside.

To actually keep a page out of the index, you use a different tool: noindex. You set it with a <meta name="robots" content="noindex"> tag in the page's HTML <head>, or with an X-Robots-Tag: noindex HTTP header (the header version also works for non-HTML files like PDFs). When a crawler fetches the page and sees noindex, it drops that page from results entirely, no matter who links to it.

Why blocking a page backfires

Here is the counterintuitive punchline. For a crawler to see the noindex, it has to be allowed to fetch the page. If you block the page in robots.txt, the crawler never fetches it, never sees the noindex, and the URL can linger in the index through inbound links. So blocking a page you want gone is exactly backwards. The correct recipe is the opposite of most people's instinct: allow the crawl, and add noindex.

This is not folklore — it is why Google retired the old habit of writing noindex inside robots.txt back in September 2019. That rule had never been documented, and in the overwhelming majority of files that used it, it contradicted other rules and quietly hurt the very sites relying on it. Today a noindex line inside robots.txt simply does nothing; the tag or header on the page itself is the only mechanism that works.

The cleanest way to hold this in your head is a simple grid. Crawling and indexing are two independent questions, which means there are four possible combinations — and each has a different right answer.

One honest footnote before moving on. This crawl-versus-index question has a close cousin worth naming. The same page often ends up reachable at several different URLs without anyone planning it: example.com/post and www.example.com/post might both work, a share link might tack on a tracking parameter like ?utm_source=twitter, and a filter might add ?sort=newest — four addresses, one identical article. Left alone, a search engine may treat them as separate pages competing against each other. The fix is a small signal called a canonical tag: a line in the page that says "of all these addresses, this one is the real version — credit that." It's a subject in its own right, but the term is worth knowing, because it lives right next door to everything here.

Sitemaps — a Hint, Not a Command

If robots.txt is the doorman, a sitemap is the friendly map you hand the courier on the way in. It is an XML file listing the URLs you would like search engines to know about, optionally with a last-modified date for each. Its whole job is to help discovery — and that word matters, because a sitemap commands nothing. Listing a URL is a suggestion to consider it, never a guarantee that it will be crawled or indexed.

Google is refreshingly honest about who actually needs one. A sitemap earns its keep if your site is large, if it is brand new with few external links pointing at it, or if it has many deep or isolated pages that internal linking misses. If your site is small and thoroughly linked — Google's rough line is around 500 pages or fewer, all reachable by following links from the home page — you probably do not need one at all. The structure, when you do want one, is simple:

A couple of details save you wasted effort. Google trusts the <lastmod> date only when it is consistently and verifiably accurate, and it ignores the optional <priority> and <changefreq> values entirely — so don't bother hand-tuning them. Each sitemap file is capped at 50,000 URLs or 50 MB uncompressed; if you exceed either, you split into several files and list them in a sitemap index file. And every URL in a single sitemap must live on the same host.

You tell an engine about your sitemap in two ways: add a Sitemap: line to robots.txt, and submit the sitemap's URL in the Sitemaps report inside Google Search Console, where you can also see when it was last read and any errors. One historical footgun: for years, tutorials told you to "ping" a special URL to notify Google of an updated sitemap. Google deprecated that endpoint in 2023, and it now simply returns an error — so ignore any guide that still recommends it, and rely on robots.txt and Search Console instead.

The three tools at a glance

Three different tools, three different jobs. The trouble people run into almost always comes from reaching for the wrong one, so it's worth keeping them straight:

Crawl Budget — Mostly Not Your Problem

"Crawl budget" is the amount of crawling an engine is willing to spend on your site, shaped by how much your server can handle and how much demand there is for your content. It is a real concern — for very large or fast-changing sites. An e-commerce catalog that spins up hundreds of thousands of filter-combination URLs, or a news archive with millions of pages, can genuinely waste an engine's attention on junk and delay the pages that matter.

For a blog, it is a non-issue, and it is worth saying plainly so you don't over-engineer. Google's own crawl-budget guidance is aimed at sites in the range of a million-plus pages, or ten-thousand-plus pages that change every day. Below that, the engine has no trouble keeping up. So don't hide internal links, sprinkle noindex, or fiddle with settings to "save budget" on a small site — that effort is not just wasted, it can actively hurt the discovery you were trying to help.

A Grounded Example: a Hand-Built Site

Managed platforms like WordPress, Wix, or Squarespace usually generate a robots.txt and a sitemap for you automatically. A hand-built site — say a small Python application on the Flask framework — does not. Flask serves only the routes you explicitly define, so unless you have added them, /robots.txt and /sitemap.xml may simply not exist and will return a 404.

So for a custom site, the useful question is not "what do my robots.txt and sitemap say?" but the more basic "do they exist at all?" You can serve them either as static files placed at the root, or with two tiny routes — one that returns plain text for /robots.txt, and one that returns XML with the correct content type for /sitemap.xml. Which path you choose matters far less than confirming the files are actually reachable and return what you expect. Don't assume — check.

What Discovery Comes Down To

Underneath the file formats and edge cases, the whole subject is small enough to carry in one hand. Search engines find pages by following links, so your internal linking is the real engine of discovery — far more than any file. robots.txt decides what may be crawled; noindex decides what may be indexed; and the two are never interchangeable. A sitemap is a helpful hint, not a command. And crawl budget is someone else's problem until your site is genuinely enormous.

None of it is instant — even when everything is configured perfectly, a new page can take days to show up, because it has to be crawled and processed first. But the through-line is the one worth keeping: discovery is a conversation, and you now know how to hold up your end of it.