A question came up this week while I was wiring up SEO for a crypto pool and block explorer site. On its face it sounds like a nothing question — the kind of thing that doesn’t belong on a project checklist at all. Should we index the mempool pages?
The honest answer took me about ten minutes of staring at the sitemap to arrive at, and once you see it, it reshapes how you think about SEO for any site with a database behind it. The answer is no. And it’s a louder no than I expected.
The shape of the problem
A block explorer is a database with a URL for every row. Every transaction gets /tx/<hash>. Every address gets /address/<addr>. Every block gets /block/<height>. Then there’s the mempool — the live view of pending transactions that haven’t been mined yet. New rows appear every few seconds. Old ones churn out when they confirm.
Even on an obscure chain with modest traffic, that’s tens of thousands of URLs a week. On a popular chain it’s millions. All of them render clean HTML. All of them have unique titles. All of them link to each other, so Googlebot can walk the whole graph from any starting point.
Which Googlebot will happily do, if you let it.
Every URL you let Googlebot crawl is a URL you’re taxing your crawl budget to crawl, whether or not that URL will ever earn a visit.
— The one sentence version
Crawl budget is a real thing
Google doesn’t crawl every URL on every site every day. It allocates attention, and the allocation is based on signals like domain authority, server response time, and how often your important pages actually change. A new domain gets a thin budget. A hundred-thousand-URL explorer on a thin budget means Googlebot shows up, crawls two thousand mempool pages that will be gone by lunch, and leaves. Your homepage didn’t get recrawled. Your “how to mine XYZ” guide didn’t get recrawled. The pages you’d like to rank got starved for the pages you’d like to hide.
The pattern isn’t unique to crypto. Same shape shows up with:
- E-commerce stores that generate a URL per color-size variant
- Forums where every reply gets a permalink
- Classifieds with filter combinations in the query string
- Any site where “pages” are really database rows
If every row is a page, Google treats every row as a page.
The two-tier split
The move is to decide which URLs are content and which URLs are data. Content earns a spot in the index. Data gets served to humans who ask for it, but it doesn’t belong in a search result.
For an explorer, the split falls pretty naturally:
Index these — they answer questions:
- Homepage, pool stats, “how it works”
- Educational content: how to mine, how to run a node, FAQs
- Guides for the specific chain you support
- Blog posts, docs, downloads
Noindex these — they are database lookups:
/tx/<hash>— someone searching a tx hash already knows the hash/address/<addr>— same, nobody is Googling an address/block/<height>/mempool,/pending,/recent- Any paginated API-ish page
The test I use: will anyone type words into Google that this page is the best answer for? If the answer involves a 64-character hex string, the page isn’t a search result, it’s a lookup. Hide it.
How to actually do it
Three places to intervene. In order of how load-bearing they are:
1. Meta robots on the rendered pages. The lookup routes get a <meta name="robots" content="noindex, follow"> in the head. follow matters — you still want Googlebot walking the internal links on these pages, because that’s how it finds the pages you do want indexed. You just don’t want them stored.
2. The sitemap only lists content. The lookup routes are not in the sitemap. Full stop. Sitemap is where you nominate the pages you’d like crawled; let it be a short, deliberate list of the pages that matter. For a bilingual site, that means child sitemaps per language, each pointing only to content URLs.
3. Headers for anything non-HTML. If you have JSON API routes under the same domain, send X-Robots-Tag: noindex as a response header. HTML meta tags are useless on JSON, but Googlebot reads the header fine.
What this looks like a month later
Here’s what actually changes once you ship this. Google’s crawl stats report in Search Console redistributes. The green bars shift away from /tx/* and /address/* and onto the pages you want it on. Your content pages get crawled more often. Your indexed-pages count in GSC drops — which looks scary but is actually the goal; a thousand indexed pages that nobody visits is a thousand pieces of weak signal diluting the ones that convert.
The pages that rank for queries like “how to mine X” or “X pool fees” start getting recrawled on a cadence that matches how often you actually change them, instead of competing with the next ten thousand pending transactions.
The real lesson
The question isn’t “how do I get Google to index all my pages?” The question is “which pages are the product, and which pages are the database?” If you can draw a clean line, your SEO work gets a lot cheaper.
For a blockchain explorer, the product is the homepage, the educational content, and the pool stats. The database is everything else. Google only needs to see the first one.
For whatever you’re building, there’s probably a similar line. Finding it is the work.