Most local sites in this market carry the same service page three times over — once for Tampa, once for St. Petersburg, once for Clearwater — and then again for every suburb. Publishing them was the easy part. Getting a crawler to care is where the plan quietly stalls.
A page that exists is not a page that has been found, and a page that has been found is not a page that has been kept. Between the file on your server and a row in a results page sit three separate events, each of which can fail on its own.
The triplicated page tree
Start with the geography, because it is what produced the problem. This market is three cities separated by water rather than one city that spread out. A customer in Palm Harbor does not casually cross to Riverview, and a business that wants both has to say so on the page. So the advice everybody follows is: build a page per city.
That advice is not wrong. What goes wrong is the multiplication. One service becomes three city pages. Three cities become fifteen once you add Largo, Brandon, Wesley Chapel, Seminole and Dunedin. Six services across that grid is ninety pages before anyone has written a word that a human would want to read, and in practice eighty of them are one file with a place name swapped in.
Say it plainly: a tree built this way is mostly duplication wearing local costume. It is not a penalty problem in the dramatic sense. It is a budget problem. Every one of those near-identical files consumes attention that could have gone to the ten pages carrying an actual argument, and the crawler cannot tell which is which any faster than a reader could.
Existing, discovered, indexed
These three words get used interchangeably in conversation and mean entirely different things in practice. Keeping them apart is most of the skill.
| Stage | What has happened | What can go wrong | Where you would see it |
|---|---|---|---|
| Exists | The URL returns a page to anyone who requests it | Nothing links to it; nothing lists it | Only in your own file tree |
| Discovered | A crawler knows the address is there | Known but never queued, or queued indefinitely | Submission log, sitemap job status |
| Crawled | A bot requested the URL and got a response | Timeouts, redirect chains, server errors | Per-URL log with timestamp and status |
| Indexed | The page was judged worth storing | Judged duplicate, thin, or not worth the slot | Search Console coverage, not the submission log |
| Ranked | It is served for something somebody types | Competitors are simply better on that query | Analytics and rank tracking |
Each of those transitions is a decision made by somebody else's system, and you can only influence the first three. The fourth is a judgment about the value of the page, and no submission mechanism in existence overrides it.
Crawl budget, and what eats it
Crawl budget is not a number anyone sends you. It is the practical outcome of two things: how much load your server will tolerate without slowing down, and how much a crawler judges your site worth returning to. A small local site rarely hits the first ceiling. It hits the second constantly, and hits it hardest when the tree is inflated.
- Near-duplicate location pages. The heaviest consumer in this market. Ninety pages that vary by one noun each get fetched, compared and mostly discarded — the cost is paid before the verdict.
- Parameter and filter URLs. Sorting, pagination and tracking parameters multiply a modest catalog into thousands of addresses that all resolve to nearly the same thing.
- Redirect chains and dead ends. Every hop is a request. An old city page redirecting to a new one that redirects to a hub spends three fetches to deliver one document.
- Slow responses. Server time is the budget in its rawest form. A page that takes two seconds is worth a fraction of one that takes two hundred milliseconds, at identical value.
- Orphans. Pages nothing links to are reached only through a sitemap, and are treated with corresponding suspicion.
The relationship worth internalizing is inverse. Attention spent on the forty pages that will not be kept is attention not spent on the ten that would have been. Pruning is not tidying; it is reallocating a finite resource toward the pages that can actually win something.
The suburb page that says nothing
Identical to the city page except for the heading, the meta description and three mentions of the town in body copy nobody reads.
- Collects impressions, almost no clicks
- Competes with your own stronger page
The shore page that earns its place
A separate St. Petersburg page with its own address, its own crew, its own hours and a paragraph about what crossing the bridge means for scheduling.
- Answers a question the other page cannot
- Holds its own position without cannibalizing
Sitemaps, read properly
A sitemap is not a ranking device and never was. It is a discovery instrument: a list saying here is everything I consider worth looking at, in an order and structure I chose. For a tree of the shape described above, that second half matters more than the first, because the structure of the file is a statement about what you think your site is.
Recursive parsing goes up to three levels deep, which is enough to model a market like this one honestly. An index at the top, one sitemap per shore beneath it, and beneath those the service groupings. A single job accepts up to 1,000 sitemaps, so even an agency handling a stack of client properties fits inside one submission rather than a morning of clicking.
Sitemap submission — by file or by URL
For getting a whole tree in front of a crawler in one operation, with its shape intact.
- Upload a file or point at an address. Either input works, so a generated sitemap and a static one are handled the same way.
- Parsed recursively to three levels. An index of indexes is followed rather than treated as one flat document.
- Up to 1,000 sitemaps in a single job. Portfolio-scale in one operation instead of one property at a time.
- Two jobs run at once, twenty wait in line. Queue the rest and let them clear; nothing needs supervising.
Two concurrent jobs with twenty in the queue is a deliberate shape rather than a limitation. Discovery is not improved by parallelism; it is improved by being complete and consistent. Submitting the same tree four times over does nothing that submitting it once did not already do, which is the reason a queue exists at all instead of an unbounded firehose.
A thousand a day, ten thousand a batch
Alongside sitemaps sits direct URL submission, and this is where the arithmetic gets useful. The daily budget is 1,000 URLs per account. A single bulk batch takes up to 10,000. Those two numbers are not in conflict — the batch is how much you can hand over in one action, the daily figure is how fast it drains.
Bulk URL submission and the tracker
For pushing a defined list and then watching what actually happened to each address.
- 1,000 URLs per day, per account. A real ceiling. It rewards deciding what matters before pasting a list.
- Up to 10,000 URLs in one batch. Hand over a full migration at once and let the daily budget work through it.
- Delivered through the IndexNow API. The protocol GoogleBot and BingBot accept for being told that an address is new or changed.
- A log kept per URL. Bot visit with a timestamp, status, error detail — evidence rather than an assumption.
IndexNow is worth understanding for what it is: a notification protocol. It replaces waiting for a crawler to stumble across a change with telling it directly that something moved. It is genuinely useful after a migration, a price change, a rewritten set of service pages. It is not a queue-jumping device, and treating it as one produces the disappointment described further down.
What the status of a job actually tells you
The counters run live: submitted, found, failed. Each of the three answers a different question, and only one of them is about your content.
- Submitted. The count you caused. It confirms the request left, nothing more, and it is the number people mistake for success.
- Found. A bot requested the URL and received a response. This is the first figure that involves anyone but you.
- Failed. The most valuable of the three. A failure carries an error detail, and error details are the only part of this process that name the problem for you.
Failures are worth reading one by one rather than as a total. A timeout is a hosting question. A 404 in a submitted list means your sitemap and your site disagree about what exists. A redirect means the address you submitted is not the address you meant. None of these are indexing problems and all of them are usually diagnosed as one.
| What the log shows | What it means | What it is not |
|---|---|---|
| Submitted, never visited | The address is known and not yet prioritized | Evidence of a penalty |
| Visited, no ranking movement | Crawled and judged; the verdict was not favorable | A discovery problem |
| Repeated timeouts on one section | A server or template issue in that part of the tree | A content quality issue |
| 404s appearing in a submitted batch | The sitemap lists pages the site no longer serves | A crawler error |
| Everything found, nothing indexed | Discovery worked perfectly and the pages were not wanted | Something more submission can fix |
That last row is the one to sit with. It is the clearest possible signal, and it points away from the Indexing Hub entirely and back at the ninety pages. A per-URL record with timestamps is what converts a vague sense that indexing is slow into a specific statement about which stage failed.
Deciding what the tree should contain
Now the uncomfortable part. Somewhere between ten and ninety lies the number of pages your site should actually have, and no tool will pick it. Four tests, applied honestly, get most firms close.
Could a local read it and tell?
If somebody from Clearwater could not tell this page was written for Clearwater with the town name removed, it is not a Clearwater page.
- Names, addresses, hours, crews
- Something true only of that place
Does the offer change?
Different travel time, different pricing, a different licensing requirement, a physical location on that shore — any of these justify a page on their own.
- Bridge time is a real scheduling fact
- Say it rather than imply it
What has it earned in a quarter?
Pages with impressions and no clicks over three months have been evaluated by the market already. Consolidate them into the parent.
- Redirect rather than delete
- Fold the useful sentence upward
Is it competing with you?
Two of your own pages ranking alternately for one query is a split, not coverage. Pick the stronger and point the other at it.
- Check the page breakdown per query
- One intent, one destination
Consolidation frequently improves indexing without a single submission, for the plain reason that a crawler visiting thirty pages sees each of them more often than a crawler visiting ninety. Where a page genuinely deserves to exist, on-site suggestions produced against your own data are more useful than a generic checklist, because they are looking at the same tree you are.
Running the numbers on your own tree
Take the firm from the opening: six services, fifteen places, ninety generated pages plus roughly twenty core pages. One hundred and ten URLs. Against a budget of 1,000 a day, the entire site can be submitted in a single afternoon with nine tenths of the allowance untouched.
That is the whole argument in four tiles. Submission was never the constraint for a business this size. Now scale it: an agency with twenty client sites of that shape holds 2,200 URLs. One batch covers it with room to spare; the daily budget clears it in three days. Still not a constraint. The constraint is that of those 2,200 addresses, perhaps 400 say something a person could not read on another page of the same site.
The calculation flips only at genuine scale — a marketplace, a directory, a property portal with tens of thousands of listings. There the daily allowance is the real planning input: 10,000 URLs handed over in one batch, drained at 1,000 a day, is a ten-day operation to sequence deliberately. Everybody smaller than that is solving a content problem while looking at a submission screen.
For firms running this alongside the rest of their marketing, the automated tiers absorb the routine — AutoSEO at $149 per month per domain for keyword discovery, link building and on-site suggestions, FullSEO at $500 with manual keyword selection and a human review step before changes go live. Our service overview sets out which of the two suits a local operation.
Common questions
We submitted every page and half of them still are not indexed. What went wrong?
Most likely nothing went wrong mechanically. Check the log: if the URLs show a bot visit with a good status, discovery succeeded and the pages were assessed and declined. That is a content verdict. Compare two of the unindexed pages side by side — if the only difference is a place name, you have your answer.
How many location pages is too many?
There is no fixed number, only a test: every page must contain something a reader could not get from the page above it. In practice a firm serving both shores usually justifies two or three strong city pages and a handful of suburb pages where it genuinely has a presence, rather than fifteen of each.
Should we delete the weak pages or leave them?
Redirect, do not delete. Fold whatever is useful into the surviving page and point the old address at it with a single hop. Deleting outright throws away any links the page collected; leaving it untouched keeps spending crawl attention on something you have already judged worthless.
Does IndexNow work for Google, or only Bing?
Submissions go out through the IndexNow API for both GoogleBot and BingBot. What it does is announce that an address is new or changed. What happens after the announcement is decided by each engine on its own terms, and neither has ever promised inclusion in exchange for a notification.
How deep should our sitemap structure go?
Recursion runs to three levels, and three is plenty. A workable arrangement here is an index at the top, one sitemap per shore below it, and service groupings under those. It also makes diagnosis easier: when one branch fails, the failure is localized to a shore or a service instead of appearing as a general problem.
How often should we resubmit?
After meaningful change, and not otherwise. A migration, a restructure, a set of rewritten pages — those are worth a batch. A weekly resubmission of an unchanged tree consumes the allowance and signals nothing. Keep the daily budget for URLs that actually moved.
The pattern in this market is consistent enough to state as a rule. Sites arrive convinced they have an indexing problem, and the log shows discovery working exactly as designed against a page tree that triplicated itself for reasons of geography and then triplicated again for reasons of habit. The bay is a real commercial boundary; it is not a reason for ninety files.
Fix the tree first and the mechanics become straightforward. Submit what exists, watch the per-URL log rather than the submitted counter, and read failures as the diagnostic they are. A live counter for submitted, found and failed URLs tells you within a day or two which of the three stages is actually holding you up, which is a very different conversation from waiting and guessing. Elsewhere on this blog the analytics side of the same question gets the same treatment.
Start by counting your own URLs and dividing them into pages that say something and pages that repeat something. Connect the site and run one sitemap job to see what the crawler has been finding. If the found count matches the submitted count and nothing is indexed, no submission tool will help — but you will at least be arguing about the right thing, which is the ninety pages rather than the queue.