Sitemap and Robots file

How we generate robots.txt and sitemap.xml

Instead of relying on manual creation and maintenance of the aforementioned files, the platform generates them (independently from the build and release process) following the approach:

1. We retrieve all the static pages defined in Sanity. Depending on their flag “Include in sitemap” we add them to the sitemap.xml or to the robots.txt file.

2. We fetch all the dynamic (menu, categories, store locator, etc.) pages and add them to the sitemap.

3. We read the additional URLs and excluded URLs lists that can be found in Sanity configuration:
   Marketing Content > SEO ([market-domain]/structure/marketingContent;seo)

Open

Open

Output

The result of these three levels are merged together and the files are generated and published nightly and optionally on-demand, independently from the release process of the app.

They can be found at the root of any market for example:

https://www.popeyes.es/robots.txt

https://www.popeyes.es/sitemap.xml

The entire process should be verified in Google Search Console.

Robots meta tags

We control page-level indexing via robots meta tags.

Here’s the logic we apply in order to determine whether we should mark a page as indexable or not:

Non-transactional URLs (e.g. static pages, restaurant pages, nutrition explorer, etc.) are indexed.

Internal application URLs (e.g. login/logout, account pages, diagnostics, etc.) are NOT indexed.

URLs listed under Sitemap excluded URLs are NOT indexed.

On top of this, we can influence the discovery of such URLs via sitemap.xml + robots.txt inclusion/exclusion mechanisms.

Results are not guaranteed as we can suggest to search engines but we cannot manually influence the outcome.