Sitemap Generation in MediaWiki

Summary: Sitemaps help search engines discover, understand, and prioritise content within a website. MediaWiki provides several methods for generating XML sitemaps, ranging from automatic generation during page edits to scheduled generation for large-scale installations. This article examines the available options, their advantages and disadvantages, and provides guidance on selecting an appropriate sitemap strategy for a MediaWiki-based Knowledgebase.

Context

What is a Sitemap?

A sitemap is a structured XML document that provides search engines with a list of URLs available within a website.

The sitemap protocol allows website owners to communicate page locations, modification dates, update frequency, and relative importance. While search engines can discover content by following links, a sitemap provides a direct inventory of available content and can improve crawl efficiency.

Why Search Engines Use Sitemaps

Search engines continuously crawl the Internet looking for new and updated content.

Although modern search engines are highly effective at discovering content through hyperlinks, a sitemap offers several advantages:

• Faster discovery of newly published articles
• Faster identification of updated content
• Improved visibility of deeply nested pages
• Better crawl coverage for large websites
• Reduced reliance on link discovery alone

Sitemaps do not guarantee indexing, but they increase the likelihood that content will be discovered promptly.

Why Knowledgebases Benefit from Sitemaps

Knowledgebases often contain large volumes of technical content organised into categories, namespaces and hierarchies.

Without a sitemap, search engines may take significantly longer to discover articles that are newly created, infrequently linked, or buried deep within the site's navigation structure.

A regularly maintained sitemap helps ensure that:

• New articles are discovered quickly
• Updated articles are re-crawled efficiently
• Search engine coverage remains comprehensive
• Organic search traffic can grow more consistently

Core Concepts

The sitemap.xml Standard

The standard sitemap format is an XML document containing one or more URL entries.

A typical entry may include:

• URL location
• Last modification date
• Suggested update frequency
• Relative priority

Example:

<syntaxhighlight lang="xml"> <url>

   <loc>https://kb.example.com/wiki/OSI_Model</loc>
   <lastmod>2026-07-13</lastmod>
   <changefreq>weekly</changefreq>
   <priority>0.8</priority>

</url> </syntaxhighlight>

URL Discovery vs XML Discovery

Search engines discover pages through two primary mechanisms:

• Link crawling
• XML sitemap submission

Link crawling remains the primary discovery mechanism.

Sitemaps should be viewed as a supplementary discovery aid rather than a replacement for good site navigation.

Well-designed internal linking remains essential.

robots.txt Integration

Search engines should be informed of a sitemap's location through the site's robots.txt file.

Example:

<syntaxhighlight lang="text"> User-agent: *

Allow: /

Sitemap: https://www.pirho.net/sitemap.xml </syntaxhighlight>

This allows supporting search engines to locate the sitemap immediately during crawl operations.

Sitemap Generation Options in MediaWiki

MediaWiki supports multiple approaches to sitemap generation.

Each approach has advantages depending on the size and update frequency of the wiki.

Option 1: AutoSitemap Extension

The AutoSitemap extension automatically rebuilds the sitemap whenever page content changes.

Events typically triggering regeneration include:

• Article creation
• Article modification
• Article deletion
• Page moves
• File uploads
• Page restoration

Advantages

• Automatic operation
• No scheduler required
• Near real-time updates
• Simple installation
• Minimal administration

Disadvantages

• Rebuilds occur frequently
• Resource usage increases with site size
• May become inefficient for large installations

Suitable Use Cases

• Small knowledgebases
• Documentation portals
• Internal wikis
• Sites with fewer than approximately 10,000 pages

Option 2: generateSitemap.php

MediaWiki includes maintenance scripts capable of generating sitemap files on demand.

Rather than regenerating whenever content changes, administrators execute the script through:

• Scheduled tasks
• Cron jobs
• Automation platforms
• Orchestration systems

Advantages

• Highly scalable
• Predictable resource usage
• Supports very large wikis
• Suitable for enterprise deployments

Disadvantages

• Requires task scheduling
• Changes are not reflected immediately
• Additional administration effort

Suitable Use Cases

• Large public knowledgebases
• Documentation platforms
• Enterprise MediaWiki deployments
• Sites exceeding tens of thousands of pages

Option 3: MediaWiki Sitemap REST API

Recent MediaWiki releases include sitemap generation through a REST API endpoint.

Rather than storing a static sitemap file, MediaWiki can generate sitemap content dynamically when requested.

Advantages

• Built into modern MediaWiki releases
• No external extension required
• Centralised platform support

Disadvantages

• Dynamic generation may increase server load
• Requires appropriate configuration
• May attract frequent crawler requests

Suitable Use Cases

• Modern MediaWiki environments
• API-centric deployments
• Administrators wishing to minimise third-party extensions

Practical Application

Small Knowledgebases (Under 10,000 Articles)

For smaller installations, automatic generation is often the simplest and most effective option.

The administrative overhead is minimal and sitemap updates occur immediately after content changes.

In practice, many smaller knowledgebases benefit from the simplicity of this approach.

Medium Knowledgebases

As article counts increase, regeneration costs become more noticeable.

At this stage, administrators should consider:

• Regeneration rate limiting
• Namespace exclusions
• Reduced generation frequency

This balances freshness against system performance.

Large Knowledgebases

Large installations should generally avoid rebuilding entire sitemaps on every page modification.

Scheduled generation is typically preferred because it:

• Reduces server load
• Improves predictability
• Scales more effectively
• Supports sitemap indexes

Design & Architecture Considerations

Regeneration on Every Edit

Automatic regeneration provides the freshest sitemap possible.

However, this convenience comes at the cost of increased server activity.

As content volume increases, the administrative simplicity may no longer justify the resource consumption.

Scheduled Generation

Scheduled generation introduces a small delay between content publication and sitemap updates.

For most knowledgebases, this delay is insignificant.

Search engines typically do not require second-by-second awareness of content updates.

Namespace Filtering

Not every namespace should appear within a public sitemap.

Common exclusions include:

• User pages
• Talk pages
• Template pages
• MediaWiki system pages
• Administrative content

A sitemap should focus on content that delivers value to visitors.

Category-Based Sitemap Strategies

Large knowledgebases can benefit from dividing content into logical sitemap groups.

Examples include:

• Guides
• Tutorials
• Deep Dives
• Reference Material
• API Documentation

This approach improves management and can assist search engines in understanding site structure.

Search Engine Crawl Budget

Search engines allocate finite resources when crawling websites.

Poorly configured sitemaps can waste crawl budget by presenting:

• Duplicate content
• Administrative pages
• Low-value content
• Redirect pages

Effective sitemaps help search engines focus on important material.

Common Pitfalls

Including Administrative Namespaces

Publishing internal administrative content can dilute sitemap quality and waste crawl resources.

Always review namespace inclusion policies.

Broken Canonical URLs

Incorrect server configuration may result in:

• Mixed protocols
• Internal hostnames
• Duplicate domains

Search engines should only receive canonical URLs.

Stale Sitemaps

A sitemap is only useful if it reflects current site content.

Regular validation should form part of routine maintenance procedures.

Oversized Sitemap Files

Very large knowledgebases may exceed practical sitemap limits.

When this occurs, sitemap indexes and segmented sitemap structures should be adopted.

Recommended Approach

SEO Knowledgebase Reference Architecture

For most MediaWiki-based technical knowledgebases, the following approach provides an effective balance:

• Use AutoSitemap for smaller installations
• Exclude administrative namespaces
• Publish sitemap location in robots.txt
• Submit the sitemap to major search engines
• Monitor indexing through search engine webmaster tools
• Migrate to scheduled generation as content volume grows

This provides a straightforward upgrade path without introducing unnecessary complexity early in the project's lifecycle.

Decision Matrix

Knowledgebase Size	Recommended Approach
Under 10,000 Pages	AutoSitemap
10,000-50,000 Pages	AutoSitemap with rate limiting
Over 50,000 Pages	generateSitemap.php
Enterprise Deployments	Scheduled sitemap indexes

Key Takeaways

• Sitemaps improve content discovery but do not replace good navigation.
• Smaller MediaWiki installations benefit from automatic sitemap generation.
• Larger installations should favour scheduled generation.
• Administrative content should generally be excluded.
• Sitemap strategy should evolve alongside the growth of the Knowledgebase.
• A well-maintained sitemap is an important component of technical SEO.

Related Topics

• robots.txt
• MediaWiki SEO
• Structured Data
• Search Engine Crawling
• Search Engine Indexing
• Technical SEO
• Category Design
• Information Architecture

References

• MediaWiki Manual: Sitemap
• MediaWiki Extension: AutoSitemap
• Sitemap Protocol Specification
• Google Search Central Documentation
• Bing Webmaster Guidelines