Dex: /* robots.txt Integration */

2026-07-13T10:35:57Z

robots.txt Integration

← Older revision		Revision as of 10:35, 13 July 2026
Line 85:		Line 85:
	Allow: /		Allow: /

	Sitemap: https://~~knowledgebase~~.pirho.net/sitemap.xml		Sitemap: https://www.pirho.net/sitemap.xml
	</syntaxhighlight>		</syntaxhighlight>

Dex: /* robots.txt Integration */

2026-07-13T10:35:35Z

robots.txt Integration

← Older revision		Revision as of 10:35, 13 July 2026
Line 85:		Line 85:
	Allow: /		Allow: /

	Sitemap: https://kb.~~example~~.~~com~~/sitemap.xml		Sitemap: https://knowledgebase.pirho.net/sitemap.xml
	</syntaxhighlight>		</syntaxhighlight>

Dex: Created page with "'''Summary:''' Sitemaps help search engines discover, understand, and prioritise content within a website. MediaWiki provides several methods for generating XML sitemaps, ranging from automatic generation during page edits to scheduled generation for large-scale installations. This article examines the available options, their advantages and disadvantages, and provides guidance on selecting an appropriate sitemap strategy for a MediaWiki-based Knowledgebase. == Context..."

2026-07-13T10:33:09Z

Created page with "'''Summary:''' Sitemaps help search engines discover, understand, and prioritise content within a website. MediaWiki provides several methods for generating XML sitemaps, ranging from automatic generation during page edits to scheduled generation for large-scale installations. This article examines the available options, their advantages and disadvantages, and provides guidance on selecting an appropriate sitemap strategy for a MediaWiki-based Knowledgebase. == Context..."

New page

'''Summary:'''
Sitemaps help search engines discover, understand, and prioritise content within a website. MediaWiki provides several methods for generating XML sitemaps, ranging from automatic generation during page edits to scheduled generation for large-scale installations. This article examines the available options, their advantages and disadvantages, and provides guidance on selecting an appropriate sitemap strategy for a MediaWiki-based Knowledgebase.

== Context ==

=== What is a Sitemap? ===

A sitemap is a structured XML document that provides search engines with a list of URLs available within a website.

The sitemap protocol allows website owners to communicate page locations, modification dates, update frequency, and relative importance. While search engines can discover content by following links, a sitemap provides a direct inventory of available content and can improve crawl efficiency.

=== Why Search Engines Use Sitemaps ===

Search engines continuously crawl the Internet looking for new and updated content.

Although modern search engines are highly effective at discovering content through hyperlinks, a sitemap offers several advantages:

* Faster discovery of newly published articles
* Faster identification of updated content
* Improved visibility of deeply nested pages
* Better crawl coverage for large websites
* Reduced reliance on link discovery alone

Sitemaps do not guarantee indexing, but they increase the likelihood that content will be discovered promptly.

=== Why Knowledgebases Benefit from Sitemaps ===

Knowledgebases often contain large volumes of technical content organised into categories, namespaces and hierarchies.

Without a sitemap, search engines may take significantly longer to discover articles that are newly created, infrequently linked, or buried deep within the site's navigation structure.

A regularly maintained sitemap helps ensure that:

* New articles are discovered quickly
* Updated articles are re-crawled efficiently
* Search engine coverage remains comprehensive
* Organic search traffic can grow more consistently

== Core Concepts ==

=== The sitemap.xml Standard ===

The standard sitemap format is an XML document containing one or more URL entries.

A typical entry may include:

* URL location
* Last modification date
* Suggested update frequency
* Relative priority

Example:

<syntaxhighlight lang="xml">
<url>
<loc>https://kb.example.com/wiki/OSI_Model</loc>
<lastmod>2026-07-13</lastmod>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>
</syntaxhighlight>

=== URL Discovery vs XML Discovery ===

Search engines discover pages through two primary mechanisms:

* Link crawling
* XML sitemap submission

Link crawling remains the primary discovery mechanism.

Sitemaps should be viewed as a supplementary discovery aid rather than a replacement for good site navigation.

Well-designed internal linking remains essential.

=== robots.txt Integration ===

Search engines should be informed of a sitemap's location through the site's robots.txt file.

Example:

<syntaxhighlight lang="text">
User-agent: *

Allow: /

Sitemap: https://kb.example.com/sitemap.xml
</syntaxhighlight>

This allows supporting search engines to locate the sitemap immediately during crawl operations.

== Sitemap Generation Options in MediaWiki ==

MediaWiki supports multiple approaches to sitemap generation.

Each approach has advantages depending on the size and update frequency of the wiki.

=== Option 1: AutoSitemap Extension ===

The AutoSitemap extension automatically rebuilds the sitemap whenever page content changes.

Events typically triggering regeneration include:

* Article creation
* Article modification
* Article deletion
* Page moves
* File uploads
* Page restoration

==== Advantages ====

* Automatic operation
* No scheduler required
* Near real-time updates
* Simple installation
* Minimal administration

==== Disadvantages ====

* Rebuilds occur frequently
* Resource usage increases with site size
* May become inefficient for large installations

==== Suitable Use Cases ====

* Small knowledgebases
* Documentation portals
* Internal wikis
* Sites with fewer than approximately 10,000 pages

=== Option 2: generateSitemap.php ===

MediaWiki includes maintenance scripts capable of generating sitemap files on demand.

Rather than regenerating whenever content changes, administrators execute the script through:

* Scheduled tasks
* Cron jobs
* Automation platforms
* Orchestration systems

==== Advantages ====

* Highly scalable
* Predictable resource usage
* Supports very large wikis
* Suitable for enterprise deployments

==== Disadvantages ====

* Requires task scheduling
* Changes are not reflected immediately
* Additional administration effort

==== Suitable Use Cases ====

* Large public knowledgebases
* Documentation platforms
* Enterprise MediaWiki deployments
* Sites exceeding tens of thousands of pages

=== Option 3: MediaWiki Sitemap REST API ===

Recent MediaWiki releases include sitemap generation through a REST API endpoint.

Rather than storing a static sitemap file, MediaWiki can generate sitemap content dynamically when requested.

==== Advantages ====

* Built into modern MediaWiki releases
* No external extension required
* Centralised platform support

==== Disadvantages ====

* Dynamic generation may increase server load
* Requires appropriate configuration
* May attract frequent crawler requests

==== Suitable Use Cases ====

* Modern MediaWiki environments
* API-centric deployments
* Administrators wishing to minimise third-party extensions

== Practical Application ==

=== Small Knowledgebases (Under 10,000 Articles) ===

For smaller installations, automatic generation is often the simplest and most effective option.

The administrative overhead is minimal and sitemap updates occur immediately after content changes.

In practice, many smaller knowledgebases benefit from the simplicity of this approach.

=== Medium Knowledgebases ===

As article counts increase, regeneration costs become more noticeable.

At this stage, administrators should consider:

* Regeneration rate limiting
* Namespace exclusions
* Reduced generation frequency

This balances freshness against system performance.

=== Large Knowledgebases ===

Large installations should generally avoid rebuilding entire sitemaps on every page modification.

Scheduled generation is typically preferred because it:

* Reduces server load
* Improves predictability
* Scales more effectively
* Supports sitemap indexes

== Design & Architecture Considerations ==

=== Regeneration on Every Edit ===

Automatic regeneration provides the freshest sitemap possible.

However, this convenience comes at the cost of increased server activity.

As content volume increases, the administrative simplicity may no longer justify the resource consumption.

=== Scheduled Generation ===

Scheduled generation introduces a small delay between content publication and sitemap updates.

For most knowledgebases, this delay is insignificant.

Search engines typically do not require second-by-second awareness of content updates.

=== Namespace Filtering ===

Not every namespace should appear within a public sitemap.

Common exclusions include:

* User pages
* Talk pages
* Template pages
* MediaWiki system pages
* Administrative content

A sitemap should focus on content that delivers value to visitors.

=== Category-Based Sitemap Strategies ===

Large knowledgebases can benefit from dividing content into logical sitemap groups.

Examples include:

* Guides
* Tutorials
* Deep Dives
* Reference Material
* API Documentation

This approach improves management and can assist search engines in understanding site structure.

=== Search Engine Crawl Budget ===

Search engines allocate finite resources when crawling websites.

Poorly configured sitemaps can waste crawl budget by presenting:

* Duplicate content
* Administrative pages
* Low-value content
* Redirect pages

Effective sitemaps help search engines focus on important material.

== Common Pitfalls ==

=== Including Administrative Namespaces ===

Publishing internal administrative content can dilute sitemap quality and waste crawl resources.

Always review namespace inclusion policies.

=== Broken Canonical URLs ===

Incorrect server configuration may result in:

* Mixed protocols
* Internal hostnames
* Duplicate domains

Search engines should only receive canonical URLs.

=== Stale Sitemaps ===

A sitemap is only useful if it reflects current site content.

Regular validation should form part of routine maintenance procedures.

=== Oversized Sitemap Files ===

Very large knowledgebases may exceed practical sitemap limits.

When this occurs, sitemap indexes and segmented sitemap structures should be adopted.

== Recommended Approach ==

=== SEO Knowledgebase Reference Architecture ===

For most MediaWiki-based technical knowledgebases, the following approach provides an effective balance:

* Use AutoSitemap for smaller installations
* Exclude administrative namespaces
* Publish sitemap location in robots.txt
* Submit the sitemap to major search engines
* Monitor indexing through search engine webmaster tools
* Migrate to scheduled generation as content volume grows

This provides a straightforward upgrade path without introducing unnecessary complexity early in the project's lifecycle.

== Decision Matrix ==

{| class="wikitable"
! Knowledgebase Size
! Recommended Approach
|-
| Under 10,000 Pages
| AutoSitemap
|-
| 10,000-50,000 Pages
| AutoSitemap with rate limiting
|-
| Over 50,000 Pages
| generateSitemap.php
|-
| Enterprise Deployments
| Scheduled sitemap indexes
|}

== Key Takeaways ==

* Sitemaps improve content discovery but do not replace good navigation.
* Smaller MediaWiki installations benefit from automatic sitemap generation.
* Larger installations should favour scheduled generation.
* Administrative content should generally be excluded.
* Sitemap strategy should evolve alongside the growth of the Knowledgebase.
* A well-maintained sitemap is an important component of technical SEO.

== Related Topics ==

* [[robots.txt]]
* [[MediaWiki SEO]]
* [[Structured Data]]
* [[Search Engine Crawling]]
* [[Search Engine Indexing]]
* [[Technical SEO]]
* [[Category Design]]
* [[Information Architecture]]

== References ==

* MediaWiki Manual: Sitemap
* MediaWiki Extension: AutoSitemap
* Sitemap Protocol Specification
* Google Search Central Documentation
* Bing Webmaster Guidelines

Sitemap Generation in MediaWiki - Revision history

Dex: /* robots.txt Integration */

Dex: /* robots.txt Integration */