Multiple Sitemaps In NextJs App Router & Sitemap Index

By Raddy in NextJs · August 16, 2024

This article is all about Next.js XML sitemaps. I’ll walk you through creating multiple sitemaps manually and dynamically. To tie everything together, I’ll show you how I created a Sitemap Index as well.

You can manually submit websites with only a few sitemaps to search engines. However, if your site is large and continuously expanding, it’s more efficient to create a Sitemap Index file that consolidates all your sitemaps.

Some examples have been taken from the official NextJs doc.

Add your Sitemap in robots.txt

It’s a good practice to let search engines know where your sitemap is by including it in robots.txt. Even if you submit your sitemap to Google, other search engines might not know about it and robots.txt is usually the first thing that bots crawl. Not everybody’s sitemap is at the root of your doc, as we will see later in the article.

Reasons to submit:

Include your sitemap location robots.txt to inform search engines.
robots.txt is typically the first filebot crawl.
Not all sitemaps are in the root directory.

In your app directory add your sitemap like this;

User-agent: ia_archiver

Sitemap: https://yourwebsite.com/sitemap.xml OR https://yourwebsite.com/sitemap_index.xml

Basic Sitemap Example

To create the most basic sitemap in NextJs, you can create a sitemap.xml file at the root of your project.

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://acme.com</loc>
    <lastmod>2023-04-06T15:02:24.021Z</lastmod>
    <changefreq>yearly</changefreq>
    <priority>1</priority>
  </url>
  <url>
    <loc>https://acme.com/about</loc>
    <lastmod>2023-04-06T15:02:24.021Z</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://acme.com/blog</loc>
    <lastmod>2023-04-06T15:02:24.021Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.5</priority>
  </url>
</urlset>

Generating a sitemap using code

To generate the same sitemap using code you can do the following:

export default function sitemap() {
  return [
    {
      url: 'https://acme.com',
      lastModified: new Date(),
      changeFrequency: 'yearly',
      priority: 1,
    },
    {
      url: 'https://acme.com/about',
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
    {
      url: 'https://acme.com/blog',
      lastModified: new Date(),
      changeFrequency: 'weekly',
      priority: 0.5,
    },
  ]
}

Result:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://acme.com</loc>
    <lastmod>2023-04-06T15:02:24.021Z</lastmod>
    <changefreq>yearly</changefreq>
    <priority>1</priority>
  </url>
  <url>
    <loc>https://acme.com/about</loc>
    <lastmod>2023-04-06T15:02:24.021Z</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://acme.com/blog</loc>
    <lastmod>2023-04-06T15:02:24.021Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.5</priority>
  </url>
</urlset>

Generating a Dynamic Sitemap from Database (MongoDB)

If you wish to create a basic sitemap where the data comes from your database, you can use the following example:

export const revalidate = 3600 // one hour
import { connectToDatabase } from "@/utils/connectMongo";

export default async function sitemap() {
  
  const client = await connectToDatabase();
  const db = client.db("database");
  let data = await db.collection("lens").find({}).toArray();

  const lens = data.map((item) => ({
    url: `${process.env.NEXT_WEBSITE_URL}/lens/${item.slug}`,
    lastModified: item.updated_at || item.created_at,
    changefreq: "monthly",
    priority: 0.6,
  })); 

  return [
    {
      url: "https://gearxplorer.com/",
      lastModified: new Date(),
      changefreq: "daily",
      priority: 1,
    },
    {
      url: "https://gearxplorer.com/about",
      lastModified: new Date(),
      changefreq: "monthly",
    },
    ...lens,
  ];
}

Multiple Sitemaps By Nesting

In this case, all you need to do to create another XML sitemap is to nest it into a route. For example, if we put sitemap.xml in a “products” route, the URL will become yourwebsite.com/products/sitemap.xml.

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://acme.com</loc>
    <lastmod>2023-04-06T15:02:24.021Z</lastmod>
    <changefreq>yearly</changefreq>
    <priority>1</priority>
  </url>
  <url>
    <loc>https://acme.com/about</loc>
    <lastmod>2023-04-06T15:02:24.021Z</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://acme.com/blog</loc>
    <lastmod>2023-04-06T15:02:24.021Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.5</priority>
  </url>
</urlset>

Multiple Sitemaps By generateSitemaps Function

Again I will be using MongoDb for this example, but you can easily convert the query into SQL. Also, there is an example with SQL on the official doc here: generateSitemaps

import { connectToDatabase } from "@/utils/connectMongo";

export async function generateSitemaps() {
  // Fetch the total number of products and calculate the number of sitemaps needed
  return [{ id: 0 }, { id: 1 }, { id: 2 }, { id: 3 }]
}
 
export default async function sitemap({ id }) {
  const client = await connectToDatabase();
  const db = client.db("database");
  // Google's limit is 50,000 URLs per sitemap
  const start = id * 50000
  const limit = 50000

  const products = await db.collection('lens').find().skip(start).limit(limit).toArray();
  
  const product = data.map((item) => ({
    url: `${process.env.NEXT_WEBSITE_URL}/product/${item.slug}`,
    lastModified: item.updated_at || item.created_at,
    changefreq: "monthly",
    priority: 0.6,
  })); 

  return [
    ...product,
  ];
}

In production, available at /.../sitemap/[id].xml. For example, /product/sitemap/1.xml.

In development /.../sitemap.xml/[id]. For example, /product/sitemap.xml/1.

import { connectToDatabase } from "@/utils/connectMongo";

export const revalidate = 3600 // one hour


export async function generateSitemaps() {
  const client = await connectToDatabase();
   const db = client.db("database");
  // Fetch the total number of products and calculate the number of sitemaps needed
  // Fetch the total number of products
  const totalProducts = await db.collection('lens').countDocuments();

  // Calculate the number of sitemaps needed (3 products per sitemap)
  const productsPerSitemap = 50000;
  const numberOfSitemaps = Math.ceil(totalProducts / productsPerSitemap);

  // Generate an array of sitemap objects
  const sitemaps = Array.from({ length: numberOfSitemaps }, (_, index) => ({ id: index }));

  return sitemaps;
}

export default async function sitemap({id}) {
  const start = id * 5000
  const limit = 5000

  const data = await db.collection('lens').find().skip(start).limit(limit).toArray();

  const product = data.map((item) => ({
    url: `${process.env.NEXT_WEBSITE_URL}/product/${item.slug}`,
    lastModified: item.updated_at || item.created_at,
    changefreq: "monthly",
    priority: 0.6,
  })); 

  return [
    ...product,
  ];
}

When you build your project, you should be able to see your newly generated sitemaps like this:

Route (app)                              Size     First Load JS
┌ ○ /                                    209 B           182 kB
├ ○ /_not-found                          0 B                0 B
├ ○ /about                               185 B          89.8 kB
├ λ /api/search                          0 B                0 B
├ λ /products                             5.88 kB         102 kB
├ ● /products/sitemap/[__metadata_id__]   0 B                0 B
├   ├ /products/sitemap/0.xml
├   ├ /products/sitemap/1.xml
├   ├ /products/sitemap/2.xml
├   └ [+15 more paths]
├ ○ /contact

When Submitted to Google

You can submit your sitemaps individually to Google. The result should be something like this:

Sitemap Index

Sitemap index files simplify managing multiple sitemaps, improving organization and search engine accessibility. They enhance page discoverability through a well-structured approach. This guide covers the advantages and recommended formats for sitemap index files.

Benefits:

Organization of multiple sitemaps efficiently
Scalability
Crawl Efficiency

Good to know:

You can submit up to 500 sitemap index files for each site in your Search Console account.
A sitemap index file may have up to 50,000 loc tags.

A sitemap index is useful if you have many individual XML sitemaps. It simplifies the task of submitting them all individually.

Official Google Documentation

To avoid breaking the original sitemap.xml/js/ts file in our app directory we can call our Sitemap “sitemap_index.xml”.

Create a normal route (folder) with the name of “sitemap_index.xml”. Here we’ll need to use the NextJs Route Handlers (by the way, this can also be done as an API).

import { NextResponse } from 'next/server';
import { connectToDatabase } from "@/utils/connectMongo";

// Calculate and output sitemap URLs ex sitemap/1.xml
export async function generateSitemaps() {
  const client = await connectToDatabase();
  const db = client.db("database");

    // Fetch the total number of products
    const totalProducts = await db.collection('lens').countDocuments();

    // Calculate the number of sitemaps needed (10 products per sitemap)
    const productsPerSitemap = 50000;
    const numberOfSitemaps = Math.ceil(totalProducts / productsPerSitemap);
  
    // Generate an array of sitemap objects
    const sitemaps = Array.from({ length: numberOfSitemaps }, (_, index) => ({
      id: index,
      url: `${process.env.NEXT_WEBSITE_URL}/compare/sitemap/${index}.xml`
    }));
  
    return sitemaps;
  }

// cache test
export async function GET() {
  try {
    // Generate sitemaps
    const dynamicSitemaps = await generateSitemaps();

    // Combine static and dynamic sitemaps
    const sitemaps = [
      `${process.env.NEXT_WEBSITE_URL}/sitemap.xml`,
      `${process.env.NEXT_WEBSITE_URL}/lenses/sitemap.xml`,
      ...dynamicSitemaps.map(sitemap => sitemap.url)
    ];

    console.log('Generated sitemaps:', sitemaps);

    const sitemapIndexXML = await buildSitemapIndex(sitemaps);

    return new NextResponse(sitemapIndexXML, {
      headers: {
        "Content-Type": "application/xml",
        "Content-Length": Buffer.byteLength(sitemapIndexXML).toString(),
      },
    });
  } catch (error) {
    console.error('Error generating sitemap index:', error);
    return NextResponse.error();
  }
}

async function buildSitemapIndex(sitemaps) {
  let xml = '<?xml version="1.0" encoding="UTF-8"?>';
  xml += '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">';

  for (const sitemapURL of sitemaps) {
    xml += "<sitemap>";
    xml += `<loc>${sitemapURL}</loc>`;
    xml += "</sitemap>";
  }

  xml += "</sitemapindex>";
  return xml;
}

That’s all.

Author

Hi! I’m Raddy. I am a web developer, often making YouTube Videos, Blog Posts and side projects. Enquiries: [email protected]

Tags: app router, dynamic sitemap, nextjs, sitemap index

Prabhas kumar says:

August 22, 2024 at 7:51 am

Great blog post! It clearly explains the process of setting up multiple sitemaps in a Next.js App Router environment and how to effectively use a Sitemap Index. The step-by-step guide is easy to follow, making it accessible even for those new to Next.js. The examples provided are practical and help in understanding how to implement this in real-world projects. A must-read for anyone looking to improve their website’s SEO and manage large content sites efficiently!

Reply

Raddy Website Design & Development Tutorials