A Go package to parse XML Sitemaps compliant with the Sitemaps.org protocol.

• Recursive parsing

• robots.txt
• XML .xml
• Gzip compressed XML .xml.gz

go get github.com/aafeher/go-sitemap-parser

import "github.com/aafeher/go-sitemap-parser"

To create a new instance with default settings, you can simply call the New() function.

s := sitemap.New()

• userAgent: "go-sitemap-parser (+https://github.com/aafeher/go-sitemap-parser/blob/main/README.md)"
• fetchTimeout: 3 seconds

To set the user agent, use the SetUserAgent() function.

s := sitemap.New()
s = s.SetUserAgent("YourUserAgent")

... or ...

s := sitemap.New().SetUserAgent("YourUserAgent")

To set the fetch timeout, use the SetFetchTimeout() function. It should be specified in seconds as an uint8 value.

s := sitemap.New()
s = s.SetFetchTimeout(10)

... or ...

s := sitemap.New().SetFetchTimeout(10)

In both cases, the functions return a pointer to the main object of the package, allowing you to chain these setting methods in a fluent interface style:

s := sitemap.New().SetUserAgent("YourUserAgent").SetFetchTimeout(10)

Once you have properly initialized and configured your instance, you can parse sitemaps using the Parse() function.

The Parse() function takes in two parameters:

• url: the URL of the sitemap to be parsed,
  • url can be a robots.txt or sitemapindex or sitemap (urlset)
• urlContent: an optional string pointer for the content of the URL.

If you wish to provide the content yourself, pass the content as the second parameter. If not, simply pass nil and the function will fetch the content on its own. The Parse() function performs concurrent parsing and fetching optimized by the use of Go's goroutines and sync package, ensuring efficient sitemap handling.

s, err := s.Parse("https://www.sitemaps.org/sitemap.xml", nil)

In this example, sitemap is parsed from "https://www.sitemaps.org/sitemap.xml". The function fetches the content itself, as we passed nil as the urlContent.

Examples can be found in /examples.