turnersoftware / sitemaptools Goto Github PK

Need to remove the requirement to the Robots Nuget package - probably can throw together a really basic implementation to read the rows of a "robots" file to find sitemaps.

Thanks for the library James, it has proven very reliable and easy to use!

EDITED: (because as always, as soon as you submit a github issue you realise where you're wrong).

The problem I'm experiencing is in SitemapQuery.GetAllSitemapsForDomainAsync(domain).

The problem occurs when a webserver returns 200-OK, but contains a "whoops" 404 style html page response.

Line 132 of SitemapQuery.cs correctly identifies the text/html response, but instead of ignoring it as an invalid response, it throws an exception because there's no parser for html (expected).

I think it would be sensible to update DiscoverSitemapsAsync function, Line 100 where it checks for successful response code, to also have it check that the content-type is one of those mentioned in the SitemapTypeMapping dictionary, and if not, consider it an invalid sitemap.

Hi there,

Thanks for this library, I'm myself unfamiliar with sitemaps but hopefully this lib seems to be able to solve my problem.

Can you please provide code examples for this just like you have for https://github.com/TurnerSoftware/RobotsExclusionTools
etc

What i want to do is provide few sitemap (xml based) links, where each sitemap itself will hv location to other sitemaps and so on finally reaching to actual records.

So what methods etc should i use for this?

Regards