wilr / silverstripe-googlesitemaps Goto Github PK

Hey folks! Client of mine just noticed that the sitemap for one of his subsites isn't returning any pages. Looks like the something isn't working right with the Subsites add-on (https://github.com/silverstripe/silverstripe-subsites). The main site returns results in the sitemap (although there appears to be some odd duplicates here and there). But any subsites return empty results. Odd.

Currently running on SilverStripe v3.4 with the Subsite module set for '1.*@dev' and Google Sitemaps set for the latest dev-master version. Tried different versions of each module but nothing worked.

Any ideas on what the problem might be?

Before the updates I was able to reach sitemap.xml perfectly in Chrome. After taking the last source code I can no longer reach sitemap.xml + it also breaks Chrome. All the tabs that are open with the website.

For example:

Tab 1: http://www.github.com/home
Tab 2: http://www.github.com/home2
Tab 3: http://www.github.com/home3
Tab 4: http://www.github.com/sitemap.xml

After going to sitemap.xml, all the three other tabs break in Chrome.

From http://open.silverstripe.org/ticket/1646 - see there for more discussion

Google's suggestion in the sitemaps protocol is to provide it as a .gz file, and not just as an .xml file carried over mod_deflate.

Enable support for accessing /sitemap.gz (or is sitemap.xml.gz?)

This will increase the CPU load for producing the sitemap, and so it would be best for this to be cached, say to /assets/.sitemap.xml.gz

E.g.

if (cachefile < 1 hour old OR ?flush=1 ) create-xml-file-and-save-to-gzipped-cache-file()

header temporary redirect to cache file()

Does the protocol support putting a comment inside the xml file to say the generation time?

The subsites module may have a different result for

$page->AbsoluteLink()

than is actually being used in the browser - especially during the test phase when one might have a test url as well as the real one configured

Thus the line 154 in GoogleSitemap.php fails:

if(($pageHttp == $hostHttp) && !($page instanceof ErrorPage)) {

I cant see any good reason why this comparison is made and it breaks subsites under test therefore I changed it to

if(!($page instanceof ErrorPage)) {

After this its a question of sorting the domain order, or primary domain settings to get what you want when you are using a different test url.

After I put this line into the config file, I get a white screen on all front-end or back-end URLs

GoogleSitemap::enable_google_notificaton();

(Had no time to debug, but I got this on 3 different systems with current git Version of framework, cms and googlesitemaps module)

From http://open.silverstripe.org/ticket/4790

When setting GoogleSiteMap? setting Not indexed is not set to selected in CMS dropdown due to the -1 value.

Patch here: http://open.silverstripe.org/attachment/ticket/4790/GoogleSitemapDecorator.patch

Hi, i am just testing the googlesitemaps module for SS 3.2. There is a problem to get a valid .xml because of the $Priority. in my sitemap.xml $Priority is not correctely formated as a float: 0,9. Becuase of this i get an NaN% instead of a value.

I am using the translatable module. In my _config.php i have set: setlocale (LC_ALL, 'de_DE@euro', 'de_DE.UTF-8', 'de_DE', 'de', 'ge');

Is there a way to set setlocale(LC_NUMERIC, 'en_US') just for the sitemap.xml?

When a link that ends with a #anchortag it has a malformed Last Edited date. For example -001-11-30 00:00. Links that are actually anchors should have the same Last Edited date as the page they reside on.

SS version 3.1.5
googlesitemaps master branch.

From http://open.silverstripe.org/ticket/5590

List assets in the googlesitemap and provide update frequency and priority meta-data fields.

You should probably only list assets that have been linked to from at least one page. This limitation should be configurable.

From http://open.silverstripe.org/ticket/5062

Google has a specific format for news, which are listed on news.google.com. http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=74288

From http://open.silverstripe.org/ticket/2799

My sitemap.xml sets the changefreq of all pages to "always". It happens on some sites, not on all. Google dislikes that very much.

See : www.fahrenheitstore.nl/sitemap.xml for an example.

Hi
How to modify module for https://support.google.com/webmasters/answer/2620865?hl=en&ref_topic=2370587 Use a sitemap to indicate alternate language pages ?

The docs state that a DataObject needs the canView method.

But this this method is never called?

https://github.com/silverstripe-labs/silverstripe-googlesitemaps/blob/master/docs/en/index.md

With about 4 published pages (included in search) the $sitemaps ArrayList appears empty for me at this point:
https://github.com/silverstripe-labs/silverstripe-googlesitemaps/blob/master/code/GoogleSitemap.php#L286

Using this configuration:

---
Name: customgooglesitemaps
After: googlesitemaps

---
GoogleSitemap:
    enabled: true
    objects_per_sitemap: 999
    google_notification_enabled: false
    use_show_in_search: true

Using this version of SilverStripe:

silverstripe/cms             3.1.x-dev d65e5b6  The SilverStripe Content Management System
silverstripe/framework       3.1.x-dev d892629  The SilverStripe framework

Currently I need the ability to customize how this module retrieves pages from the database in ::get_sitemaps() and in ::get_items(). However, this is not (easily) possible since this interface is completely static. If it were instance-level (and extending Object), I could decorate GoogleSitemap object with my own custom extensions and then setup a call to $this->extend("alterDataList", $instances, $class) with my own ->alterDataList() method to facilitate advanced modification of queries going to the database.

This is necessary not only future general extensibility and separation of concerns, but also because (in my case) I have many thousands of pages in my system which need to be better segmented on a per-domain basis (instead of just calling ->canIncludeInGoogleSitemap() on every single one of the thousands of pages in the database).

I will submit a relatively low impact pull request which demonstrates this as a proof of concept, including backward compatibility with the old static API 😄

There has been a lot of changes in the project since the last release. It would be great to tag a new release for packagist.

Latest release missing some good pull requests that have been merged to master.

In some cases where DataObjects are registered, and that override the canView method (such as when posts are to be moderated before viewing, or could have some automatic expiration built in) it's possible for a sitemap file to be generated that's completely empty.

For example, I have the following dataobject Classified that expires after a set number of days.

My sitemap.xml file looks like the below;

<sitemapindex>
    <sitemap>
        <loc>http://www.mysite.co.nz/sitemap.xml/sitemap/SiteTree/1</loc>
        <lastmod>2013-09-19</lastmod>
    </sitemap>
    <sitemap>
        <loc>http://www.mysite.co.nz/sitemap.xml/sitemap/Classified/1</loc>
        <lastmod>2013-12-09</lastmod>
    </sitemap>
    <sitemap>
        <loc>http://www.mysite.co.nz/sitemap.xml/sitemap/Classified/2</loc>
        <lastmod>2014-01-01</lastmod>
    </sitemap>
    <sitemap>
        <loc>http://www.mysite.co.nz/sitemap.xml/sitemap/Classified/3</loc>
        <lastmod>2014-01-05</lastmod>
    </sitemap>
</sitemapindex>

in this case the older classifieds, while they still exist as dataobjects, return a value of false for canView (as they should), and thus do not appear in the individual sitemaps, but this means that the last two sitemaps are blank; Only Classifieds/1 has any entries, and google complains about /2 and /3 with errors.

A fix could either be:

• The list of items is filtered by canIncludeInGoogleSitemap (least efficient, since it is o(n) speed).
• Set some kind of extension in GoogleSitemap::get_sitemaps and GoogleSitemap::get_items to augment the DataList prior to page segmentation. Maybe something like singleton($class)->filterGoogleSitemapItems($list);
• As a variation of the second option, there could perhaps some kind of query parameter on the DataList that's picked up by GoogleSitemapExtension.

The first option is the simplest, but least efficient, while the second allows items to be pre-filtered at the point of query, but requires additional code on the user's part.

As an aside, I apologise for the lack of attention to my other issue; I'll hopefully get more time in the coming months to review my outstanding PRs and issues. :)

class GoogleSitemapExtension extends DataExtension {

    ....

    public function canIncludeInGoogleSitemap() {
...
        $this->owner->invokeWithExtensions('alterCanIncludeInGoogleSitemap', $can);
...
        return $can;
    }

    ....
}

The call to alterCanIncludeInGoogleSitemap does not have any effect as the invokeWithExtensions $argument argument is a call by-value and not a call-by-reference:

public function invokeWithExtensions($method, $argument = null) 

I have seen in a WordPress SEO Plugin (by yoast), it is possible to include Image URLs in Sitemaps, so the images find a faster way into google images.
In that plugin it is a feature you can enable/disable.

XML looks like this:

<url>
        <loc>http://www.yourwebsite.de/gallery-page/</loc>
        <lastmod>2012-09-25T09:18:01+00:00</lastmod>
        <changefreq>weekly</changefreq>
        <priority>0.6</priority>
        <image:image>
            <image:loc>http://www.yourwebsite.de/imageURL.jpg</image:loc>
            <image:caption>Caption Image 1</image:caption>
        </image:image>
        <image:image>
            <image:loc>http://www.yourwebsite.de/image2URL.png</image:loc>
            <image:caption>Nice Image Caption</image:caption>
        </image:image>
</url>

Same is with Video:

<url>
 ...
  <video:video>
     <video:thumbnail_loc>http://www.xml-sitemaps.com/gen/data/d3f06a1dce1c85e5542c2adc8437d394.jpg</video:thumbnail_loc> 
     <video:title>Is it still important to offer a site map to users?</video:title>
     <video:description>In addition to using Webmaster Tools to submit an XML Sitemap to Google, how important is it to also have a sitemap for users? It seems like nobody does this anymore. Is it still important in Web 2.0?</video:description>
     <video:publication_date>2010-10-05T18:52:47.000Z</video:publication_date>
     <video:player_loc allow_embed="yes" autoplay="ap=1">http://www.youtube.com/v/t5LIlkhxl2s</video:player_loc>
     <video:duration>110</video:duration>
   </video:video>

(Code from here: http://www.xml-sitemaps.com/sitemap_video.xml)

For some reason, the nice styling for humans to view is gone for me. I'm on latest master.

From http://open.silverstripe.org/ticket/4495

on all my sites, i install google sitemap. This is more superstition than being based on any good rational reason. I believe that it makes live easier for Googlebot and I would do anything to help it - hence I add it. I know what you dont need it, but it might speed up indexing.

Anyway - now, the googlesitemap has been given its own tab so that the user can mark priority. Tabs are precious as too many tabs will turn-off any CMS user. We are supposed to keep it simple for them. Over the last few months we have talked a lot about weight, this again, adds weight that is not necessary in my opinion.

Marking priority for each page is something that very few people will do. Just watch average site owner. They are NOT interested in doing stuff unless it will directly benefit them, they understand this benefit and they see direct results.

Also, having a tab for just one field is a waste in my opinion. Therefore I recommend

that the priority field can be added using a _config setting with the default being "not added".
OR

that the priority field is added to the metatag section as the metatag section is all about invisible content and stuff relation to search engines.
We could even consider renaming the metatags tab "Search Engine" and moving the Page Meta Title to the content Tab. I would prefer that because then you can see all the titles at the same time.

also, the little (?) next to the priority dropdown does not go anywhere... I would suggest a few changes:

make the additional information a pop-up or let it open in a new window - no-one wants to loose their entire CMS to check out a help item.
either do it for all "hard-to-understand" fields or not at all. A "tell me more about this field thing" should be standardised throughout the CMS. I like the idea, but I like the idea even better IF the fields are so well-labelled that we do not need it. For the google sitemap priority field, we could say something like " tell Search Engines the relative priority of each page within your website" - we really dont need to add more.
at present the link is broken.

From http://open.silverstripe.org/ticket/6780

I installed the Google sitemaps on a Silverstripe 3.0.0 alpha1 site, and the main content editing panel in the CMS disappeared (i.e., no TinyMCE to edit the page's contents). Instead, the CMS page editing panel ended up with two "Content" tabs. The main "Content" tab misses the content editing panel, and the second tab contains the Google sitemaps GUI.

The problem lies in line 42 of GoogleSitemapDecorator?.php:

$tabset = $fields->findOrMakeTab('Root.Content');

Changing that line to:

$tabset = $fields->findOrMakeTab('Root.Main');

will eliminate the additional Content tab, and put the Google Sitemaps GUI underneath the content editing panel.

I fixed it here: https://github.com/wolfv/silverstripe-googlesitemaps/tree/6870-sitemaps-for-3.0

I don't know, whats the best way to go with 3.0-specific issues like this one?

master -> dev-master doesn't get updated on packagist.

When creating websites with multiple languages, shouldn't this module recognize all of the languages? On my site it currently shows only the site structure of the default language.

whats the best version to use for the current realse of Silverstripe SilverStripe CMS v3.0.5 or is it still in ongoing development.

i tried V1.1 and that seems to only create a sitemap.xml with 1 page.

the master version didn't work .. i think thats because i see its says its updated for 3.1

Currently some tests are marked as incomplete because the CMS module is not installed.

Travis should install the CMS module as part of the test set-up to allow the full test-suite to run

Hi,

is it just me having problems with this module? :-(

After installing the module, everything I get is one entry on the sitemap:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>
http://www.kago-pelletheizung.de/sitemap.xml/sitemap/SiteTree/1
</loc>
<lastmod>2013-01-17</lastmod>
</sitemap>
</sitemapindex>

Is it a bug or am I a bug?

Best, Mario

Google Webmaster Tools says "URL not allowed" for the external links.

Have a look here: http://www.comune.mondovi.cn.it/sitemap.xml/sitemap/SiteTree/1

This relates loosely to my older issue (#90) and PR (#91). I would like to discuss the existence updateGoogleSitemaps and updateGoogleSitemapItems extension hook. Currently the functionality here is such that these will operate on the final results coming from->getSitemaps() and ->getItems() but it's only called in the controller. While both of those core methods do already tie into the newer alterDataList extension hook, it may still make sense to keep them around since they affect the results coming out to the page, so that's fine.

However, for consistency, shouldn't these be centralized and located instead inside those methods instead of only in the controller to ensure the output coming from those methods matches what you would see in the final template? Why alter this only in the controller?

i have in the server installed SSL certificate and domain is always using https. And when i trying to reflush the sitemap.xml i get an error with page not found. When i trying to run in the console:

php framework/cli-script.php /sitemap.xml "flush=all" i get the content of the xml.

problem: not working with https.

Hi,

I get the following error when I change the priority value of a page and then save it:

Fatal error: Access to undeclared static property: GoogleSitemap::$enabled in /googlesitemaps/code/GoogleSitemap.php on line 239

I created the "_config" folder like mentioned in the docs and there is "enabled: true". When I comment out line 239 to 241 then it seems to work.

Best, Mario

In file: GoogleSitemapSiteTreeExtension.php line 110:

return (is_float($priority) && $priority <= 1.0) ? $priority : 0.5;

For some reason "is_float($priority)" returns false, and therefore all set priorities are defaulted to 0.5

For example, I have a site which is about famous people, which are linked with movies.
Now, what I want is:
http://mysite.com/person/{personname}
This works fine.
But, I also have a sub-link "today", showing the movies that are on TV today, which feature this person.
Thus, my custom extra in the sitemap needs to be:
http://mysite.com/person/{personname}/today

But, the /today isn't a DO or anything, it's just an "OtherID" feature I implemented.

hello,
there is an xml error when calling mysite.com/sitemap.xml
error on line 11 at column 8: Opening and ending tag mismatch: base line 0 and head

I used newest versions of silverstripe cms and framework from git.

Regards,
Florian

All sitemaps except sitemap.xml itself give me the following error in Google WMT:
Missing XML tag

This required tag is missing. Please add it and resubmit.

1

Parent tag: urlset
Tag: url

Hi there,
I know google is No.1, but notifications to other search engines about sitemap.xml would enhance this module.
Hendrik

Title speaks for itself.

I am investigating multiple-domain based localisation for a single site, and have been following the google sitemap guide at https://support.google.com/webmasters/answer/75712?hl=en

This is not so much a problem as a critique of the method followed by GoogleSitemapExtension::canIncludeInSitemap(). This code seems to explicitly exclude items that may be visible under a different hostname.

Why is it necessary to limit to a single domain if the specification doesn't prohibit it?

when add this module on my SS3 project have this error...

Fatal error: Object::add_extension() - Can't find extension class for "" in /mywebsite/framework/core/Object.php on line 460

in _config file added:

// Google SiteMaps
GoogleSitemap::enable_google_notification();

and add folder _config inside mysite folder with googlesitemaps.yml.

When accessing google priority in the Google Site tree extension, the check if priority is a float is returning false and therefore returning 0.5 for all manually set pages.

I added "$priority = (float)$this->owner->getField('Priority');" on line 74, when accessing the priority to fix the issue.

From http://open.silverstripe.org/ticket/4892

Re googlesitemaps current build, I noticed choosing the option of "Not Indexed' did not seem to persist when reloading a page and editing something else on it.

It saves to the database correctly when first changed, but then saves incorrectly as 0 or null if one reloads the page in the admin interface and then saves and publishes again. The problem is that the DropdownField? selected value is incorrectly set to "Auto-set based on page depth" (0 or null) when generating the Google Sitemap tab, so reloading and saving the page changes this value without user intervention.

I fixed this when I noticed the code in the function getPriority() returns 0 when it reads a -1 from the database. When I removed this conditional and let it return a -1, it behaves correctly and the value of the dropdown is set to what is stored in the db and hence does not change without user intervention.

I am unsure however, if this 'bug' was intentional (the presence of this conditional raises the question), and by fixing this problem I may have broken something else that counted on that conditional returning 0 when the db yields -1.

Does that conditional serve some other purpose? If not, I suggest committing this to the svn tree.

--tim

in GoogleSitemapDecorator?.php

function getPriority() {

if(!$this->owner->getField('Priority')) {

$parentStack = $this->owner->parentStack(); $numParents = is_array($parentStack) ? count($parentStack) - 1: 0; return max(0.1, 1.0 - ($numParents / 10));

} elseif($this->owner->getField('Priority') == -1) {

return 0;

} else {

return $this->owner->getField('Priority');

}

}

was changed to

function getPriority() {

if(!$this->owner->getField('Priority')) {

$parentStack = $this->owner->parentStack(); $numParents = is_array($parentStack) ? count($parentStack) - 1: 0; return max(0.1, 1.0 - ($numParents / 10));

} else {

return $this->owner->getField('Priority');

}

}

see also my message to the dev google group at:

http://groups.google.com/group/silverstripe-dev/browse_thread/thread/2db9de143457e48a/76cebe3d0c1230f3#76cebe3d0c1230f3

See ticket: http://open.silverstripe.org/ticket/4790

after updating my json file from 3.0.5 to 3.1.x-dev. there's an error. (screenshot)

This page contains the following errors:
error on line 11 at column 8: Opening and ending tag mismatch: base line 0 and head
Below is a rendering of the page up to the first error.

Hmmm... I'm sure I have set everything up correctly but I get this error when trying to visit

mysite/sitemap.xml

I have done a build, flush and added googlesitemaps.yml file.

Cheers

There hasn't been a release for 11 months. Would be nice to get the fixes since then into a release.

There is an error in GoogleSitemap.ss

instead of <% if $ChangeFreq %>$ChangeFreq<% end_if %>

it should be:

<% if $getChangeFrequency %>$getChangeFrequency<% end_if %> to get a value.

From http://open.silverstripe.org/ticket/5061

At the moment, a website with the googlesitemaps module will recreate the file whenever any of its contained pages has been changed or a new one has been created. This becomes a problem with websites with several thousand pages.

Partitioning can relieve this into multiple files, some of which might never need to be regenerated. A good partitioning scheme seems to be by year based on Created date.

See a reference implementation from Hamish: http://trac.silverstripe.com/silverstripe/browser/projects/peoplebeforeprofit/trunk/mysite/code/Sitemap.php http://trac.silverstripe.com/silverstripe/browser/projects/peoplebeforeprofit/trunk/mysite/_config.php#L66

Its specific to Google news entries, but the year partitioning and routing could be easily reused.

From Google's recommendations on http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184

A Sitemap file can contain no more than 50,000 URLs and be no larger than 10MB when uncompressed. If your Sitemap is larger than this, break it into several smaller Sitemaps. These limits help ensure that your web server is not overloaded by serving large files to Google.

Also for reference - http://www.sitemaps.org/protocol.php#index

I've got XML Parsing error.

Using silverstripe 3.1.6 / framework 3.1.6

XML Parsing Error: mismatched tag. Expected: .
Location: http://localhost/ss316/sitemap.xml
Line Number 11, Column 3:
--^

Hello,

I've followed the instructions on how to include dataobjects in the sitemap and I get the following error.

[User Error] Uncaught Exception: Object->__call(): the method 'absolutelink' does not exist on 'GoogleSitemapController'
GET /smart-numbers-v2/sitemap.xml/sitemap/Prefix/1

Line 750 in C:\xampp\htdocs\smart-numbers-v2\framework\core\Object.php
Source

741 default :
742 throw new Exception (
743 "Object->__call(): extra method $method is invalid on $this->class:"
744 . var_export($config, true)
745 );
746 }
747 } else {
748 // Please do not change the exception code number below.
749 $class = get_class($this);
750 throw new Exception("Object->__call(): the method '$method' does not exist on '$class'", 2175);
751 }
752 }
753
754 // --------------------------------------------------------------------------------------------------------------
755
756 /**

Trace

Object->__call(absoluteLink,Array)
Prefix.php:142
GoogleSitemapController->absoluteLink()
Prefix.php:142
Prefix->Link()
Prefix.php:147
Prefix->AbsoluteLink()
GoogleSitemapExtension.php:19
GoogleSitemapExtension->canIncludeInGoogleSitemap()
call_user_func_array(Array,Array)
Object.php:718
Object->__call(canIncludeInGoogleSitemap,Array)
GoogleSitemap.php:209
Prefix->canIncludeInGoogleSitemap()
GoogleSitemap.php:209
GoogleSitemap::get_items(Prefix,1)
GoogleSitemapController.php:63

Issues so far I have found:

_config.php
SiteTree::add_extension('SiteTree','GoogleSitemapSiteTreeExtension');
-- should be --
Object::add_extension('SiteTree','GoogleSitemapSiteTreeExtension');

GoogleSitemapController.php
private static $allowed_actions = array(
'index',
'sitemap'
);
--- should be ---
public static $allowed_actions = array(
'index',
'sitemap'
);

But after that I get lost :(
If I visit /sitemap.xml I get the following only:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>http://ofs.nimeso/sitemap.xml/sitemap/SiteTree/1</loc>
<lastmod>2013-06-23</lastmod>
</sitemap>
</sitemapindex>

Any thoughts?

Cheers heaps for all your hard work :)