thesciencemuseum / collectionsonline-sitemap Goto Github PK

The lambda must be triggered by an event which will depend on the frequency we want to update the sitemap. Once a week? At each new item inserted in elasticsearch (might cost a lot if new items are added often)?

Another solution might be to trigger the lambda manually, easy to implement but maybe not very effecient and time consuming for @jamieu.

@jamieu do you have an idea on long we want to wait before triggering the lambda? Or is there a specific event we can monitor which will triger the lambda?

@Danwhy @nelsonic do you have any more specific ideas on this?

Can we deploy the new Lambda to aws when a PR is merged?

Add:

• modification date (of original record)
  <lastmod>2006-11-18</lastmod>

• priority of SERP pages and FONDs records.
  <priority>0.7</priority> SERP pages (categories, museums)
  <priority>0.6</priority> Top level FOND
  <priority>0.5</priority> All other records

From https://aws.amazon.com/lambda/faqs/:

Q: How long can an AWS Lambda function execute?
All calls made to AWS Lambda must complete execution within 300 seconds. The default timeout is 3 seconds, but you can set the timeout to any value between 1 and 300 seconds.

So we need to be sure that the lambda can finish the work in the 300 seconds timeout (5mn). At the moment we are creating the sitemap by scanning all data from elasticsearch. The lambda has the time to finish to create the sitemap but if the database content is getting bigger and bigger we migth arrive to the limit.

• How long the lambda take to scan the database?

We might be able to speed the process by only getting the recently updated items or the new intems from Elasticsearch by checking the version of the documents