Giter VIP home page Giter VIP logo

gatsby-plugin-algolia's Introduction

Gatsby plugin Algolia

This plugin is in beta and not officially supported yet

Feel free to open issues for any questions or ideas

You can specify a list of queries to run and how to transform them into an array of objects to index. When you run gatsby build, it will publish those to Algolia.

graph LR
    A[Source 1] --> |query| Gatsby
    B[Source 2] --> |query| Gatsby
    C[Source 3] --> |query| Gatsby

    Gatsby --> |gatsby build| Algolia

Here we have an example with some data that might not be very relevant, but will work with the default configuration of gatsby new

yarn add gatsby-plugin-algolia

First add credentials to a .env file, which you won't commit. If you track this in your file, and especially if the site is open source, you will leak your admin API key. This would mean anyone is able to change anything on your Algolia index.

// .env.production
ALGOLIA_APP_ID=XXX
ALGOLIA_API_KEY=XXX
ALGOLIA_INDEX_NAME=XXX
require('dotenv').config({
  path: `.env.${process.env.NODE_ENV}`,
});

// gatsby-config.js
const myQuery = `
  query {
    pages: allSitePage {
      nodes {
        # querying id is required
        id
        component
        path
        componentChunkName
        jsonName
        internal {
          # querying internal.contentDigest is required
          contentDigest
          type
          owner
        }
      }
    }
  }
`;

const queries = [
  {
    query: myQuery,
    queryVariables: {}, // optional. Allows you to use graphql query variables in the query
    transformer: ({ data }) => data.pages.nodes, // optional
    indexName: 'index name to target', // overrides main index name, optional
    settings: {
      // optional, any index settings
      // Note: by supplying settings, you will overwrite all existing settings on the index
    },
    mergeSettings: false, // optional, defaults to false. See notes on mergeSettings below
  },
];

module.exports = {
  plugins: [
    {
      // This plugin must be placed last in your list of plugins to ensure that it can query all the GraphQL data
      resolve: `gatsby-plugin-algolia`,
      options: {
        appId: process.env.ALGOLIA_APP_ID,
        // Use Admin API key without GATSBY_ prefix, so that the key isn't exposed in the application
        // Tip: use Search API key with GATSBY_ prefix to access the service from within components
        apiKey: process.env.ALGOLIA_API_KEY,
        indexName: process.env.ALGOLIA_INDEX_NAME, // for all queries
        queries,
        chunkSize: 10000, // default: 1000
        settings: {
          // optional, any index settings
          // Note: by supplying settings, you will overwrite all existing settings on the index
        },
        mergeSettings: false, // optional, defaults to false. See notes on mergeSettings below
        concurrentQueries: false, // default: true
        dryRun: false, // default: false, only calculate which objects would be indexed, but do not push to Algolia
        continueOnFailure: false, // default: false, don't fail the build if Algolia indexing fails
        algoliasearchOptions: undefined, // default: { timeouts: { connect: 1, read: 30, write: 30 } }, pass any different options to the algoliasearch constructor
      },
    },
  ],
};

The index will be synchronised with the provided index name on Algolia on the build step in Gatsby. This is not done earlier to prevent you going over quota while developing.

Partial Updates

This plugin will update only the changed or deleted nodes on your Gatsby site.

We rely on Gatsby's default contentDigest field, so make sure it is queried.

Settings

You can set settings for each index individually (per query), or otherwise it will keep your existing settings.

Merge Settings

mergeSettings allows you to preserve settings changes made on the Algolia website. The default behavior (mergeSettings: false) will wipe out your index settings and replace them with settings from the config on each build.

When set to true, the config index settings will be merged with the existing index settings in Algolia (with the config index settings taking precendence).

NOTE: When using mergeSettings, any deleted settings from the config settings will continue to be persisted since they will still exist in Algolia. If you want to remove a setting, be sure to remove it from both the config and on Algolia's website.

Concurrent Queries

Sometimes, on limited platforms like Netlify, concurrent queries to the same index can lead to unexpected results or hanging builds. Setting concurrentQueries to false makes it such that queries are run sequentially rather than concurrently, which may solve some concurrent access issues. Be aware that this option may make indexing take longer than it would otherwise.

Transformer

The transformer field accepts a function and optionally you may provide an async function.

Adding a transformer parameter can be useful if the internal.contentDigest is more stable than your object. You can for example replace the Gatsby-provided internal.contentDigest with a hash of the object.

const crypto = require('crypto');

function transformer(data) {
  return data.map(item => {
    const hash = crypto
      .createHash('md5')
      .update(JSON.stringify(item))
      .digest('hex');

    return {
      ...item,
      internal: {
        ...item.internal,
        contentDigest: hash,
      },
    };
  });
}

Feedback

This is the very first version of our plugin and isn't yet officially supported. Please leave all your feedback in GitHub issues ๐Ÿ˜Š

FAQ

Why do my updates not show up?

This could be happening for a few reasons:

Development mode

You are using the gatsby-plugin-algolia plugin in development mode. The plugin will only push to Algolia when you run gatsby build. This is to prevent you from going over your quota while developing.

Static internal.contentDigest

Some Gatsby plugins don't create a new internal.contentDigest, even if the content has changed. To fix this, use a transformer to create a new internal.contentDigest based on the content of the node.

gatsby-plugin-algolia's People

Contributors

ahmetkuslular avatar akheron avatar alanaktion avatar andrewocc avatar avigoldman avatar bartdeslagmulder avatar bdenham avatar brettinternet avatar bruce773 avatar c4ndybar avatar caelinsutch avatar carolirod avatar colliercz avatar epbarger avatar froddd avatar haroenv avatar janosh avatar johndaskovsky avatar lekoarts avatar marcelovicentegc avatar marcspicer avatar mrtnvh avatar phil-linnell avatar prichey avatar samouss avatar seanmcn avatar sidharthachatterjee avatar stefanprobst avatar thecodingwizard avatar u12206050 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

gatsby-plugin-algolia's Issues

Version 0.6.0 - Method not allowed with this API key

Hi!

I've updated the plugin from 0.5.0 to 0.6.0 and it does not work anymore. Should I update some index settings? ๐Ÿค” Current index ACL: addObject, deleteObject, editSettings. It works with the 0.5.0 version.

e587abe

7:21:37 PM: Algolia: 1 queries to index
7:21:37 PM: error failed to index to Algolia
7:21:37 PM: 
7:21:37 PM:   AlgoliaSearchError: Method not allowed with this API key
7:21:37 PM:   
7:21:37 PM:   - AlgoliaSearchCore.js:377 success
7:21:37 PM:     [repo]/[gatsby-plugin-algolia]/[algoliasearch]/src/AlgoliaSearchCore.js:377:    32
7:21:37 PM:   
7:21:37 PM:   - runMicrotasks
7:21:37 PM:   
7:21:37 PM:   - task_queues.js:97 processTicksAndRejections
7:21:37 PM:     internal/process/task_queues.js:97:5
7:21:37 PM:   
7:21:37 PM:   - gatsby-node.js:32 async doQuery
7:21:37 PM:     [repo]/[gatsby-plugin-algolia]/gatsby-node.js:32:29
7:21:37 PM:   
7:21:37 PM:   - async Promise.all
7:21:37 PM:   
7:21:37 PM:   - gatsby-node.js:70 async Object.exports.onPostBuild
7:21:37 PM:     [repo]/[gatsby-plugin-algolia]/gatsby-node.js:70:5
7:21:37 PM:   
7:21:37 PM: 
7:21:38 PM: not finished onPostBuild - 1.093s

Reduce build operations

A continuation of #5 and #1

What happens currently is:

  1. Gatsby builds
  2. the extraction logic runs and turns this into objects
  3. these objects are indexed into a separate index
  4. this separate index replaces the original index

We should use atomic-algolia or algolia-indexing preferably to make this into a flow where no extra index needs to be created, but rather we can index only the changes since last push, rather than all.

cc @pixelastic (when you're back from parental leave of course)

Security Implication for index name

This is something I've been thinking about, might be completely off-base.

Given that I have one algolia app (Starter Plan) and I host 2 gatsby websites with integrated algolia search:

  • One website is internal facing (ie: it's hidden behind a firewall and only internal computers can access the website)
  • The other website is public facing for a product we offer.

I was thinking whether it would be possible to access the internal documents of the internal facing website through the public facing website by just brute force guessing the index name?

To add some context to this, with the algolia plugin I specify the following:

appId: process.env.GATSBY_ALGOLIA_APP_ID,
apiKey: process.env.ALGOLIA_ADMIN_KEY,
indexName: process.env.GATSBY_INDEX,

Since both my internal and external company app are under the same APP ID (I just created different indices for each website), and the way I specify the proper index to use in my gatsby app is through the react-instantsearch-dom module:

<InstantSearch indexName={process.env.GATSBY_INDEX}>

Would it be possible for a user to pour through the js code of the front-end facing search component and just change the index name and instantly be able to access the other index? and even if you didn't know the index, you could keep guessing names until one actually returned a query?

I don't know enough about how react/gatsby compiles its final build to see if any of those values are obfuscated, but was just curious. Hopefully this makes sense, can provide more detail if needed.

Multiple indices

Thanks for this plugin, it's quite helpful. For multi-language sites it's mentioned that one index might be needed per language, however the plugin seems to accept a single indexName. Could it be enhanced to support that scenario?

Algolia plugin breaks gastby develop, but works during gatsby build

When I run gatsby build, I see no issues with algolia search, however on gatsby develop I keep getting different errors. First, the css normally renders incorrectly, though when I remove the line for <Input onFocus={() => setFocus(true)} {...{ collapse, focus }} /> it renders fine. On top of that, any click I make on the screen brings up this error:

HTMLDocument.detectClickOutside
src/components/Search/index.js:26
  23 | 
  24 | const useClickOutside = (ref, handler, events) => {
  25 |   if (!events) events = [`mousedown`, `touchstart`]
> 26 |   const detectClickOutside = event =>
  27 |     !ref.current.contains(event.target) && handler()
  28 |   useEffect(() => {
  29 |     for (const event of events)```

Allow configuring index settings from the queries

I need to specify certain attributes to be snippetable. Unlike highlighting, snippeting has to be enabled proactively on a per-attribute basis.

Of course, I can manually specify snippetable attributes in Algolia's web interface. However, it appears as though gatsby-plugin-algolia overwrites these settings on every build. Having to reset those regularly would be quite the hassle.

The docs state that this can also be done programmatically using something like

index.setSettings({
  attributesToSnippet: [
    'content:80',
    'description'
  ]
});

Does this carry over to gatsby-plugin-algolia and if so could this be documented somewhere?

FYI, I tried the following without success:

const queries = [
  {
    query: pageQuery,
    transformer: ({ data }) =>
      data.pages.edges.map(({ node }) => ({
        title: node.title.title,
        slug: node.slug,
        ...node.body.data,
        objectID: node.id,
      })),
    indexName: `Pages`,
+    setSettings: {
+      attributesToSnippet: [`excerpt:15`],
+    },
  },
]

[Feedback] `gatsby develop` doesn't trigger index upload

Today I spent a bit of time reading the code before I realized that gatsby develop didn't trigger exports.onPostBuild, so no data from my queries were uploaded to Algolia. It might be good to advertise it a bit; something like:

gatsby-plugin-algolia uses the "onPostBuild" entry point which is not triggered when using gatsby develop. Make sure you use gatsby build to upload data to your indices.

For reference:

gatsbyjs/gatsby#2698 (comment)

This is a problem if you are using the gatsby-plugin-algolia search plugin. The cache is only (re)generated when gatsby build is run. In the case of Algolia, the exports.createPages entry point is a viable alternative to exports.onPostBuild if you want to edit your node_modules/gatsby-plugin-algolia/gatsby-node.js file.

Only take chunksize?

I'm using the Algolia plugin in a static blog site from the gatsby-starter-hero-blog and I'm running into an issue where my blog post is larger than the document size of 10k. Is it possible to set up the plugin so it merely take the 'first' chunksize amount without barfing an error over a larger-sized file? I'd prefer to simply get a warning that it's 'only' taking the first 10k or something, as opposed to trying to whittle down the size of the blog post.

Thanks!

Facets and Settings Irregularly Missing

There is an irregular bug with the plugin which is causing the facets and other settings which are set up in the plugin config to be sporadically missing from the updated index.

I noticed that when this problem occurs, the settings and facets exist on the tmp version of the index but not on the primary version.

This issue has happened several times during our build over the last 3 months and is difficult to replicate. In testing locally today, 2 of the 3 times resulted in one of our indexes missing its settings, and the third time the settings were present.

The irregularity of the bug makes me suspect a race condition at play but I've not been able to identify the problem.

Support pathPrefix/prefixPaths

Currently, the user has to append the Gatsby pathPrefix https://www.gatsbyjs.org/docs/path-prefix/ manually in their transformer in order for Algolia to have the correct URL (assuming the slug is rendered directly in the search results).

It would be helpful if there were at least a discussion of the best practices for search when the site is hosted at a pathPrefix.

This is what I ended up doing:

url = require('url')

const postQuery = `{
  posts: allMarkdownRemark(
    edges {
      node {
        objectID: id
        fields {
          slug
        }
        frontmatter {
          title
          date(formatString: "MMM D, YYYY")
          description
          tags
        }
        excerpt(pruneLength: 1000)
      }
    }
  }
  site {
    pathPrefix
    siteMetadata {
      siteUrl
    }
  }
}`

const flatten = (arr, baseUrl) =>
  arr.map(({ node: { frontmatter, fields, ...rest } }) => ({
    ...fields,
    ...frontmatter,
    ...rest,
    slug: url.resolve(baseUrl, fields.slug)
  }))
const settings = { attributesToSnippet: [`excerpt:20`] }

const queries = [
  {
    query: postQuery,
    transformer: ({ data }) => flatten(data.posts.edges,
      url.resolve(data.site.siteMetadata.siteUrl, data.site.pathPrefix)),
    indexName: `Posts`,
    settings,
  },
]

module.exports = queries

Hits Per Page Not Working as Expected?

I am trying to limit the amount of hits or search results that algolia returns. I thought I could do the following inside the gatsby config.js file to limit the results to 8.

{
  resolve: 'gatsby-plugin-algolia',
  options: {
    appId: process.env.GATSBY_ALGOLIA_APP_ID,
    apiKey: process.env.GATSBY_ALGOLIA_ADMIN_KEY,
    indexName: process.env.GATSBY_ALGOLIA_INDEX_NAME,
    queries,
    hitsPerPage: 8,
    chunkSize: 10000,
  },

However, reducing that number does not seem to be doing anything.
Am I missing something?

Attributes for faceting being removed?

  • From the dashboard I assign some attributes which will be used for refinement over on Display > Faceting
  • When the site gets rebuilt (on content change or whatever) the faceting options are reset

Is this because I am not setting these in gatsby-config.js? If so how do I do that?

Thanks!

Update without rewriting index

Is it is possible to run the plugin so that it only updates the index selected rather than completely erasing and reinserting the objects?

The issue I'm having is that I'm wanting to use one index to search for both items on a Gatsby site and an offsite event platform. I'm pulling in the event data through a node process happening on a separate server, but when I go back and perform gatsby build on my site, the event data is removed.

If there's no way around this, I can simply pull all Algolia updates into the node process, but I'd like to keep it integrated within Gatsby if possible.

Async transformer function brokes the code

In queries object we have this transformer function, which, as you know, proves to be very helpful when it comes to preprocessing data that we want to send to Algolia side:

const queries = [
  {
    query: myQuery,
    transformer: ({ data }) => data.allSitePage.edges.map(({ node }) => node), // optional
    indexName: 'index name to target', // overrides main index name, optional
    settings: {
      // optional, any index settings
    },
    matchFields: ['slug', 'modified'], // Array<String> overrides main match fields, optional
  },
];

In readme we can read, that transformer could be an async function, but in reality passing an async function breaks the code completely.

Consider this situation in which I had intention to additionally process some of the fields using 3rd party libs (though I abstracted everything to ease understanding):

const someAsyncFn = async (rawBody) => useThirdParty(rawBody)

// adapt query results for algolia
const flatten = (arr) =>
  Promise.all(
    arr.map(async ({ children }) => {
      const postData = { ...children[0] };
       const text = await someAsyncFn(postData.rawBody);
      return {
        ...postData.frontmatter,
        content: text,
        objectID: postData.objectID,
      };
    }),
  );

const postsQuery = `{
 // query body
}`;

const queries = [
  {
    query: postsQuery,
    transformer: async ({ data }) => {
      const res = await flatten(data.posts.nodes);
      return res;
    },
    indexName: algoliaIndexName,
  },
];
module.exports = queries;

This lead to this error:
image

If we proceed to the source code, here is exactly the place where it breaks:

 const objects = await transformer(result).map((object) => ({
      objectID: object.objectID || object.id,
      ...object,
    }));

I think this is a bug and a .map method should not be chained right away since there could be no mappable object at all, just like in the case above, where after evaluation we have something like:

const objects = await Promise<any>.map(...)

Which is the main cause of crashing.

If you could separate transform and map steps into 2, everything would work as expected:

 const objectsData = await transformer(result);

const objects = objectsData.map((object) => ({
      objectID: object.objectID || object.id,
      ...object,
    }));

Here is the fiddle with everything I've just tried to explain.

Please, let me know if I had misunderstood something and a bug actually hides in my code or if I should provide more information related to this issue.

Algolia show results example

Good evening,
I'm trying to use gatsby-algolia
I downloaded this project, inserting my parameters of algolia and launching gatsby build, can successfully connect to Algolia.

success index to Algolia - 2.005 s
info Done building in 19.89 sec

I would like an example with gatsby to see the results in the frontend, instantsearch or autocomplete.
At the moment if I start gatsby development, the pages start correctly, but I do not see the algolia data or a search.
Is there a working example from which to start?
Thank you

Partial Updates ACLs

Do I need any additional ACL for partial updates? According to the documentation, addObject should be enough:

Required API Key: any key with the addObject ACL

My index ACLs:

image

Error:

9:20:11 AM: Algolia: query 0: executing query
9:20:11 AM: Algolia: query 0: graphql resulted in 67 records
9:20:11 AM: Algolia: query 0: starting Partial updates
9:20:12 AM: 
9:20:12 AM: error failed to index to Algolia
9:20:12 AM: 
9:20:12 AM: 
9:20:12 AM: 
9:20:12 AM:   AlgoliaSearchError: Method not allowed with this API key
9:20:12 AM:   
9:20:12 AM:   - AlgoliaSearchCore.js:377 success
9:20:12 AM:     [repo]/[gatsby-plugin-algolia]/[algoliasearch]/src/AlgoliaSearchCore.js:377:    32
9:20:12 AM:   
9:20:12 AM:   - runMicrotasks
9:20:12 AM:   
9:20:12 AM:   - task_queues.js:97 processTicksAndRejections
9:20:12 AM:     internal/process/task_queues.js:97:5
9:20:12 AM:   
9:20:12 AM: 
9:20:12 AM: 
9:20:12 AM: not finished onPostBuild - 1.363s

gatsby-config.js, query data

Implement the cache

Right now every build would add as many objects as there are results, we should cache them using the Gatsby cache

Indexing won't come to an end

Somehow when building it gets stuck when indexing to algolia. Left it running for about an hour and it just kept spinning. I was just testing it with 2 records so can't be the data I guess..

I am using Gatsby 2.4.1, this is my gatsby-config:

const myQuery = `{
  allDataJson {
    edges {
      node {
        slug
        name
      }
    }
  }
}`;

const queries = [
  {
    query: myQuery,
    transformer: ({ data }) => data.allDataJson.edges.map(({ node }) => node),
    indexName: 'testshop',
  },
];

module.exports = {
  siteMetadata: {
    title: 'Gatsby Default Starter',
  },
  plugins: [
    'gatsby-plugin-react-helmet',
    {
      resolve: `gatsby-plugin-algolia`,
      options: {
        appId: 'XXX',
        apiKey: 'XXX',
        indexName: "test",
        queries,
        chunkSize: 10000,
      },
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `data`,
        path: `${__dirname}/src/data`,
      },
    },
    `gatsby-transformer-json`,
    `gatsby-transformer-sharp`,
    `gatsby-plugin-sharp`,
  ],
}

I guess I am just missing something here but just can't spot why this wouldn't work. Any ideas or more infos you need? :)

Add an ability to perform complex async work

my use case needs to fetch a few datasets prior to creating the index, which this plugin does not allow me to do.

I modified the plugin to accept an exported function that the product developer can fetch data and return the index to the plugin to upload to algolia.

https://github.com/kdichev/gatsby-plugin-algolia-index

// gatsby-config.js
module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-algolia`,
      options: {
        appId: process.env.ALGOLIA_APP_ID,
        apiKey: process.env.ALGOLIA_API_KEY,
        chunkSize: 10000, // default: 1000
        path: 'path/to/custom/indexing-config-name' // optional, defaults to [algolia-index-config.js] in root
      },
    },
  ],
};
// algolia-index-config.js
module.exports = async (/* graphql */) => {
  // can do async work by using graphql client
  // can do index transforms
  return [{ indexName: 'index1', indexData: [{ title: 'index1' }] }, ...]
}

Let me know if you think this can be added to this plugin.
@Haroenv

npm ERR! code ELIFECYCLE, npm ERR! errno 1 ...

I've had algolia running fine with a different API, now I'm trying to get it to work with Sanity.io and have what i think is a very simple query to get going with. This query works fine in graphiQL:

Here's the code, and to debug i put all the API credentials in the code and took it out of .env.production incase there was an issue there. I'm building locally:

// gatsby-config.js
const myQuery = `{
  allSanityPeople {
    edges {
      node {
        displayName
      }
    }
  }
}`

const queries = [
  {
    query: myQuery,
    transformer: ({ data }) =>
      data.allSanityPeople.edges.map(({ node }) => node), // optional
    indexName: "MyIndexName", // overrides main index name, optional
    settings: {
      // optional, any index settings
    },
    matchFields: ["slug", "modified"], // Array<String> overrides main match fields, optional
  },
]

In my config settings:

{
      resolve: `gatsby-plugin-algolia`,
      options: {
        appId: "myappid",
        // Careful, no not prefix this with GATSBY_, since that way users can change
        // the data in the index.
        apiKey: "myAdminAPIKey",
        // indexName: process.env.ALGOLIA_INDEX_NAME, // for all queries
        queries,
        chunkSize: 10000, // default: 1000
        settings: {
          // optional, any index settings
        },
        enablePartialUpdates: true, // default: false
        matchFields: ["slug", "modified"], // Array<String> default: ['modified']
      },
    },
Algolia: 1 queries to index
Algolia: query 0: executing query
โ ‹ onPostBuild
not finished onPostBuild - 0.159s
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: `gatsby build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/alistairmcclymont/.npm/_logs/2020-05-05T19_08_29_715Z-debug.log
The terminal process terminated with exit code: 1

From the log mentioned above:

0 info it worked if it ends with ok
1 verbose cli [
1 verbose cli   '/Users/alistairmcclymont/.nvm/versions/node/v13.13.0/bin/node',
1 verbose cli   '/Users/alistairmcclymont/.nvm/versions/node/v13.13.0/bin/npm',
1 verbose cli   'run',
1 verbose cli   'build'
1 verbose cli ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
5 info lifecycle [email protected]~prebuild: [email protected]
6 info lifecycle [email protected]~build: [email protected]
7 verbose lifecycle [email protected]~build: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]~build: PATH: /Users/alistairmcclymont/.nvm/versions/node/v13.13.0/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/Users/alistairmcclymont/Sites/RCA/rca2020public/node_modules/.bin:/Users/alistairmcclymont/.nvm/versions/node/v13.13.0/bin:/Applications/MAMP/bin/php/php7.1.8/bin:/Applications/MAMP/Library/bin/:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/local/bin
9 verbose lifecycle [email protected]~build: CWD: /Users/alistairmcclymont/Sites/RCA/rca2020public
10 silly lifecycle [email protected]~build: Args: [ '-c', 'gatsby build' ]
11 silly lifecycle [email protected]~build: Returned: code: 1  signal: null
12 info lifecycle [email protected]~build: Failed to exec build script
13 verbose stack Error: [email protected] build: `gatsby build`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/Users/alistairmcclymont/.nvm/versions/node/v13.13.0/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:332:16)
13 verbose stack     at EventEmitter.emit (events.js:315:20)
13 verbose stack     at ChildProcess.<anonymous> (/Users/alistairmcclymont/.nvm/versions/node/v13.13.0/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:315:20)
13 verbose stack     at maybeClose (internal/child_process.js:1026:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
14 verbose pkgid [email protected]
15 verbose cwd /Users/alistairmcclymont/Sites/RCA/rca2020public
16 verbose Darwin 19.4.0
17 verbose argv "/Users/alistairmcclymont/.nvm/versions/node/v13.13.0/bin/node" "/Users/alistairmcclymont/.nvm/versions/node/v13.13.0/bin/npm" "run" "build"
18 verbose node v13.13.0
19 verbose npm  v6.14.4
20 error code ELIFECYCLE
21 error errno 1
22 error [email protected] build: `gatsby build`
22 error Exit status 1
23 error Failed at the [email protected] build script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

I want to try and debug this, but can't understand how - if anyone can help me there too that would be great.

Partial updates only working partially

I have an index I'd like to update only when the underlying files change. Right now, I'm using a modified field that is the last time the file was modified (gitLogLatestDate). This decreases the amount of updates that get done but does not eliminate them(currently: [insert/update: 1503, total: 8573]).

We're planning on adding more files, but if we keep having these updates, we'll run over our limits. Does anyone have any insights into how we could get the number to 0 when the files haven't changed? Thanks.

Simple Settings Question: Sorting by Date Descending!

Hiya! I just have a really simple question on creating replicas and sorting by date - desc. I am following this documentation

In my gatstby-config.js Ive added:

const queries = [
  {
    query: siteQuery,
    transformer: ({ data }) => data.allMarkdownRemark.nodes,
    settings: {  
      replicas: [  // Is this the right place to add replica settings?
        'articles_date_desc'
      ]
    }
  }
];

Is settings the right place to add replicas? Follow up, where would I add

replicaIndex.setSettings({
  ranking: [
    "desc(date_tag)",
    "typo",
    "geo",
    "words",
    "filters",
    "proximity",
    "attribute",
    "exact",
    "custom"
  ]
});

Thanks in advance!

Add option to create index it it does not exist

Currently you need to create the index first in the Algolia dashboard for this plugin to work.
I would be nice if this plugin could create the index for me it the specified index does not exists yet.

Not indexing / errors only on build

I'm not having any luck following tutorial and getting to index properly. There's a post to search based on the dashboard but nothing is indexed? No errors in console on gatsby develop.

Went ahead and tried gatsby build from local to see if that would work and it did throw an error that it's not getting the appID:

"gatsby-plugin-algolia" threw an error while running the onPostBuild lifecycle:

Please provide an application ID. Usage: algoliasearch(applicationID, apiKey, opts)

  16 |   const activity = report.activityTimer(`index to Algolia`);
  17 |   activity.start();
> 18 |   const client = algoliasearch(appId, apiKey);
     |                  ^
  19 |
  20 |   setStatus(activity, `${queries.length} queries to index`);
  21 |

File: node_modules/gatsby-plugin-algolia/gatsby-node.js:18:18

  AlgoliaSearchError: Please provide an application ID. Usage: algoliasearch(applicationID, apiKey, opts)

Considered perhaps the options had been renamed and attempted to update to applicationID instead of appID but no joy. I've tried sifting through node_modules but I am a noob still and this stuff feels rather over my head. Any and all assistance would be most appreciated.

How to work with `gatsby-image` and Algolia on build?

Introduction

First of all, thank you for making such a great plugin for implementing Algolia. It has been a real blessing for our development team. I am not really sure where to address this issue, but I've found out this issue occurs while using gatsby-plugin-algolia. Hence writing the issue here. Please let me know if I should bring this up somewhere else and this is unrelated to this plugin.

Problem

We have implemented search with Algolia which returns a title and an image of an art object. To optimize the loading times, we've used gatsby-image to render the GraphQL query returned by Algolia with featuredImage.childImageSharp.fluid. This works perfectly on gatsby develop.

However, when deploying or gatsby build && serve the images don't load and give a 404.

algolia-load-bug_example

Hypothesis

  • Synchronising our index with Algolia happens on gatsby-build
  • Generated URL's of thumbnails bygatsby-image happens on gatsby build too

Our guess of the problem is that the URL's Algolia stores are not in sync with the URL's of the images of the built website.

Expected result

Algolia returns the same URL as requested in the rendered image

Actual Result

The URL Algolia returns is not the same as requested by the image

# Example requested URL
--- /static/1531268446cf37a32509968330a34e28/a9afa/98-theeservies.jpg

# Example stored URL at Algolia
+++ /static/1531268446cf37a32509968330a34e28/a07a5/98-theeservies.jpg

Steps to reproduce

  1. Create a new Algolia Index and implement this plugin with your Gatsby website
  2. Create a <customHits /> component and render an <Img fluid={yourSrc.fluid}/>
  3. Query the data in gatsby-config.js something like this example
const yourQuery = `
{
  allMarkdownRemark(filter: {frontmatter: {templateKey: {eq: "your-item"}}}) {
    nodes {
      id
      fields {
        slug
      }
      frontmatter {
        title
        templateKey
        portrayedPersons
        featuredImage {
          childImageSharp {
            fluid(maxWidth: 600, quality: 80, toFormat: JPG) {
              aspectRatio
              src
              srcSet
              sizes
              originalImg
              originalName
            }
          }
        }
      }
    }
  }
}
`

const finalQuery = [
  {
    query: yourQuery,
    transformer: ({ data }) => data.allMarkdownRemark.nodes
  }
]

module.exports = {
  plugins: [
    {
      resolve: "gatsby-plugin-algolia",
      options: {
        appId: process.env.GATSBY_ALGOLIA_APP_ID,
        apiKey: process.env.ALGOLIA_ADMIN_KEY,
        indexName: process.env.ALGOLIA_INDEX_NAME,
        queries: finalQuery
      }
    }
    /* your other plugins such needed for gatsby-images, read more about that here https://www.gatsbyjs.org/docs/gatsby-image/#images-that-stretch-across-a-fluid-container */
  ]
}
  1. Run gatsby develop and everything should be working
  2. Running gatsby build && gatsby serve should give a 404 on the image

Question

How can we make sure the URL's are the same to get gatsby-image and Algolia working on build? Is it possible to trigger synchronisation manually?

Environment

  System:
    OS: macOS 10.15.2
    CPU: (4) x64 Intel(R) Core(TM) i5-7287U CPU @ 3.30GHz
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 13.11.0 - /usr/local/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.4 - /usr/local/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 80.0.3987.149
    Firefox: 66.0.3
    Safari: 13.0.4
  npmPackages:
    gatsby: ^2.20.9 => 2.20.9 
    gatsby-cli: ^2.11.3 => 2.11.3 
    gatsby-image: ^2.3.1 => 2.3.1 
    gatsby-plugin-algolia: ^0.5.0 => 0.5.0 
    gatsby-plugin-netlify: ^2.2.1 => 2.2.1 
    gatsby-plugin-netlify-cms: ^4.2.2 => 4.2.2 
    gatsby-plugin-prefetch-google-fonts: ^1.4.3 => 1.4.3 
    gatsby-plugin-react-helmet: ^3.2.1 => 3.2.1 
    gatsby-plugin-sass: ^2.2.1 => 2.2.1 
    gatsby-plugin-sharp: ^2.5.3 => 2.5.3 
    gatsby-plugin-typography: ^2.4.1 => 2.4.1 
    gatsby-remark-copy-linked-files: ^2.2.1 => 2.2.1 
    gatsby-remark-images: ^3.2.1 => 3.2.1 
    gatsby-remark-relative-images: ^0.3.0 => 0.3.0 
    gatsby-source-filesystem: ^2.2.2 => 2.2.2 
    gatsby-transformer-remark: ^2.7.1 => 2.7.1 
    gatsby-transformer-sharp: ^2.4.3 => 2.4.3 
  npmGlobalPackages:
    gatsby-cli: 2.10.4

How to create/sync replicas?

There currently doesn't appear to be an API here to create replicas following an index creation/sync. Our transform is somewhat complicated and does some asynchronous actions outside of the Gatsby DataLayer, so we'd like to avoid repeating the queries/transforms with each replica. Here are two possible approaches to consider.

Assume we have an index and two replicas:

const indexName = ALGOLIA_INDEX_PREFIX + ALGOLIA_INDEX_NAME
const replicas = {
  DATE_DESC: `${indexName}_date_desc`,
  DATE_ASC: `${indexName}_date_asc`,
}

Option 1

We allow query items to not contain the "query" attribute, and assume it's a replica if the indexName is contained in some query that does have a "query" attribute. Notice in this example the replicas only contain an index name and settings.

// gatsby-config.js
const commonSettings = { // since forwardToReplicas isn't available
  searchableAttributes: ["description", "title", "content"],
  attributesToSnippet: ["content"],
  attributesForFaceting: ["category", "_tags"],
  distinct: true,
  attributeForDistinct: "url",
  ranking: [
    "typo",
    "geo",
    "words",
    "filters",
    "proximity",
    "attribute",
    "exact",
    "desc(date_timestamp)"
  ]
};

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-algolia`,
      options: {
        appId: env.ALGOLIA_APP_ID,
        apiKey: env.ALGOLIA_API_KEY,
        queries: [
          {
            query: postQuery,
            transformer: [Function],
            indexName,
            settings: {
              replicas: Object.values(replicas),
              ...commonSettings
            }
          },
          {
            indexName: replicas.DATE_DESC,
            settings: {
              ...commonSettings,
              ranking: [
                "desc(date_timestamp)",
                "typo",
                "geo",
                "words",
                "filters",
                "proximity",
                "attribute",
                "exact"
              ]
            }
          },
          {
            indexName: replicas.DATE_ASC,
            settings: {
              ...commonSettings,
              ranking: [
                "asc(date_timestamp)",
                "typo",
                "geo",
                "words",
                "filters",
                "proximity",
                "attribute",
                "exact"
              ]
            }
          }
        ]
      }
    }
  ]
};

Option 2

Or we add a completion hook to the library where the user may sync the replicas with algoliasearch JS library.

// gatsby-config.js
module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-algolia`,
      options: {
        appId: env.ALGOLIA_APP_ID,
        apiKey: env.ALGOLIA_API_KEY,
        queries: [...],
        onComplete: syncSettings
      }
    }
  ]
};

What do you think about these approaches? What's your take on incorporating replica creation/sync in this library?

not finished onPostBuild

Hi,
I created a simple gatsby project and integrated the algolia appropriately.
But when I say yarn build. I get the following error.

Algolia: 1 queries to index
Algolia: query 0: executing query
โ ด onPostBuild
not finished onPostBuild - 0.493s

i can't make sense. What exactly is the problem?

my gatsyby.config


require("dotenv").config({
  path: `.env.${process.env.NODE_ENV}`,
})

const blogQuery = `
  {
    allMarkdownRemark {
      nodes {
        frontmatter {
          title
          date
          description
        }
        fields {
          slug
        }
        html
      }
    }
  }
`

const queries = [
  {
    query: blogQuery,
    transformer: ({ data }) => {
      return data.allMarkdownRemark.nodes;
    },
  },
]


module.exports = {
  siteMetadata: {
    title: `Gatsby Default Starter`,
    description: `Kick off your next, great Gatsby project with this default starter. This barebones starter ships with the main Gatsby configuration files you might need.`,
    author: `@gatsbyjs`,
  },
  plugins: [
    {
      resolve: "gatsby-plugin-algolia",
      options: {
        appId: process.env.ALGOLIA_APP_ID,
        apiKey: process.env.ALGOLIA_API_KEY,
        indexName: process.env.ALGOLIA_INDEX_NAME,
        queries,
        chunkSize: 1000,
      },
    },
    `gatsby-plugin-react-helmet`,
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `${__dirname}/content/blog`,
        name: `blog`,
      },
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `${__dirname}/content/assets`,
        name: `assets`,
      },
    },
    `gatsby-transformer-remark`,
    `gatsby-transformer-sharp`,
    `gatsby-plugin-sharp`,
    {
      resolve: `gatsby-plugin-manifest`,
      options: {
        name: `gatsby-starter-default`,
        short_name: `starter`,
        start_url: `/`,
        background_color: `#663399`,
        theme_color: `#663399`,
        display: `minimal-ui`,
        icon: `src/images/gatsby-icon.png`, // This path is relative to the root of the site.
      },
    },
  ],
}

Possible to export to Index outside of gatsby-config.js?

I'm programmatically getting data to build pages from another plugin and would like to build the Algolia index after getting data from the other plugin. Is it possible to push to the index after my pages are built?

My data isn't available at the time the algolia plugin is resolved.

Seems like the query is required in the resolving of the gatsby-plugin-algolia.

Documentation Update

It should be mentioned in the docs that you have to insert this plugin last in the gastby config, otherwise it gives a non-descriptive error and fails. Confused me for quite a while because nowhere mentions this.

Manually trigger index

Is there a way of manually triggering the index operation during develop, without performing the whole build-step?

Can not push data pulled from airtable into gatsby to algolia

I can fetch data from airtable using gatsby graphql playground while gatsby running on development mode, but I am unable to push data pulled from airtable into gatsby to algolia when I run npm run build.

gatsby-config.js

const queries = require("./src/utils/algolia")

module.exports = {
  plugins: [
    `gatsby-plugin-react-helmet`,
    `gatsby-plugin-styled-components`,
    {
      resolve: `gatsby-source-airtable`,
      options: {
        apiKey: `${process.env.AIRTABLE_API_KEY}`,
        tables: [
          {
            baseId: `${process.env.AIRTABLE_BASE_ID}`,
            tableName: `Products`
          },
          {
            baseId: `${process.env.AIRTABLE_BASE_ID}`,
            tableName: `Categories`
          }
        ]
      }
    },
    {
      resolve: `gatsby-plugin-algolia`,
      options: {
        appId: `${process.env.GATSBY_ALGOLIA_APP_ID}`,
        apiKey: `${process.env.ALGOLIA_ADMIN_KEY}`,
        indexName: `${process.env.ALGOLIA_INDEX_NAME}`,
        queries,
        chunkSize: 10000,
      },
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `images`,
        path: `${__dirname}/src/images`,
      },
    },
    `gatsby-transformer-sharp`,
    `gatsby-plugin-sharp`,
    {
      resolve: `gatsby-plugin-manifest`,
      options: {
        name: `gatsby-starter-default`,
        short_name: `starter`,
        start_url: `/`,
        background_color: `#663399`,
        theme_color: `#663399`,
        display: `minimal-ui`,
        icon: `src/images/gatsby-icon.png`,
      },
    },
  ],
}

algolia.js

const ProductsQuery = `{
  allAirtable(filter: {table: {eq: "Products"}}) {
    nodes {
      data {
        productId
        name
        slug
        sku
        description
        price
        categories
      }
    }
  }
}`

const flatten = arr =>
  arr.map(({ data }) => ({
    ...data
  }))

const queries = [
  {
    query: ProductsQuery,
    transformer: ({ data }) => flatten(data.allAirtable.nodes),
  }
]

module.exports = queries

Error message
ERROR #11321 PLUGIN
"gatsby-source-airtable" threw an error while running the sourceNodes lifecycle:
Could not find what you are looking for(NOT_FOUND)[Http code 404]
โ ง source and transform nodes
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: gatsby build
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/kayali/.npm/_logs/2019-10-09T17_46_13_630Z-debug.log

/Users/kayali/.npm/_logs/2019-10-09T17_46_13_630Z-debug.log

0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'run', 'build' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
5 info lifecycle [email protected]~prebuild: [email protected]
6 info lifecycle [email protected]~build: [email protected]
7 verbose lifecycle [email protected]~build: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]~build: PATH: /usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/Users/shopgo/Desktop/kayali/snipcart/myjam-gatsby/node_modules/.bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/mongodb/bin:/usr/local/mongodb/bin:/usr/local/mongodb/bin
9 verbose lifecycle [email protected]~build: CWD: /Users/shopgo/Desktop/kayali/snipcart/myjam-gatsby
10 silly lifecycle [email protected]~build: Args: [ '-c', 'gatsby build' ]
11 silly lifecycle [email protected]~build: Returned: code: 1  signal: null
12 info lifecycle [email protected]~build: Failed to exec build script
13 verbose stack Error: [email protected] build: `gatsby build`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:301:16)
13 verbose stack     at EventEmitter.emit (events.js:198:13)
13 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:198:13)
13 verbose stack     at maybeClose (internal/child_process.js:982:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid [email protected]
15 verbose cwd /Users/shopgo/Desktop/kayali/snipcart/myjam-gatsby
16 verbose Darwin 18.7.0
17 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "build"
18 verbose node v10.16.2
19 verbose npm  v6.9.0
20 error code ELIFECYCLE
21 error errno 1
22 error [email protected] build: `gatsby build`
22 error Exit status 1
23 error Failed at the [email protected] build script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

Time frame to exit beta?

A Gatsby plugin for Algolia would be awesome and much appreciated! Is there already a timeframe on when to exit beta and add documentation?

Freezing builds on netlify

Since adding gatsby-plugin-algolia to some of my projects, they have been suffering from freezing builds on netlify. The build logs below show that the failing step is

8:20:30 PM: Algolia: query 1: moving copied index to main index

This happens on every 4th or 5th deploy, I'd say. Interestingly, so far this has never happened when building locally. So it's very likely related to netlify in some way.

@Haroenv Can you tell from the logs what exactly could be causing this issue and how we might fix it?

8:20:23 PM: Algolia: 3 queries to index
8:20:24 PM: Algolia: query 2: copying existing index
8:20:24 PM: Algolia: query 0: copying existing index
8:20:24 PM: Algolia: query 1: copying existing index
8:20:26 PM: Algolia: query 2: executing query
8:20:26 PM: Algolia: query 2: splitting in 1 jobs
8:20:26 PM: Algolia: query 0: executing query
8:20:26 PM: Algolia: query 0: splitting in 1 jobs
8:20:28 PM: Algolia: query 1: executing query
8:20:28 PM: Algolia: query 1: splitting in 1 jobs
8:20:29 PM: Algolia: query 0: moving copied index to main index
8:20:30 PM: Algolia: query 1: moving copied index to main index
8:49:28 PM: error UNHANDLED EXCEPTION
8:49:29 PM: 
8:49:29 PM:   TypeError: Cannot read property 'id' of undefined
8:49:29 PM:   
8:49:29 PM: Build exceeded maximum allowed runtime
8:49:29 PM:   - nodes.js:21 module.exports
8:49:29 PM:     [repo]/[gatsby]/dist/redux/reducers/nodes.js:21:37
8:49:29 PM:   
8:49:29 PM:   - redux.js:451 combination
8:49:29 PM:     [repo]/[redux]/lib/redux.js:451:29
8:49:29 PM:   
8:49:29 PM:   - redux.js:211 dispatch
8:49:29 PM:     [repo]/[redux]/lib/redux.js:211:22
8:49:29 PM:   
8:49:29 PM:   - index.js:79 action
8:49:29 PM:     [repo]/[gatsby]/dist/redux/index.js:79:91
8:49:29 PM:   
8:49:29 PM:   - redux.js:468 Object.deleteNode
8:49:29 PM:     [repo]/[redux]/lib/redux.js:468:12
8:49:29 PM:   
8:49:29 PM:   - gatsby-node.js:150 emitter.on.action
8:49:29 PM:     [repo]/[gatsby]/dist/internal-plugins/internal-data-bridge/gatsby-node.js:15    0:23
8:49:29 PM:   
8:49:29 PM:   - mitt.js:1 
8:49:29 PM:     [repo]/[mitt]/dist/mitt.js:1:268
8:49:29 PM:   
8:49:29 PM:   - Array.map
8:49:29 PM:   
8:49:29 PM:   - mitt.js:1 Object.emit
8:49:29 PM:     [repo]/[mitt]/dist/mitt.js:1:252
8:49:29 PM:   
8:49:29 PM:   - index.js:102 store.subscribe
8:49:29 PM:     [repo]/[gatsby]/dist/redux/index.js:102:11
8:49:29 PM:   
8:49:29 PM:   - redux.js:220 dispatch
8:49:29 PM:     [repo]/[redux]/lib/redux.js:220:7
8:49:29 PM:   
8:49:29 PM:   - index.js:79 action
8:49:29 PM:     [repo]/[gatsby]/dist/redux/index.js:79:91
8:49:29 PM:   
8:49:29 PM:   - redux.js:468 
8:49:29 PM:     [repo]/[redux]/lib/redux.js:468:12
8:49:29 PM:   
8:49:29 PM:   - api-runner-node.js:48 doubleBoundActionCreators.(anonymous function).args
8:49:29 PM:     [repo]/[gatsby]/dist/utils/api-runner-node.js:48:13
8:49:29 PM:   
8:49:29 PM:   - gatsby-node.js:86 
8:49:29 PM:     [repo]/[gatsby-plugin-page-creator]/gatsby-node.js:86:19
8:49:29 PM:   
8:49:29 PM:   - Map.forEach
8:49:29 PM:   
8:49:29 PM: 
8:49:29 PM: Execution timed out after 15m0s
8:49:29 PM: Error running command: Command did not finish within the time limit
8:49:29 PM: Failing build: Failed to build site

Problems creating replica indices

I ran into a number of problems while attempting to create replica indices with this plugin. In the end, I decided to remove the plugin and write my own script.

I was passing queries generated by mapping over a list of settings like this (the algoliaQueries was eventually passed to the queries field of this plugin):

// Primary index
algoliaQueries.push({
  forwardToReplicas: true,
  settings: {
    replicas: [
      `${indexName}_${lang.code}_most_popular`,
      `${indexName}_${lang.code}_newest`,
      `${indexName}_${lang.code}_oldest`,
      `${indexName}_${lang.code}_alpha`
    ],
    unretrievableAttributes: [
      ...commonUnretrievableAttributes,
      ...refinements.map(r => withPrefix(r.facetingAttributeName))
    ],
    attributesForFaceting: refinements.map(r =>
      withPrefix(r.facetingAttributeName)
    ),
    sortFacetValuesBy: "alpha"
  },
  transformer: ({ data }) => {
    const items = getContentItemsFromQueryResult(data);
    const result = items.map(addAlgoliaFields);
    return result;
  },
  query: query(lang.name),
  indexName: `${indexName}_${lang.code}`
});

I then had a separate script that created my replicas as detailed in this issue: #41

The replica generation script ran first followed by the plugin's code that created the primary indices. However, halfway through the process (half of my indices still had _tmp tacked onto the end of them in the Algolia web portal), I would get an error saying AlgoliaSearch: cannot move with a primary/replica index as source. This error seemed to persist no matter what I did.

Eventually, I ended up just writing my own implementation of the plugin.

Add tests

  1. add jest to the project
  2. test function with all Gatsby parts being mocked (algoliasearch the module should also be mocked)
  3. think of a proper e2e test later

These are the steps I'm thinking of. @janosh, you said you were interested in contributing, these are the steps I was thinking of

trying to connect data pulled in from airtable into gatsby graphql to algolia, plugin not working

this plugin doesn't seem to be working in my gatsby-config when i run a build (doesn't populate my algolia index) -- i've already pushed data into my index using algoliasearch and a json file, but i want this to be automatically hooked up whenever i build -- so the data is always current with my airtable data.

i've tried the 'gatsby-plugin-algolia' approach via the documentation on github (placed in my gatsby-config.js file)

const myQuery = `{
  allSitePage {
    edges {
      node {
        # try to find a unique id for each node
        # if this field is absent, it's going to
        # be inserted by Algolia automatically
        # and will be less simple to update etc.
        objectID: id
        component
        path
        componentChunkName
        jsonName
        internal {
          type
          contentDigest
          owner
        }
      }
    }
  }
}`;

const queries = [
  {
    query: myQuery,
    transformer: ({ data }) => data.allSitePage.edges.map(({ node }) => node), 
    indexName: 'cardDemo',
  },
];

module.exports = {
  plugins: [
        {
      resolve: 'gatsby-source-airtable-linked',
      options: {
        apiKey: process.env.MY_API_KEY,
        tables: [
          {
            baseId: process.env.MY_BASE_ID,
            tableName: 'Streams',
            tableView: 'DO NOT MODIFY',
          },
        ],
      },
    },
    {
      resolve: 'gatsby-plugin-algolia',
      options: {
        appId: process.env.MY_AGOLIA_APP_ID,
        apiKey: process.env.MY_AGOLIA_API_KEY,
        indexName: 'cardDemo',
        queries,
        chunkSize: 1000000,
      },
    },
  ],
};

i've also subbed out the 'myQuery' for a more specific instance that i'm using on a component via airtable, shown below

const myQuery = `{
      items: allAirtableLinked(
      filter: {
        table: { eq: "Items" }
      }
    ) {
      edges {
        node {
          id
          data {
            title
            thumbnail_url
            thumbnail_alt_text
            url_slug
            uberflip_stream_id
            uberflip_id
          }
        }
      }
    }
    }`;

if anyone has this plugin running and working -- i could definitely use some hints as to how to get this working (not much documentation on this package)

thank you!

ENV Variables not recognised when deploying

I am having problems trying to deploy a Gatsby site with Algolia search functionality included.
I have a .env file with my ENV_VARIABLES set up and I have transferred these to the deploy settings section of the netlify ui.

These build and run locally, however I am getting netlify build failures when I deploy, that say:

12:10:12 PM:   AlgoliaSearchError: Please provide an application ID. Usage: algoliasearch(app  licationID, apiKey, opts)
12:10:12 PM:   
12:10:12 PM:   - AlgoliaSearchCore.js:51 AlgoliaSearchNodeJS.AlgoliaSearchCore
12:10:12 PM:     [repo]/[gatsby-plugin-algolia]/[algoliasearch]/src/AlgoliaSearchCore.js:51:1    1
12:10:12 PM:   
12:10:12 PM:   - AlgoliaSearch.js:11 AlgoliaSearchNodeJS.AlgoliaSearch
12:10:12 PM:     [repo]/[gatsby-plugin-algolia]/[algoliasearch]/src/AlgoliaSearch.js:11:21

I have the following in my .env file:

GATSBY_ALGOLIA_APP_ID=057XXXXXXX
GATSBY_ALGOLIA_SEARCH_ONLY_API_KEY=b79XXXXXXXXX
ALGOLIA_ADMIN_API_KEY=fdc75d20eb2dXXXXXXXXX
GATSBY_ALGOLIA_INDEX_NAME=XXXXXX

I have transferred all these (except the admin key) to the build section of netlify but I still get the error. I've tried with and without the GATSBY_ prefix as well but still no luck.

Any ideas or recommendations for how to set this up to deploy?
Thanks

Disable indexing on build?

Hi,

I want to be able to disabling indexing of data on dev builds in my local environment. Is there anyway to block this? In the past I just set the following environment variables to garbage values:

ALGOLIA_ADMIN_KEY
GATSBY_ALGOLIA_APP_ID
GATSBY_ALGOLIA_SEARCH_KEY

but lately that doesn't work as the dev build errors out in my CI/CD pipeline with this:


Algolia: 1 queries to index
--
53 | Algolia: query 0: executing query
54 | Algolia: query 0: splitting in 1 jobs
55 | error failed to index to Algolia
56 | ย 
57 | ย 
58 | AlgoliaSearchError: Invalid Application-ID or API key
59 | ย 
60 | - AlgoliaSearchCore.js:377 success
61 | [website]/[algoliasearch]/src/AlgoliaSearchCore.js:377:32
62 | ย 
63 | - task_queues.js:97 processTicksAndRejections
64 | internal/process/task_queues.js:97:5
65 | ย 


Anyway to get around this?

read and uses settings from the original index when settings argument is not given

If user doesn't pass settings argument, then we need to read the settings from Algolia server and apply them to the new temporary index. Otherwise, the settings will be lost.

https://github.com/algolia/gatsby-plugin-algolia/blob/master/gatsby-node.js#L167

Before this if statement, we could do something like

if (!settings && indexToUse === tempIndex) {
  settings = await index.getSettings()
}

If it's the same index, we don't need to get the whole settings and apply them again.

ERROR #11321 PLUGIN --> AlgoliaSearchError: Please provide an application ID.

Hiya! One moment, everything was working just fine. And the next, I started receiving an error. When I run gatsby build, I now receive:

frontend.v1|Algolia โ‡’ gatsby build
success open and validate gatsby-configs - 0.042 s
success load plugins - 1.139 s
success onPreInit - 0.017 s
success delete html and css files from previous builds - 0.029 s
info One or more of your plugins have changed since the last time you ran Gatsby.
As
a precaution, we're deleting your site's cache to ensure there's not any stale
success initialize cache - 0.046 s
success copy gatsby files - 0.066 s
success onPreBootstrap - 0.022 s
success source and transform nodes - 0.182 s
success Add explicit types - 0.019 s
success Add inferred types - 0.214 s
success Processing types - 0.102 s
success building schema - 0.433 s
success createPages - 0.048 s
success createPagesStatefully - 0.061 s
success onPreExtractQueries - 0.013 s
success update schema - 0.043 s
success extract queries from components - 0.398 s
success write out requires - 0.015 s
success write out redirect data - 0.013 s
success Build manifest and related icons - 0.177 s
success onPostBootstrap - 0.195 s
โ €
info bootstrap finished - 6.407 s
โ €
success run static queries - 0.047 s โ€” 2/2 59.81 queries/second
success Building production JavaScript and CSS bundles - 9.594 s
success Rewriting compilation hashes - 0.027 s
warn code block or inline code language not specified in markdown. applying generic
 code block
success run page queries - 0.645 s โ€” 10/10 16.18 queries/second
success Building static HTML for pages - 1.744 s โ€” 10/10 33.95 pages/second
info Generated public/sw.js, which will precache 7 files, totaling 351300 bytes.
The following pages will be precached:
/offline-plugin-app-shell-fallback/index.html
โ  
 ERROR #11321  PLUGIN

"gatsby-plugin-algolia" threw an error while running the onPostBuild lifecycle:

Please provide an application ID. Usage: algoliasearch(applicationID, apiKey, opts)



  AlgoliaSearchError: Please provide an application ID. Usage: algoliasearch(applic  ationID, apiKey, opts)
  
  - AlgoliaSearchCore.js:50 AlgoliaSearchNodeJS.AlgoliaSearchCore
    [frontend.v1]/[algoliasearch]/src/AlgoliaSearchCore.js:50:11
  
  - AlgoliaSearch.js:11 AlgoliaSearchNodeJS.AlgoliaSearch
    [frontend.v1]/[algoliasearch]/src/AlgoliaSearch.js:11:21
  
  - AlgoliaSearchServer.js:17 AlgoliaSearchNodeJS.AlgoliaSearchServer
    [frontend.v1]/[algoliasearch]/src/server/builds/AlgoliaSearchServer.js:17:17
  
  - node.js:83 new AlgoliaSearchNodeJS
    [frontend.v1]/[algoliasearch]/src/server/builds/node.js:83:23
  
  - node.js:68 algoliasearch
    [frontend.v1]/[algoliasearch]/src/server/builds/node.js:68:10
  
  - gatsby-node.js:18 Object.exports.onPostBuild
    [frontend.v1]/[gatsby-plugin-algolia]/gatsby-node.js:18:18
  
  - api-runner-node.js:234 runAPI
    [frontend.v1]/[gatsby]/dist/utils/api-runner-node.js:234:37
  
  - api-runner-node.js:347 Promise.catch.decorateEvent.pluginName
    [frontend.v1]/[gatsby]/dist/utils/api-runner-node.js:347:15
  
  - debuggability.js:427 Promise._execute
    [frontend.v1]/[bluebird]/js/release/debuggability.js:427:9
  
  - promise.js:518 Promise._resolveFromExecutor
    [frontend.v1]/[bluebird]/js/release/promise.js:518:18
  
  - promise.js:103 new Promise
    [frontend.v1]/[bluebird]/js/release/promise.js:103:10
  
  - api-runner-node.js:346 
    [frontend.v1]/[gatsby]/dist/utils/api-runner-node.js:346:12
  
  - util.js:16 tryCatcher
    [frontend.v1]/[bluebird]/js/release/util.js:16:23
  
  - reduce.js:166 Object.gotValue
    [frontend.v1]/[bluebird]/js/release/reduce.js:166:18
  
  - reduce.js:155 Object.gotAccum
    [frontend.v1]/[bluebird]/js/release/reduce.js:155:25
  
  - util.js:16 Object.tryCatcher
    [frontend.v1]/[bluebird]/js/release/util.js:16:23

Here is my repo! I imagine my .env file will not be visible, but it does exist with my Angolia credentials accurately stored. Any debugging types are gladly welcomed! Thanks in advance.

Unresolved URLs

URLs are undefined when searching and clicking on a hit. I have been searching pretty much everywhere, but I think it is an issue with the way my hits are set up and the queries I am doing, however I don't know where to look.

// algolia.js GraphQL queries required in gatsby-config.js
const postQuery = `{
  posts: allMarkdownRemark(
    filter: { fileAbsolutePath: { regex: "/src/pages/" } }
  ){
    edges {
      node {
        fields {
          slug
        }
        objectID: id,
        frontmatter {
          title
          date(formatString: "MMM DD, YYYY")
          spoiler
        }
        excerpt(pruneLength: 500)
      }
    }
  }
}`


const flatten = arr =>
  arr.map(({ node: { frontmatter, slug, ...rest } }) => ({
    ...frontmatter,
    ...slug,
    ...rest
  }))
const settings = { attributesToSnippet: [`excerpt:20`] }

const queries = [
  {
    query: postQuery,
    transformer: ({ data }) => flatten(data.posts.edges),
    indexName: `articles`,
    settings,
  },
]

module.exports = queries
// Search component

import React, { useState } from 'react'
import styled from '@emotion/styled'
import algoliasearch from 'algoliasearch/lite'
import {
  Configure,
  connectHits,
  connectSearchBox,
  InstantSearch,
  Highlight
} from 'react-instantsearch-dom'
import themes from './themes'
import { FaSearch } from 'react-icons/fa'
import Overlay from './Overlay'
// import '../utils/algolia'

const client = algoliasearch('VVKLD2O8OY', 'b5faa712b4c07dc61c03eec1e79a4954')

const SearchArea = styled('div')`
  height: 100vh;
  margin-top: 0;
  overflow-y: scroll;
  padding: 3rem 5%;
  width: 100%;
`

const List = styled('ul')`
  list-style: none;
  margin: 0 auto;
  max-width: 650px;
  padding: 0;
  color: ${themes.light.primary.text.normal};
`

const Result = styled('li')`
  margin-top: 2rem;
  color: ${themes.light.primary.text.normal};
`

const Heading = styled('h2')`
  font-size: 1.25rem;
  font-weight: 600;
  a {
    color: ${themes.light.primary.text.link};
    text-decoration: none;
    :active,
    :focus,
    :hover {
      color: ${themes.light.secondary.text.link};
    }
  }
`

const Link = styled('a')`
  display: inline-block;
  font-size: 0.75rem;
  letter-spacing: 0.1em;
  margin-top: 0.5rem;
  text-decoration: none;
  text-transform: uppercase;
  color: ${themes.light.primary.text.normal} !important;
`

const Hits = connectHits(({ hits }) => (
  <List>
    {hits.map(hit => (
      <Result key={hit.objectID}>
        <Heading>
          <a href={`/${hit.slug}`}>
            <Highlight attribute="title" hit={hit} tagName="mark" />
          </a>
        </Heading>
        <p>
          <Highlight attribute="spoiler" hit={hit} tagName="mark" />
        </p>
        <Link href={`/${hit.slug}`}>Read this post &rsaquo;</Link>
      </Result>
    ))}
  </List>
))

const OpenSearch = styled('a')`
  align-self: center;
  border: 2px solid transparent;
  color: ${themes.light.primary.header};
  height: 100%;
  margin: 0;
  padding: 0 0.625rem;
  width: 2.375rem;
  :active,
  :focus,
  :hover {
    background-color: transparent;
    color: ${themes.light.secondary.text.link};
  }
  :focus {
    border: 2px solid ${themes.dark.inlineCode.background};
    border-radius: 0;
  }
  @media ${themes.breakpoints.s} {
    width: 2.5rem;
  }
`

const Icon = styled(FaSearch)`
  height: 100%;
  margin: 0;
  position: relative;
  top: -0.125em;
`

const Label = styled('label')`
  display: block;
  margin: 0 auto;
  max-width: 650px;
`

const Input = styled('input')`
  border: 2px solid ${themes.light.primary.text.title};
  border-radius: 4px;
  display: block;
  font-size: 1.25rem;
  margin-top: 0;
  padding: 0.5rem 0.75rem;
  width: 100%;
`


const Search = connectSearchBox(({ currentRefinement, refine, setActive }) => (
  <form noValidate action="" role="search">
    <Label htmlFor="search">
      <span>Search the Blog</span>
      <Input
        type="search"
        id="search"
        value={currentRefinement}
        onBlur={() => {
          if (currentRefinement === '') {
            setActive(false)
          }
        }}
        onChange={event => {
          setActive(true)
          refine(event.currentTarget.value)
        }}
      />
    </Label>
  </form>
))

const SearchContainer = styled('div')`
  display: flex;
  align-items: flex-start;
  margin-left: auto;
  margin-top: 0;
  color: ${themes.light.secondary.text.normal};
`

export default () => {
  const [active, setActive] = useState(false)

  return (
    <InstantSearch
    searchClient={client}
    indexName="articles"
    root={{ Root: SearchContainer }}
    >
      <Configure distinct={1} />
      <OpenSearch
        href="/search"
        onClick={event => {
          event.preventDefault()
          setActive(true)
        }}
      >
        <Icon title="Search the blog" />
      </OpenSearch>
      <Overlay
        hidePopover={() => {
          setActive(false)
        }}
        visible={active}
      >
        <SearchArea>
          <Search setActive={setActive} />
          <Hits />
        </SearchArea>
      </Overlay>
    </InstantSearch>
  )
}

I used @janosh's tutorial for most of this and another implementation of the search component but I am a bit lost as to the undefined urls.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.