openfoodfoundation / integrations Goto Github PK

Description:

Since n8n is self hosted by OFN. There is a limited amount of storage/memory available for it (this can be increased/decreased by paying more/less for the hosting space). All the n8n instances/accounts share this storage/memory. Therefore, if multiple instances are running memory intensive workflows at the same time then some running workflows may crash. We would get an error like: n8n has run out of memory or execution can't be found etc. But those same workflows may work fine if then are run at different time.

I had this issue multiple times when I run a memory intensive workflows like working with bulk products API with an output of over 1000+ variants using function nodes or other modules like airtable.

As more and more people/instances start using n8n, we might face the memory issue often which will in turn make n8n automations less reliable.

There are ways/good practices to reduce memory usage of a workflow for ex. using sub-workflows for memory intensive tasks in conjunction with split in batches, avoid using code/function module if possible etc. Obviously we can have a list of methods/tips to reduce memory of a workflow, but it might not be that effective. We need a plan on how to increase in n8n usage overtime while limiting the memory issue.
Alternatively we can look a hosting n8n in multiple locations. Instances with heavy use of n8n can self host their own n8n, while smaller instances can still use the common n8n server.
Well theses are some thoughts as of now. Need to brainstorm and check what other options we have.

Description:

Task List:

• Updating Global Dashboard with data from other instances for 2022
• Completing the n8n workflow by adding modules for all the remaining instances
• Documentation (incl. link) to workflows used.

Delivery Stage

• Initial proposal
• Agree implementation proposal
• Estimating
• Funding Confirmed
• Delivered

1. What is the need / problem?

As some of their producers would like to communicate directly with the customer about their order.

2. Which type of users does this problem affect (and how many, if known)?

Producers

3. What tools does this involve?

e.g. - Email

4. Links to connected discussions

https://openfoodnetwork.slack.com/archives/CG7NJ966B/p1680037636709299

5. Implementation Proposal

6. Estimates and funding partners

e.g. Number of dev days. Note contributing instances/partners.

7. Link to Workflows, User Guide etc

Delivery Stage

• Initial proposal
• Agree implementation proposal
• Estimating
• Funding Confirmed
• Delivered

1. What is the need / problem?

Basically - as a supplier to multiple hubs, I need a dashboard to help me manage products (import/export), track my opening stock balance, add in more stock as things are ready in the field, track how much stock has sold in multiple order cycles (I'm thinking 3 OCs - 1 in hub A and 2 order cycles - current week + next week) in hub B.), and calculate how much of my opening + newly added stock is available for sale across these OCs.

Key requirements (added after convo with Theresa)

• Searchable product list (can be filtered/grouped)
• Opening Stock balance (at the start of the week) + updating stock balance few times per week
• Whats selling in which OC (ongoing OCs) (Critical)
• Functionality to add more stock/update stock

2. Which type of users does this problem affect (and how many, if known)?

Producers

3. What tools does this involve?

Airtable

4. Links to connected discussions

https://openfoodnetwork.slack.com/archives/C04U02X3806/p1681411299156449

5. Implementation Proposal

Proposed solution: Building a dashboard in Airtable with features mentioned below.
Method: Use Products API and Postgres DB calls via n8n to get all the relevant data and to run the automations/updates, while keeping all the data in Airtable to simplify searching, filtering, and editing the data.

Features:

• Searchable Product List: Using Products API we can get the product list for a producer in Airtable for the first time, which can be used to search, filter, or group products or variants
• Opening Stock Balance (regular updates): Running Products API at regular intervals to get updated product list and then matching the data obtained via the API call with the variants in Airtable. Intervals for running the update needs to be agreed upon.
• Whats selling in which OC (ongoing OCs): So this one is slightly tricky. We can use Postgres DB calls via n8n to access exchanges (linked with exchange variants, variants, OCs etc.) and get a list of variants selling in an order cycle. But this becomes tricky when a producer is selling products in multiple OCs at the same time. Mostly because of data structure and exchanges being not super clear.
  Therefore, I would like to propose a small test to check the accuracy of data.
  Test: A test producer selling some of their products across multiple test OCs (multiple hubs/single hub - either should be fine) at the same time. Then, developing and running Postgres query (using exchanges, exchange variants etc.) to get the list of variants being sold across different OCs for that producers and comparing the results. Next part would be to change the products being sold across OCs and see if the Postgres query is giving the updated results.
• Functionality to add more stock/update stock: Producer will update the stock value of variants in Airtable, which will then be updated on OFN using Products API (via n8n). This workflow can be started by Producer by clicking a button within Airtable. (Similar to what we have in Aus)

6. Estimates and funding partners

Estimate for test: 5-7 hours (a successful test would roughly complete 30% of the total workload for this issue).

Remaining tasks would involve setting up airtable dashboard, setting up n8n workflows (production), and testing with data from actual enterprises and hubs.

7. Link to Workflows, User Guide etc

Place for @div-yansh-1 to note how he's going with documentation of integrations

Delivery Stage

• Initial proposal
• Agree implementation proposal
• Estimating
• Funding Confirmed
• Delivered

1. What is the need / problem?

UK zapier hosts some automation around dev recruitment:

• When someone fills in the recruitment form they are sent an email
• A message is posted in slack

2. Which type of users does this problem affect (and how many, if known)?

e.g. - Shoppers - Hub Managers (MCFEs) - Producers - Instance Managers - Other (Please specify)

OFn Core team

3. What tools does this involve?

e.g. - Email, Instagram, Airtable, Google Sheets. Only mention tools that you are certain of. If the user needs an email sent then 'email' will be a tool, though other tools (eg N8N, Zapier, API) are implementation decisions.

Google forms, email, slack

4. Links to connected discussions

e.g. - Chats in Slack, Discourse etc

5. Implementation Proposal

6. Estimates and funding partners

e.g. Number of dev days. Note contributing instances/partners.

7. Link to Workflows, User Guide etc

8. Notes and Workings

Specs/requirements being gathered now - could be based on current Zapier integrations in use in the UK (and elsewhere?).

Description

Creating customers on OFN USA from the customer list csv file provided by a hub.

Notes:

• Roughly 600+customers with contact details and addresses (CSV file)
• No tags for this case
• Using V1 API (Customers) and n8n
• Turned on the API V1 feature toggle for a specific user required for the upload only

Delivery Stage

• Initial proposal: monitor feature roll-out through recurrent DB queries, on core-instances
• Discuss on the best process to do so: Metabase vs N8N vs. Ansible task
• Implement the agreed proposal

1. What is the need / problem?

In addition to the communication already done with instances (via Slack, Discourse) it would be good to actually be able to monitor the state in which a given feature is at, in terms of roll-out (which actors, percentage, fully enabled/disabled).

2. Which type of users does this problem affect (and how many, if known)?

• Relevant to the delivery team, to manage feature roll-out
• Relevant to interpret Matomo data

3. What tools does this involve?

Metabase, N8N, code-changes on ofn-install (Ansible)

4. Links to connected discussions

Delivery circle meeting from 04.04.2023

5. Implementation Proposal

WIP, to be discussed: Metabase vs N8N vs. Ansible

Describe the bug
Issue 1 - the export file is NOT capturing all the products in the enterprise product list. Products that are missing all have one thing in common. On the product list, they are showing an import date. All products, on my product list, showing an import date, are not included in the export file. All products, on my product list, without an import date, are included in the export file. (SO - maybe previously imported products are not included in the current n8n download?)

Issue 2 - for this recent export David used a unique login for an enterprise called Garden Party (162) (thats the enterprise I want the export for) But SOME products (and not all) from 2 other enterprises were included in the export file also. The export for this unique login included products from Garden Party Plants (333) and Garden Party Flower Farm (243). Those 2 enterprise (333 & 243) have given enterprise 162 permission for 'everything', so perhaps this is why some of their products got included?
I've tried to figure out why some of these 'other enterprise' products are being included in the export file, but others are not and i can't see anything that defines them.
But for now - it seems that when doing the n8n export file for a given enterprise, that enterprise needs to have 1. a unique login, and ALSO 2. no other enterprise has given 'everything' permissions.

How did you experience it
Running the N8N workflow

Expected behaviour
Ideally it should give a csv file with all the products/variants for an enterprise in the format required for OFN product upload file. This will allow hubs to edit the CSV file with new stock level and upload the same directly on OFN website.

The product list returned should be ALL products for the specified enterprises and no products for enterprises that this enterprise does not have permission to access.

Workflows or tools involved
N8N to CSV export
Link to workflow: https://n8n.openfoodnetwork.org.uk/workflow/55

Additional context
Add any other context about the problem here.
Slack thread

Description:

Product export automation in n8n is having some issues. Ideally it should give a csv file with all the products/variants for an enterprise in the format required for OFN product upload file. This will allow hubs to edit the CSV file with new stock level and upload the same directly on OFN website.

Issues mentioned in the slack thread:

Issue 1 - the export file is NOT capturing all the products in the enterprise product list. Products that are missing all have one thing in common. On the product list, they are showing an import date. All products, on my product list, showing an import date, are not included in the export file. All products, on my product list, without an import date, are included in the export file. (SO - maybe previously imported products are not included in the current n8n download?)

Issue 2 - for this recent export David used a unique login for an enterprise called Garden Party (162) (thats the enterprise I want the export for) But SOME products (and not all) from 2 other enterprises were included in the export file also. The export for this unique login included products from Garden Party Plants (333) and Garden Party Flower Farm (243). Those 2 enterprise (333 & 243) have given enterprise 162 permission for 'everything', so perhaps this is why some of their products got included?
I've tried to figure out why some of these 'other enterprise' products are being included in the export file, but others are not and i can't see anything that defines them.
But for now - it seems that when doing the n8n export file for a given enterprise, that enterprise needs to have 1. a unique login, and ALSO 2. no other enterprise has given 'everything' permissions.

Delivery Stage

• Initial proposal
• Agree implementation proposal
• Estimating
• Funding Confirmed
• Delivered

1. What is the need / problem?

At the end of every order cycle, Hub manager wants an email with a list of producer who had sales in a given order cycle (preferably in form of CSV).

2. Which type of users does this problem affect (and how many, if known)?

Hub Managers

3. What tools does this involve?

Email, CSV

4. Links to connected discussions

https://openfoodnetwork.slack.com/archives/C04U02X3806/p1679964247999209

5. Implementation Proposal

To use a n8n workflow which automatically triggers when an order cycle closes for the hub (using polling with Postgres and function nodes). Then using a SQL query (via the Postgres node in n8n), we will get the list of producers sales and info for that order cycle. After which, the data will be converted into CSV file and will be emailed to the hub manager.

Fields in CSV:

• Order Cycle Name and Id
• Supplier Enterprise Name
• Supplier Enterprise Id
• Supplier Enterprise "Notifications" email address
• Number of orders that include the supplier for the hub's order cycle
• Sum of Quantity*price for items sold in that order cycle

Enterprise Id and OC id are very easy to add in the workflow. So, no extra time is needed for that.

Whats Polling (in context of this issue): Basically, n8n will keep on checking at fixed intervals (every hour - can be reduced if needed), if a new order cycle has closed for the specified enterprise. It's similar to Zapier trigger.

Notes:

• The workflow will be made/developed in USA's n8n account and will be documented in the API handbook.
• Polling interval will be 1 hr initially (i.e. there can be a time gap of up to 1 hr between order cycle closing and the email being sent) - This can be reduced if needed. Polling can be replaced with OFN Order cycle close webhook once it's ready.
• This workflow could be adapted for other hubs by replacing the enterprise id used in SQL query and the email id of the hub manager.
• Email subject and body can be anything (text) (USA can propose what they want email subject and contents to be)
  - Need to connect a Gmail account in USA's n8n account to send emails. To do this, I would need access to the email account which would be sending the email.

6. Estimates and funding partners

10-12 hrs

7. Link to Workflows, User Guide etc

Scope/Exact details are being worked out at the moment.