implerhq / impler.io Goto Github PK

Describe the bug
Whenever the user tries terminating ongoing import from the middle of the import process and mapping stage, the API throws the following error,

To Reproduce
Steps to reproduce the behavior:

1. Start any import by clicking on the Import button.
2. Select any file and click on See mapping.
3. Click on Upload Again to reset the import.
4. See the API failing

Expected behavior
The terminate API should handle the edge case record model not found in the database.

Desktop (please complete the following information):

• Browser Chrome
• Version v0.16.1

Additional context
Add any other context about the problem here.

Describe the bug
While providing schema in the OUTPUT editor, the static keys are getting deleted. The provided schema is as follows,

# Chunk Format
{
  "params": {
    "taskType": "process",
    "operationType": "inboundService"
  },
  "entities": "{{data}}"
}
# Record Format
{
  "type": "account",
  "data": {
    "attributes": {
      "baxptyiqviaid": {
        "values": [
          {
            "value": "{{baxptyiqviaid}}",
            "locale": "en-US"
          }
        ]
      }
    }
  }
}

While the receiving OUTPUT is,

{
  "params": {},
  "entities": [
    {
      "data": {
        "attributes": {
          "baxptyiqviaid": { "values": [{ "value": "asdf" }] }
        }
      }
    }
  ]
}

Which removed the static keys.

To Reproduce
Steps to reproduce the behavior:

1. Create import.
2. Update the OUTPUT format as described.
3. Try doing imports.
4. See the webhook data received.

Expected behavior
The static keys should not be removed from the webhook.

• When a developer provides S3 details to the application and the application starts storing files in S3, the AWS SDK throws a NoSuchBucket exception if the specified bucket does not exist.
• Currently, this exception is not handled gracefully, leading to a backend application crash.
• To improve the user experience, we should control this exception and return a user-friendly error response to the frontend instead of crashing the backend application.
• It's an important case when working with Localstack on local computer.

Steps to Reproduce:

1. Provide valid S3 details, and invalid bucket name in the API env configuration.
2. Start the application.
3. Attempt to store a file in S3 using the application.

Current Behavior:
The backend application crashes with a NoSuchBucket exception when the specified bucket does not exist.

Expected Behavior:
The application should handle the NoSuchBucket exception gracefully and return a well-formatted error response to the front end, indicating that the specified bucket does not exist.

Proposed Solution:

• Catch the NoSuchBucket exception when interacting with the AWS SDK.
• Instead of letting the exception propagate and crash the application, handle it with try...catch and construct an error response.
• Include relevant information in the error response, such as the bucket name provided by the developer.
• Return the error response to the frontend for display.

Additional Information:

• It would be beneficial to log the exception for debugging purposes.

Priority:
High

chunkSize in template refers to how many data will be sent to application at a time while processing file. Currently we support sending data to application via webhook call to the URL provided to the template.

chunkSize parameter takes 0 if value is not provided while creating template. It will be good if we can provide default value to it.

Tracking issue for:

• https://github.com/implerhq/impler.io/security/code-scanning/13

While running impler, in the self-hosted version, the /v1/health-check API returns storage = down even though credentials for AWS are valid, and services are storing files in S3 Bucket.

While using Button from @impler/react, it throws an error, in nextjs application.

Server Error
TypeError: n.button is not a function

This error happened while generating the page. Any console logs will be displayed in the terminal window.

• Introduce a feature in Impler.io that enables time-limited API calls when interacting with destination APIs, particularly for webhook integrations. This enhancement is designed to address scenarios where the destination system has limitations on the capacity to process the request.
• By implementing time constraints on API calls, Impler.io can ensure a controlled and sustainable data transfer process, preventing potential system crashes and accommodating environments with low-capacity servers.

Use Case Scenario:
In situations where Impler.io needs to push data to an external system via a webhook, the destination API may have limitations on the rate at which it can accept requests. For instance, the API may only want to process one call every 20 seconds to avoid overwhelming the system. With the time-limited API calls feature, Impler.io can adhere to this restriction, preventing disruptions and ensuring smooth data processing.

Expected Behavior:

• Impler.io allows users to configure time limits for API calls when setting up webhook integrations.
• API calls to the destination system adhere to the specified time limit, with a predefined interval between consecutive calls.
• The system automatically manages and schedules API calls to prevent exceeding the allowed frequency.

Additional Context:

Priority:
Medium

Attachments:

Acceptance Criteria:

1. Users can set up time limits for API calls when configuring webhook integrations in Impler.io.
2. API calls adhere to the specified time limits, preventing calls at a frequency that exceeds the destination API's constraints.
3. The feature is seamlessly integrated, and the system effectively manages and schedules API calls based on the configured time limits.

Is your feature request related to a problem? Please describe.
Current usage of jQuery for showing tooltips is not recommended while displaying error messages to users.

Describe the solution you'd like
We can try implementing https://floating-ui.com/docs/useHover to check whether we can optimize and show errors more carefully during the review phase. Review data is getting rendered at Table Component, which is using jQuery tooltip for rendering error message, which can be altered with floating-ui tooltip.

Additional Context

• https://floating-ui.com/docs/useHover

Describe the bug
When providing template prop to Import button from @impler/react and selecting csv file to upload, mapping button stops working.

To Reproduce
Steps to reproduce the behavior:

1. Setup application with widget.
2. Provide template to Import button from @impler/react
3. Select file to import
4. Click on See mapping to see error in network tab.

Expected behavior
Template selection should work

Screenshots

Desktop (please complete the following information):

• OS: Linux
• Browser: chrome
• Version 0.1.5

Additional context

Is your feature request related to a problem? Please describe.
In current scenario, if a user uploads an Excel file that has multiple sheets then by default Impler processes the first sheet and tries importing it. Which is not the case always.
So when a user uploads a file with multiple sheets, the platform should display a modal popup, allowing the user to choose the specific sheet they want to import.
This enhancement will streamline the import process for multi-sheet Excel files, providing users with greater control and flexibility.

Describe the solution you'd like

1. When a user uploads an Excel file, the import widget checks if the Excel file contains multiple sheets.
2. If it is, the widget displays a modal popup listing all available sheets in the file.
3. Users can select the desired sheet from the modal popup, specifying the sheet to be imported.
4. Upon selection, the import widget initiates the import process using the chosen sheet's data.

Is your feature request related to a problem? Please describe.
It's always frustrating for developers and people visiting impler to understand What is impler? How it works? etc. Any visitor wants to see a live demo, something to play, something to do. So adding a demo website, where anyone can try importing data, will give a better understanding of functionality.

Describe the solution you'd like
The demo site should be a one-page app, that allows:

• Downloading sample data file
• Import sample data file by clicking on the Import button
• See imported data in the table when import gets completed

Describe alternatives you've considered

Additional context

Describe the bug

• Here it does not show that someone can create a new project by clearing the text and clicking on add button.
  

To Reproduce
Steps to reproduce the behavior:

1. Go to https://web.impler.io
2. Log in to the dashboard.

Expected behavior
It should explicitly indicate that there is a way to create a project.

Desktop (please complete the following information):

• Browser: Chrome

here is the error that encountered when starting application using docker-compose

widget           | > @impler/[email protected] envsetup:docker /usr/src/app/apps/widget
widget           | > chmod +x ./env.sh && ./env.sh && mv ./env-config.js ./build/env-config.js
widget           | ./env.sh: line 27: .env: No such file or directory

Describe the bug
Here while waiting for review data, it should show a message like Loading review data... instead of No data found, because the data is already there.

To Reproduce
Steps to reproduce the behavior:

1. Go to demo
2. Start the import
3. Complete the Upload and Mapping step
4. See the review

Expected behavior
The table should show the proper message

Screenshots

Desktop

• Browser - chrome

• Implement a feature that sends an invalid imported file to the user for review when Impler.io rejects the file due to issues such as file corruption or inability to open.
• This enhancement aims to provide teams with more transparency and control over the import process by allowing them to inspect and rectify issues with the file.

Use Case Scenario:
Imagine a scenario where a user attempts to import a dataset, but the file is corrupted or cannot be opened by the platform. Instead of outright rejecting the file, the platform can provide an option to send the file to the user for review. The user can then analyze and fix the problems and possibly inform the customer.

Expected Behavior:

• Platform will provide import settings where there will be a checkbox for whether to get an invalid file for review.
• When an imported file is deemed invalid, the platform sends an email to the user with the file as an attachment.

Additional Context:

Priority:

Attachments:

Acceptance Criteria:

• The Import settings will have an option for the user to receive an invalid file for review.
• Impler.io identifies and notifies users about invalid imported files.

Describe the bug
The sample URLs that get generated for Templates is not generating correctly.

To Reproduce
Steps to reproduce the behavior:

1. Create Project.
2. Create Template.
3. Add columns to the template
4. Get templates to see sampleFileUrl, sample file URL is showing as https://bucket.amazonaws.com//impler/638487e5f4152caeea1ac28a/sample.csv which has extra / before impler.

Expected behavior
Generated URL should be valid and publicly accessible.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Ubuntu
• Browser: Chrome
• Version v0.1.4

Additional context
Add any other context about the problem here.

Implement a Readymade Templates feature in Impler.io to provide developers with pre-configured templates for common import use cases.
This functionality aims to streamline the import process by offering ready-to-use templates for scenarios such as employee import, leads import, and others.
Developers can use these templates as boilerplates, to significantly reduce the time spent on configuration and iteration.

Use Case Scenario:
Consider scenarios where developers frequently perform similar imports, such as employee import or lead import. With Readymade Templates, developers can choose a template that aligns with their specific use case, eliminating the need to configure each import from scratch. This accelerates the setup process and ensures consistency across similar imports.

Expected Behavior:

• Impler.io provides a library of Readymade Templates for common import use cases.
• Users can select a template when creating a new import, pre-configuring the import settings and schema based on the selected template.
• Templates will only contain import schema configuration, developer can use any template as a base and modify it after selection.

Priority:
High

Attachments:
*will be added after design is ready

Acceptance Criteria:

• Impler.io provides a library of Readymade Templates for common import scenarios.
• Users can select a template during the import creation process to pre-configure settings and schema.
• Templates are customizable, allowing users to modify specific settings as needed.

Implement a Column Ordering Facility which will allow developers to customize the arrangement of columns in the import configuration table. With this feature developers and users will have greater flexibility and ease of use while working with the platform.

Use Case Scenario:
In scenarios where imports involve a substantial number of columns, the ability to customize the order of columns becomes crucial. For example, a developer may prefer to have essential columns displayed at the beginning for quick reference. The Column Ordering Facility caters to such needs, offering a user-friendly interface for rearranging columns in the import configuration.

Expected Behavior:

• Users can access a drag-and-drop interface to reorder columns within the import configuration table.
• The custom column order is preserved for each import configuration without clicking on any button.
• The sequence property in the column table will be updated according to order.

Priority:
Medium

Attachments:

Acceptance Criteria:

• Users can drag and drop columns within the import columns table to reorder them.
• Changes to the column order are immediately saved without any button click.

Steps to Reproduce:-

1. Open Widget
2. Select Template, File and click on See Mapping
3. Click on Upload Again

Template data will not be visible.

It should,

1. Reset AppState
2. Reset Queries
3. Template data should be visible.

Handle error if window.impler is not defined or script is not loaded

Integrate a Duplicate Import functionality in Impler.io to allow users to effortlessly replicate existing imports with the same schema and configuration.
This feature aims to reduce the developer's effort when dealing with repeated imports or when similar configurations are required for different environments.
It will minimize manual work in setting up identical imports.

Use Case Scenario:
Consider a scenario where a developer frequently needs to import data with the same schema at different places and environments (e.g., development, testing, production). Instead of recreating the import configuration each time, the Duplicate Import feature allows the developer to clone an existing import, saving time and ensuring consistency across environments.

Expected Behavior:

• Users can select an existing import configuration to duplicate.
• The Duplicate Import function replicates the selected import, including its schema, settings, and configurations.
• The duplicated import appears as a new entry with an option to customize its specific details if needed.

Additional Context:

Priority:
High

Attachments:

Acceptance Criteria:

• Users can easily duplicate an existing import configuration on the platform.
• The duplicated import retains the schema, settings, and configurations of the original import, regardless of history.
• Users have the option to customize specific details of the duplicated import.

Introduce the ability for users to perform multi-select in a column within Impler.io's data import feature. This enhancement includes adding support for selecting multiple items from a dropdown in the created Excel sample.
Additionally, during the widget review phase, an error should be triggered if the selected items are not from the provided list. This feature aims to open doors for a wide range of data import needs where the user wants to import data like tags, categories etc.

Use Case Scenario:

• Consider a scenario where a user is importing a dataset containing information about products.
• The dataset includes a column for product categories. With the multi-select feature, the user can now choose multiple categories for each product directly in the Excel dropdown.
• This is particularly beneficial when dealing with products that belong to multiple categories simultaneously.

Expected Behavior:

1. Users can select multiple items from a dropdown in the Excel sample for a specific column.
2. During the widget review phase, an error is triggered if any selected item in the multi-select column is not part of the predefined list.

Additional Context:

Priority:
Medium

Acceptance Criteria:

1. Users can select multiple items from a dropdown in the Excel import interface.
2. The widget review phase triggers an error if any selected item is not from the predefined list.
3. The feature is successfully integrated and does not affect the existing import process.

Describe the bug
When navigating to the page from the app, where the Impler button is added it works well. But hot refreshing the page throws an error.

To Reproduce
Steps to reproduce the behavior:

1. Add Import Button and embed script
2. Go to the import page where the button is added
3. Refresh the page
4. See the error

Expected behavior
It should work well

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu
• Browser: Chrome
• Version: 0.1.7

Additional context

Installation of @impler/react fails but it gets installed correctly. Possibly need to remove the preinstall script that builds the package.

Is your feature request related to a problem? Please describe.
It seems obvious that the createdAt, updatedAt, and __v fields are not used anywhere on the front end. so we can remove them from API responses.

Describe the solution you'd like

1. Add an Exclude decorator to an entity, like,

import { Exclude } from 'class-transformer';
export class MyEntity {
  ...
  @Exclude()
  createdAt: Date;
  @Exclude()
  updatedAt: Date;
  @Exclude()
  __v?: number;
}

2. Add __v in the SchemaDocument,

interface IMyDocument extends MyEntity, Document {
  _id: never;
  __v: never;
}

4. In the controller apply the ClassSerializerInterceptor interceptor, like,

import { Body, ClassSerializerInterceptor, UseInterceptors } from '@nestjs/common';
...

@UseInterceptors(ClassSerializerInterceptor)
export class MyController {
  ...
}

5. Build dal lib by running pnpm build:dal.
6. Test respective API is sending the correct response.

Describe the bug
The background was transparent, but version 0.1.7 has an issue in that background is grayed out.

To Reproduce
Steps to reproduce the behavior:

1. Open the widget, and see an outer area of the modal.

Expected behavior
The outer area of the widget should be a transparent bit.

Describe the bug
Email must be valid error whenever i import bulk user.

To Reproduce
Steps to reproduce the behavior:

1. Go to bulk import and select file
2. Click on see maping
3. Click on Review Data
4. See error

Expected behavior
validate email proper.

Screenshots

Desktop (please complete the following information):

• Browser [e.g. chrome]

Describe the bug
When the user changes items on a par page and the user is on a page that is out of context for the new page size. The imports page shows empty imports.

To Reproduce
Steps to reproduce the behavior:

1. Create more than 50 imports.
2. Keep per page records size 10 and go to the last page of imports.
3. Switch per page records size to 50.
4. See the empty imports page

Expected behavior
When a user tries navigating among sizes of the imports page, if the current page number is within the context of the new page size then keep it else switch the page to the last page of the new page size.

Screenshots
https://github.com/implerhq/impler.io/assets/50201755/cb18c549-02e3-4074-8e3c-d6b528dcf911

Currently, the buttons are coming one after another, it should come in line.

Steps to reproduce

• Clone the repository
• Copy .env.example to the .env file
• Put the access-token value
• Start the application and run get projects API, you won't see the need for authorization

Possible Fix

• Change token name from access-token to access-key

Currently widget is persisting its state like last phase, that should not happen.
Create method to reset state when SHOW_WIDGET event gets called.

Integrate pagination functionality on the Imports page in Impler.io to enhance the user experience and streamline the management of multiple imports.
Currently, as developers create separate imports for different environments, the Imports page can become cluttered, requiring excessive scrolling to locate specific imports.
Adding pagination will allow users to navigate through import entries more efficiently.

Use Case Scenario:
In environments where developers regularly create multiple imports for various purposes (e.g., production, staging, development, UAT), the Imports page can become overwhelming, making it challenging to find specific imports.
Pagination provides a solution by organizing imports into manageable sections, reducing clutter and simplifying navigation.

Expected Behavior:

• The Imports page displays a specified number of imports per page.
• Users can navigate through multiple pages to find and manage their imports.
• Options for customizing the number of imports per page is available.
• With search input imports can be searched.

Additional Context:

Priority:
High

Attachments:

Acceptance Criteria:

• The Imports page displays a specified number of imports per page, with pagination controls.
• Users can navigate through multiple pages to find and manage their imports.
• Users have the option to customize the number of imports displayed per page in their settings.
• Users can search over their imports.

Implement a feature in Impler.io that allows for the automatic import of files from FTP (File Transfer Protocol) locations.
This enhancement is particularly useful for scenarios where data, such as attendance records from tracking devices, is stored in FTP locations. Enabling Impler.io to seamlessly fetch and import files from FTP locations and ensuring real-time integration with external systems.

Use Case Scenario:
Consider a situation where attendance tracking devices generate daily reports and store them in an FTP directory. With the automated FTP import feature, the platform can be configured to regularly check the FTP location, retrieve the latest attendance file, and import it into the system. This eliminates the need for manual intervention and ensures that attendance data is consistently and promptly updated to the application.

Expected Behavior:

• Users can configure Impler.io to connect to an FTP location and specify the relevant credentials.
• Impler.io automatically fetches files from the specified FTP location based on a defined schedule or trigger.
• The imported data is processed and integrated into the platform without requiring manual intervention and will be sent to the application.

Additional Context:

Priority:

Attachments:

Acceptance Criteria:

• Users can configure FTP connection details within Impler.io.
• Impler.io successfully fetches files from the specified FTP location based on the configured schedule or trigger.
• Imported data from FTP locations is processed and integrated into the platform and gets sent to the application.

Describe the bug
When trying to bulk update columns, getting an error.

TypeError: Cannot set properties of null (setting 'recordVariables')
    at UpdateTemplateColumns.<anonymous> (/usr/src/app/apps/api/dist/main.js:3276:43)
    at Generator.next (<anonymous>)
    at fulfilled (/usr/src/app/apps/api/dist/main.js:3225:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

To Reproduce
Steps to reproduce the behavior:

1. Try updating the columns via bulk update API.

Expected behavior
It should update the columns

Screenshots

Desktop (please complete the following information):

• Browser: Chrome

When running a self-hosted version of impler, even though ACCESS_KEY is mentioned in the /docker/.env file, APIs are returning a response without a need for Authentication.

Referencing the description in discussion link
#105

Is your feature request related to a problem? Please describe.
When the upload gets started, the developer may want to listen for events regarding the upload being started, upload being completed, or upload being terminated. By listening to events, the application may like to call the API, change status, or show breadcrumb.

Describe the solution you'd like
The following events should be enough to implement something on the application,

• UPLOAD_STARTED - When the user selects the file to upload and start mapping
• UPLOAD_COMPLETED - When the upload is completed
• UPLOAD_TERMINATED - When the upload is stopped by the user

Describe alternatives you've considered

Additional context

Implement interactivity between widget and API such that one can Upload file.

iFrame compoent gets space of 100vh which makes it look like another application is opened, instead make it to take only space that is used by widget component.

How it looks currently,

1. A spelling mistake in Download Tempreture Data
2. Website flicker in styling before it is laid out properly
3. Same name Latitude columns in the table
4. Update links in README.md and documentation
5. Styling issue in min size laptop
   
6. Show what is the issue in invalid data

Setup Two Projects,

1. One for Npm Library that renders UI
2. Actual UI application

Describe the bug
The table gets overflowed from the widget.

To Reproduce
Steps to reproduce the behavior:

1. Go to demo
2. Start the import and complete the Upload and Mapping step
3. Open Devtools from bottom
4. See the design

Expected behavior
The table should stay inside the widget modal and should not overflow.

Screenshots

Desktop (please complete the following information):

• Browser - chrome

Make Mapping stage to interact with API and make updates