chartbrew / chartbrew Goto Github PK

How to reproduce

• Ensure there is an API connection with global headers already saved in your Chartbrew project
• Create new chart and prepare your API request in the API request builder
• Before clicking next untick global headers > press send > tick the global headers again > press send > press next now
• Configure the dataset in the next step and you will get an error if the request needs the global headers to run
• Go back to the API request builder to find the global headers ticked but hiding the headers (acting like it's falsy)
• The unsync nightmare ensues

You can see in the screenshot below that the global headers are not showing

The errors format is currently all over the place. It will be good to have a module that handles all the errors to standardize a format across the whole API - controllers.

This issue will be updated with any discussion points.

Hi,

I've got this message when I try to initialise the database with the migrate command.

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'run', 'db:migrate' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'predb:migrate', 'db:migrate', 'postdb:migrate' ]
5 info lifecycle [email protected]predb:migrate: [email protected]
6 info lifecycle [email protected]db:migrate: [email protected]
7 verbose lifecycle [email protected]db:migrate: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]db:migrate: PATH: /usr/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/opt/chartbrew/server/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
9 verbose lifecycle [email protected]db:migrate: CWD: /opt/chartbrew/server
10 silly lifecycle [email protected]db:migrate: Args: [ '-c', 'cd models && npx sequelize-cli db:migrate' ]
11 silly lifecycle [email protected]db:migrate: Returned: code: 1 signal: null
12 info lifecycle [email protected]db:migrate: Failed to exec db:migrate script
13 verbose stack Error: [email protected] db:migrate: cd models && npx sequelize-cli db:migrate
13 verbose stack Exit status 1
13 verbose stack at EventEmitter. (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:332:16)
13 verbose stack at EventEmitter.emit (events.js:198:13)
13 verbose stack at ChildProcess. (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at ChildProcess.emit (events.js:198:13)
13 verbose stack at maybeClose (internal/child_process.js:982:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid [email protected]
15 verbose cwd /opt/chartbrew/server
16 verbose Linux 3.10.0-1062.4.3.el7.x86_64
17 verbose argv "/usr/bin/node" "/usr/bin/npm" "run" "db:migrate"
18 verbose node v10.22.0
19 verbose npm v6.14.6
20 error code ELIFECYCLE
21 error errno 1
22 error [email protected] db:migrate: cd models && npx sequelize-cli db:migrate
22 error Exit status 1
23 error Failed at the [email protected] db:migrate script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

Regards,

Steve

Currently, the SavedQueries have the Project foreign key and it's causing the Project deletion to fail.

Instead of fixing, maybe it's worth making the SavedQueries part of the Team instead of Project. That way the queries can be used across multiple dashboards.

Error when trying to delete a Project that has a SavedQuery:

{
	"name": "SequelizeForeignKeyConstraintError",
	"parent": {
		"code": "ER_ROW_IS_REFERENCED_2",
		"errno": 1451,
		"sqlState": "23000",
		"sqlMessage": "Cannot delete or update a parent row: a foreign key constraint fails (`chartbrew`.`SavedQuery`, CONSTRAINT `SavedQuery_ibfk_1` FOREIGN KEY (`project_id`) REFERENCES `Project` (`id`) ON DELETE NO ACTION ON UPDATE CASCADE)",
		"sql": "DELETE FROM `Project` WHERE `id` = '158'"
	},
	"original": {
		"code": "ER_ROW_IS_REFERENCED_2",
		"errno": 1451,
		"sqlState": "23000",
		"sqlMessage": "Cannot delete or update a parent row: a foreign key constraint fails (`chartbrew`.`SavedQuery`, CONSTRAINT `SavedQuery_ibfk_1` FOREIGN KEY (`project_id`) REFERENCES `Project` (`id`) ON DELETE NO ACTION ON UPDATE CASCADE)",
		"sql": "DELETE FROM `Project` WHERE `id` = '158'"
	},
	"sql": "DELETE FROM `Project` WHERE `id` = '158'",
	"fields": ["project_id"],
	"table": "Project",
	"index": "SavedQuery_ibfk_1",
	"reltype": "parent"
}

What's the issue

When a request is made using the API or Query builder the data is not used further and it causes the preview function to request the same data from the backend (and consequently from the connection) again.

How can this work better

It should follow these steps:

1. Make/build query or API request
2. Request the data from the connection
3. The user goes to the Build Chart step and configures the dataset
4. When the _onPreview function in the AddChart container fires, it reuses the data from step 2
5. The Back-end will need to be modified to accept this data without making a new request to the connection
6. When the user clicks on the Create or Update button to end the editing, the data should be reused again so that the loading doesn't take forever when dealing with big datasets

Would be nice to be able to change the order of the Datasets, especially on the KPI charts:

• New field in the Dataset model called order
• New sequelize migration to reflect the new schema changes
• Front end rendering based on the new order field
• Front end functionality to change the order of the datasets - ideally drag-n-drop the elements (top-right of the screenshot below)

• User presses Add Chart
• An empty chart draft is created in the database
• The User can always go back to edit the chart
• The draft charts are displayed differently in the dashboard?

Code progress in #24

The motivation

A lot of future tasks will be based on real-time communication through websockets:

• Collaborative chart creation
• Collaborative dashboard editing
• Support for streams of data (Database streams like on MongoDB for example)

Architecture

The socket.io will be a new module in the server-side code. It should have the following functionality:

• Register new events
• Put together a standard of how data is formatted
• Ability to call socket services from the rest of the application
• Support identifiers so that the broadcasts go to the right users

Chart.js is drawn on an HTML <canvas /> that should be straightforward to export to a PDF or any other image format.

The option to export to PDF should be included in the contextual menu of every chart (screenshot below). This shouldn't require any back-end work.

Thanks a lot for you great efforts and nice product
please look to http://clickhouse.yandex/docs/en/
ClickHouse is a blazing fast open source Columnar database for Analytics

could you add ClickHouse as data source in your product?

How to contribute to this issue:

• Scroll down below to check the existing PRs so you don't open a PR for a component waiting for review
• Refactor components based on the description below - also check the guidelines for PRs
• Reference this issue with #58 in the PR description so we can track which components are being worked on
• Open one PR per component

Examples of final results:

• Easy, clean example
• Extremely long and complicated example

The refactoring should address the following:

class components => functional components

class CoolComponent {
  constructor(props) {}
}

// to

function CoolComponent(props) {
  // deconstruct all the props here, at the beginning
  const { someProp, anotherOne } = props;
}

Lifecycle methods switched to useEffect

componentDidMount()

// to

useEffect(() => {}, []);

componentDidUnmount()

// to

useEffect(() => {
  return () => {
    // do stuff here when closing down the component
  }
}, []);

componentDidUpdate()

// to

useEffect(() => {}, [someProp]);
// someProp should be the prop or multiple props that the component should watch for

Internal this.state => useState

this.state = {
  loading: true,
};

// to

const [loading, setLoading] = useState(true);

All previous class methods should now be constant variables

class CoolComponent {
  _someMethod = () => {}
}

// to

function CoolComponent() {
  const _someMethod = () => {};
}

This could work in two different ways:

1. Have an option in the User dashboard to duplicate the project and everything will be duplicated

This approach might be tricky to implement because of the granularity involved: have to search for all the associated items to duplicate - connections, charts, saved queries, api requests, datasets

2. The option can be placed on individual items:

For example, a connection can have an option to be copied over to another project. The same with charts, but this entails the same issue identified in 1. where all associated items need to be duplicated as well

This should be applied to both client and server

When does it happen?

When the response object is in a format like below:

[
  {
    "url": "https://api.github.com/repos/octocat/Hello-World/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e",
    "commit": {
      "url": "https://api.github.com/repos/octocat/Hello-World/git/commits/6dcb09b5b57875f334f61aebed695e2e4193db5e",
      "author": { // selecting this field
        "name": "Monalisa Octocat",
        "email": "[email protected]",
        "date": "2011-04-14T16:00:49Z"
      },
      "committer": {
        "name": "Monalisa Octocat",
        "email": "[email protected]",
        "date": "2011-04-14T16:00:49Z"
      },
      "message": "Fix all the bugs",
      "tree": {
        "url": "https://api.github.com/repos/octocat/Hello-World/tree/6dcb09b5b57875f334f61aebed695e2e4193db5e",
        "sha": "6dcb09b5b57875f334f61aebed695e2e4193db5e"
      },
      "comment_count": 0,
      "verification": {
        "verified": false,
        "reason": "unsigned",
        "signature": null,
        "payload": null
      }
    },
    "author": { // confused by this field
      "login": "octocat",
      "id": 1,
      ....
    },
    ....

As can be seen in the object above, there are two properties called author; one in the root object and the other one in commit.author.

If the user wants to select commit.author.date as the date field to use for a time series, sometimes the dataFinder modules tries to find this field in the root author folder returning undefined.

For example in the screenshot below, I would like to name:

• null or empty with Not defined
• win32 to Windows
• darwin to MacOS.

The current active testing is done against MySQL and APIs Connections. It will be good to have at least one connection of each type to increase the quality of the tests before each release.

• MySQL
• API
• NodeJS
• PostgreSQL

What happens

The project and dashboard visually change, but the URL stays the same. This causes page refreshes to go back to the previous project. Some error artefacts like the connection not appearing on the charts happen as well.

Expected behaviour

The URL should change to point towards the new project ID.

There might be some Popup components and Responsive that need changing. https://react.semantic-ui.com/migration-guide

An issue to keep track of changes in the documentation.

How to replicate

• creating a new chart
• in the timeseries global settings turn Allow zeros on and off multiple times
• sometimes the server returns a 400
• I traced it back to the cache which causes that somehow

Currently, Chartbrew works only if the native auth is used. The mysql2 and sequelize libraries will need to be migrated to the latest version to make this work.

On both, client and server start there should be a check for env vars to make sure the compulsory variables are all set. Throw an error otherwise.

Steps to reproduce

• Create a new connection
• Edit a connection
• The top margin messes up
• Need to refresh the page to fix

The current models don't have migration rules and any database changes need to be done automatically. This is obviously not a good thing and needs fixing fast.

The ORM that CB uses is Sequelize. This is the documentation for migrations:

https://sequelize.org/master/manual/migrations.html

Steps to reproduce

• Start building a new chart
• Preview after configuring the first dataset
• Don't fiddle with Show Zeros options
• The preview acts like this options is false
• Create the chart
• Show zeros is set to true on the dashboard

Deprecation warning: crypto.createCipher is deprecated

simplecrypt is not receiving any other updates and it's using the deprecated API. Needs to be replaced with another library.

A good contender might be: bcrypt

⚠Things to take into account:

• All the encrypted content in the database might need to be migrated to the new encryption system.
• Existing users must be able to log in as usual after the move

Currently, react-ace uses brace which is huge for a front-end app. The latest version of react-ace uses their own syntax builds and can speed up the front-end app considerably (just a hypothesis).

The new version might have some breaking changes other than removing brace - make sure to test.

How to reproduce

• Start building a chart
• Make sure there are a few draft autosaves happening
• Configure a dataset before creating the chart
• Create chart
• Go to edit the chart => a lot of duplicate datasets

Screenshot

• Complete the connection form
• Click on Save Connection
• The button loads continuously and the form does not disappear
• The Connection is created correctly

This was observed in the API connection form

The database migration script is currently not running in the CI workflow.

Add a new task with npm run db:migrate

Normally, Chartbrew looks for the span of the dataset, and if it extends over multiple years, it adds the year to the X Label.

If Dataset#1 spans over multiple years the format will be -> YYYY MMM D
Dataset#n will then get the same format

The problem arises when:

Dataset#1 doesn't span over multiple years. The format will then be -> MMM D
Dataset#2 spans over multiple years, it will take the format of the 1st dataset and it will mess up the order of the dates

At the moment, any changes made when editing a live chart will be visible right away in all places that the chart is made available.

To improve the experience, the #28 PR disabled the autosave flag when editing a live chart to avoid premature edits.

A better experience would be:

• Click Edit on a live chart
• A draft gets created and it will record in the database which is the live chart that it represents
• The user can make any edits and all be saved in the draft
• When drafting is complete, the user can choose to save the new changes into the live chart
• The temporary drafts gets removed from the database

CB version v1.0.0-beta.1

When editing the Pattern Bar Chart, the colour changes just on the first bar, the others change only when saving pressing done. See SS below:

I selected the back colour and the bar is still blue.

Follow the design below

Completed? ❌
Code work done here: #24

https://www.figma.com/file/3Ywgwux25q5jpjOwgVt8R6/Chartbrew---AddChart-One-UI

What

This issue will track the design and implementation of a new style of interaction when creating a chart. Instead of the wizzard-type interaction implemented at the moment, the new interface will have all the elements present in one page:

On the same page, a user can do the following:

• Add the name of the chart
• Add new connections (multiple connections per chart < insert new issue here when created >)
• Access to connections configuration
• Select the type of visualisation

At the moment, checking the roles (if the current user can access the resource) requires a lot of repetitive code:

return projectController.findById(req.params.project_id)
      .then((project) => {
        return teamController.getTeamRole(project.team_id, req.user.id);
      })
      .then((teamRole) => {
        const permission = accessControl.can(teamRole.role).updateAny("chart");
        if (!permission.granted) {
          throw new Error(401);
        }
        // and only here everything is ready to continue
      })

This also makes the API routing a bit complicated as it's required to always include the project ID in the route.

BUT for starters, it will be good to make the check above shorter by modularising the access role check.

Hi,

when you want to use the Dockerfile it returns that "npm ERR! enoent ENOENT: no such file or directory, open '/code/package.json'"
And there is no code directory on the GitHub repository.

Regards,

Steve

There could be an option on the dashboard to select a date range and inject this value to all the charts present on the dashboard. This could be useful for monthly reports, for example.

I could be reading the setup wrong, but it appears the only way the Y axis value is determined is by the number of records returned in the results. It would be nice to set the value used for y axis in the query itself. For example, if we wanted to see number of users by role we would currently query like this:

SELECT role_name FROM users

It would be nice to offload this to the SQL like this:

SELECT role_name, count(*) as y_axis

This allows for more custom/advanced charts.

Is this already supported and I'm just missing it in the docs?

This depends on #18 to work properly.

The .watch() method documentation:
https://docs.mongodb.com/manual/reference/method/db.collection.watch/

Hi! After some ups and downs I have been able to configure chartbrew and make it run in my synology docker. The issue that I'm facing is that when I enter the frontend and try to sign-up as new user nothing happens! I mean, I complete name, mail and password but when I press the button nothing happens...

Thanks for your help

Some email providers are forcing OAuth2 authentication for SMTP (e.g. Gmail).

Documentation material for the implementation: https://nodemailer.com/smtp/oauth2/

This will be a bigger issue to track since it involves a lot of pages. So here's a checklist with all that needs to be addressed:

• User Dashboard (just addressed in a6f8160)
• Project Dashboard (side-menu should go at the bottom with most used items)
• The Navbar component (get rid of the project selector and move other items to the User Icon dropdown)
• Account settings page (side menu up top or at the bottom)
• Team management page (side menu up top or at the bottom)
• Chart builder page (more complicated, but lower priority since the mobile version will most likely be used for reporting only)

Compile a list with what's needed to update an instance of Chartbrew.

Also troubleshooting in case:

• The update command fails
• There are merge issues
• Things that can be used to solve merge issues
• Alternative update methods

Docker will limit these anonymous pulls, so they will need to be authenticated.

Instructions here: https://circleci.com/docs/2.0/private-images/

This is the error message:

ApiConnectionForm.js

  121 |   };
  122 |   this.setState({
  123 |     connection: {
> 124 |       ...connection, optionsArray: [...connection.optionsArray, option]
  125 |     },
  126 |   });
  127 | }

If a line has a draft and a completed chart the line looks like this:

The expected behaviour when hiding the draft is for CB Users to go to the left of the line and CB Charts take the place of CB Users.

Instead, this happens:

This feature will add events when certain data is detected in the Charts. For example, when the last data node in a chart hits a determined value, Chartbrew will send an email to the user.

Email events

The email event can be the default behavior. Let's say you might track the memory usage of your server and then the memory reaches a certain threshold. You will get an email when that happens.

Webhooks

The event configuration can have the option to configure webhook endpoints. Chartbrew will them make a POST request with a payload that contains:

• Value that triggered the event
• Chart configuration
• Which chart, which project
• Direct URL to the chart
• Direct URL to the dashboard
• Generated image of the chart? 🤔🤔🤔 (can then be automatically distributed to other people/services for display)

Request your own:

Comment below if you have any other suggestions

Steps to reproduce:

• Select any Line or Bar aggregating chart
• Pull some data (ideally a larger dataset)
• Expand date settings and choose the weekly time interval
• The value will be slightly off

Currently, the migrations are run with npm run db:migrate, but this is not so elegant. It will be great to run these automatically when starting the server application.

This should be fairly easy with this tool: https://github.com/sequelize/umzug

pm2 serve is for serving static sites and it will give a 404 if the user refreshes the page.

pm2 should instead use the external serve module to serve the site and for this it needs a configuration file like so:

{
  apps : [
    {
      name      : "chartbrew-client",
      script    : "npx",
      interpreter: "none",
      args: "serve -s . -p 5100"
    }
  ]
}