Giter VIP home page Giter VIP logo

qlik-repo-api's Introduction

Qlik Sense Repo API (Beta)

ko-fi

Node.js/browser package to interact with Qlik Sense Repository API (QSEoW)

NOT AFFILIATED WITH QLIK

Please check out the Wiki section for details and examples

Installation

npm install qlik-repo-api

Note Node version >= 16.0.0

Authentication

The package itself will NOT perform authentication. All authentication information is passed in the config (see next section).

But multiple authentication configuration can be provided:

  • certificates
  • JWT
  • header
  • session
  • ticket

Initialization

  • JWT

    const repoApi = new QlikRepoApi.client({
      host: "my-sense-server.com",
      proxy: "virtualProxyPrefix", // optional
      authentication: {
        token: "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
      },
    });
  • Certificates

    const https = require("https");
    const fs = require("fs");
    
    // read the certificate files
    const cert = fs.readFileSync(`/path/to/client.pem`);
    const key = fs.readFileSync(`/path/to/client_key.pem`);
    
    // create httpsAgent. Pass the certificate files and ignore any certificate errors (like self-signed certificates)
    const httpsAgentCert = new https.Agent({
      rejectUnauthorized: false,
      cert: cert,
      key: key,
    });
    
    // the actual initialization
    const repoApi = new QlikRepoApi.client({
      host: "my-sense-server.com",
      port: 4242, // optional. default 4242
      httpsAgent: httpsAgentCert,
      authentication: {
        user_dir: "USER-DIRECTORY",
        user_name: "userId",
      },
    });
  • Header

    const https = require("https");
    
    // httpsAgent (Node.js only) can be used with other authentication methods to ignore certificate errors (if any)
    const httpsAgentIgnoreSelfSigned = new https.Agent({
      rejectUnauthorized: false,
    });
    
    // the actual initialization
    const repoApi = new QlikRepoApi.client({
      host: "my-sense-server.com",
      proxy: "virtualProxyPrefix",
      httpsAgent: httpsAgentIgnoreSelfSigned,
      authentication: {
        header: "SomeHeader",
        user: "USER-DIRECTORY\\userId",
      },
    });

Browser usage

Although interacting with Qlik Repository API is mainly for automation and administration purposes (backend/server-to-server) this package can be used in browser environment as well.

There are couple of limitations in this case and they are both because of https package not being available in the browser. For this reason certificate config authentication can't be used and any certificate issues can't be ignored (rejectUnauthorized: false).

Methods

Full list of available methods can be found here

Generic clients

The package expose two extra (generic) methods. These methods are not "bound" to specific method/object (aka raw methods). These methods can be used in the cases where this package is not handling some specific endpoint. For them the url and body (for Post and Put methods) must be provided.

  • repoClient - client that uses /qrs as prefix. The required url should be passed without the /qrs prefix
  • genericClient - client that have no prefix. Useful for downloading temporary files (but not only)

General usage

The package expose few logical methods. For example: apps, streams, users etc. Each of these methods:

  • have get method that returns instance of the returned object
  • multiple methods that operates on multiple objects.

The get method will then have methods that are operating on the single returned object.

For example:

The apps method will expose the following methods: Apps methods

If we use the get method then the result variable will have additional methods: App methods

All these methods (copy, export, publish etc.) will be executed in the context of the app id provided in the get method (some-app-id)

details property will contain all the meta data for the app:

App details

The other methods (apart from get) might return array of object instances. For example apps.getFilter method will return an array of app instances:

const someQlikApps = await repoApi.apps.getFilter({
  filter: "name sw 'License Monitor'",
});

The someQlikApps variable will be an array of the App class. Each element of array will have details and the single app methods

Apps

Apps0

Usage examples

Update app

For a single app: change the name, add custom properties and tags

const someQlikApp = await repoApi.apps.get({ id: "some-app-id" });

console.log(someQlikApp.details.name);

const updateResponse = await someQlikApp.update({
  name: "new app name",
  tags: ["tag-name-1", "tag-name-2"],
  customProperties: [
    "customProperty1=value1",
    "customProperty1=value2",
    "customProperty2=some-value",
  ],
});

Update multiple apps

For a single app: change the name, add custom properties and tags

// all apps with their name is starting with "License"
const licenseMonitorApps = await repoApi.apps.getFilter({
  filter: "name sw 'License'",
});

// update each app by adding custom properties
const updateResults = await Promise.all(
  licenseMonitorApps.map((app) =>
    app.update({
      customProperties: [
        "customProperty1=value1",
        "customProperty1=value2",
        "customProperty2=some-value",
      ],
    })
  )
);

Once all apps are updated the updateResults variable will be:

[
  { id: "app-id-1", status: 200 },
  { id: "app-id-2", status: 200 },
  { id: "app-id-3", status: 200 }
  ...
]

As a "side effect" details for each app in licenseMonitorApps will also be updated in the variable and there is no need to call getFilter to retrieve the updated information.

Download files

When downloading content (mainly because of apps) the return data is in stream format (instead of buffer). This for performance reasons. If the apps are small then its not an issue to have them as buffer but some Qlik apps can be few GB in size. Because of this the data have to be streamed.

// get instance of the app
const app = await repoApi.apps.get({
  id: "1111111-2222-3333-4444-555555555555",
});

// export the app request
// at this point "stream" variable will have "file" property
// "file" property will be of type IncomingStream
const stream = await app.export();

// prepare the writer. it will write the incoming data to the provided path
const writeToFile = fs.createWriteStream("location/to/write/some-app.qvf");

// Start piping the chunks of incoming data to the stream writer
stream.file.pipe(writeToFile);

// we have to wait for all the data to be received and written
// for this reason we'll "listen" for the "end" event on the stream writer
await new Promise((resolve, reject) => {
  stream.file.on("end", () => {
    resolve("");
  });

  stream.file.on("error", () => {
    reject();
  });
});

// at this point the app is completely downloaded
let a = 1;

Upload files

For performance reasons uploading files should be made through streams.

const streamContent = fs.createReadStream("path/to/some-file.qvf");

const app = await repoApi.apps.upload({
  name: "Some app name",
  file: streamContent,
});

Stream data between servers

Corner case but it is possible to stream apps/content from one QS cluster to another (or to multiple).

Since both download and upload app operations are based on streams then we can start downloading an app and at the same time upload it to the destination cluster(s). This way the app/data stays in memory and there is no need to write the output to a local file and then upload it.

// repoApi - source QS cluster. From where the app will be exported
// repoApiDestination - destination QS cluster. Where the app will be imported


// get instance of the app
const sourceApp = await repoApi.apps.get({
  id: "1111111-2222-3333-4444-555555555555",
});

// export the app request
// at this point "stream" variable will have "file" property
// "file" property will be of type IncomingStream
const streamSourceApp = await sourceApp.export();

// push the stream (chunked) data to the upload method
// "file" property accepts Buffer or IncomingMessage or createReadStream
// and streamSourceApp.file is of type IncomingMessage
const destinationApp = await repoApiDestination.apps.upload({
  name: "some name", // or if we want the same name - sourceApp.details.name
  file: streamSourceApp.file,
});

//at this point the app is uploaded without being saved locally
let a = 1;

More examples to follow

qlik-repo-api's People

Contributors

countnazgul avatar dependabot[bot] avatar renovate-bot avatar renovate[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

qlik-repo-api's Issues

Phase 1 objects

Since the work on these is almost done, only one issue will be allocated for them

  • About
  • App
  • Content library
  • Custom property
  • Engine
  • Extension
  • Stream
  • System rule
  • Table
  • Tag
  • Task

(Update common) Tags append

Provide the option to append tags values. At the moment all existing values will be overritten with the provided values

Update methods to return the object

At the moment all update methods are returning only the update response status (usually 200). For conistency all these methids should return the updated object data as well

xxxGetAll methods

At the moment all xxxGet requests accept id as optional parameter. If id is NOT specified the method returns IxxxCondensed[] if not Ixxx[].
Dont like methods to return different data formats.

For this reason xxxGet should require id parameter as mandatory and return Ixxx (no array)
And new methods should be implemented xxxGetAll (without any params) which will return IxxxCondensed[]

Additional methods

Once v1 of the package is released, all test are written and all (known) bugs are resolved ... can think of:

  • expose additional methods. At v1 the package will contain only the "main" methods. The Repo API contains more methods that can be exposed.
  • extra methods can be added. These methods do not exists in the Repo API directly and will be based on combination of the "main" methods

Nested classes?

At the moment all the methods are called via repoApi.xxx. For example: repoApi.appGet() or repoApi.streamGet() etc.

Think it will be better to change this to: repoApi.App.Get(), repoApi.Stream.Get() etc

Is it possible at all?

Error messages

Since the codebase is changed all thrown error messages should be changed to reflect it

Allow reloadTask.create and reloadTask.upadate to accept app filter

Method name
reloadTask.create and reloadTask.upadate

Describe the issue
At the moment both methods accept only appId. If appFilter is provided the methods will search for app, based on the filter, and get its ID from the response. If the app filter returns more than one app or 0 app the methods will throw an error. If appId and appFilter are provided then appId is with priority

Code example
n/a

Possible alternatives?
n/a

Additional context
n/a

Do not return baseUrl and repoClient

At the moment each object method will return an extra data - baseUrl and repoClient values. These are not required and should be excluded. The return data should be only relevant to the object itself. baseUrl is informative but essentuially useles. repoClient also is irrelevant in the context of the object. repoClient instance is the one running the code and there is no need to be returned in each object instance

IRemoveFilter and IHttpReturnRemove

  • IHttpReturnRemove - the name should be different. It doesent make sense anymore
  • IRemoveFilter - this can be removed. Essentially it can be replaced by IHttpReturnRemove[]

Phase 4 checklist

  • tests for all phases are completed
  • no outstanding or known bugs/TODO/REVIEW

[BUG] Wrong function reference for Weekly trigger schema

Method name
reloadTask.addTriggerMany and reloadTask.addTriggerSchema

Describe the issue
When specifying Weekly repeat, the function that is "converting" the day names to day numbers is incorrectly referenced

Code example
n/a

Possible alternatives?
n/a

Additional context
n/a

Big code change

At the moment each object class expose its methods. For example:

  • Tag
    • create
    • get
    • remove
    • update
    • getFilter
    • ...

The new code change will (sometimes) return new instance of class and expose methods relevant to the object. For example:

  • Tags
    • remove - returns the data
    • removeFilter - returns data
    • get - returns new instance of the Tag class
    • create - returns new instance of the Tag class
    • getFilter - returns array of instances of the Tag class

The instance of the Tag class will have its own methods:

  • remove
  • update

So ... the old way code was:

const tag = await repoApi.tag.get("id-goes-here");
// tag variable will have the details of the tag: id, name, createdDate, etc
const createTag = await repoApi.tag.create("id-goes-here");
// createTag variable will have the details of the tag: id, name, createdDate, etc
const removeTag = await repoApi.tag.create("id-goes-here");
// removeTag variable will have the response details

the new way:

const tag = await repoApi.tag.get("id-goes-here");
// at this point the tag variable will expose the details of the tag and some methods
// tag.details - id, name, createdDate etc
// await tag.remove() - method that will remove the tag. no need to specify the id
// await tag.update("new tag name") - method to update the name of the tag. id is not needed

methods that are returning array of instances:

const allTags = await repoApi.tag.getAll();
// each element of the array exposes Tag class instance
// allTags[0].details
// await allTags[0].remove()
// await allTags[0].update("new tag name")

an example how to remove multiple tags:

// get all tags which name starts with "test"
const filterTags = await repoApi.tag.getFilter(`name sw 'test'`);
const removeResp = await Promise.all(
  return filterTags.map(tag => tag.remove())
);
// removeResp will be an array of remove responses:
// [ 
//    {id: "some-tag-id", status: 204},
//    {id: "some-other-tag-id", status: 204},
//    ...
// ]

The change will not affect some objects. Since not all objects are required to return class instance. Example objects/endppoints About and ServiceStatus

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.