asyncapi / optimizer Goto Github PK

Reason/Context

This issue has been created to track the progress of the v1 release of the optimizer.

• #168
• #198
• #194
• #192

Build a library that code-first users would benefit from the most but not only. A library that as the input gets AsyncAPI document and as output provides optimized AsyncAPI document. For example:

• Reuse of components
• Duplicated schemas moved to components and ref-ed from a message
• Remove not used components

The goal is to shrink the document to a minimum.

More requirements:

• There should be flags to disable specific parts of optimization, like removal of not used components for example.
• The structure of the library should make it easy to add more optimizations that are not listed above.
• I should be able to just get an optimization report, without actually changes, something like npm publish --dry-run that just indicates how the package will look like after publishing, without actually publishing it.

Non-functional requirements:

• can be JS or TS, whatever feels better for you
• must be a library that later I can integrate in the UI or CLI. No file system operations.
• As input, you get an AsyncAPI document parsed with https://github.com/asyncapi/parser-js/. As output, you either get a modified AsyncAPI document or/and just a report of possible optimizations and where they can be done inside the document

Use case:
AsyncAPI offers many different ways to reuse certain parts of the document like messages or schemas definitions or references to external files, not to even mention the traits. There is a need for a tool that can be plugged into any workflows and optimize automatically documents that are generated from code.

Dummy issue.

Reason/Context

When I look at the output of the optimizer, it is more complicated than it should be. And that is what to be expected because we replace lots of schemas and components with $refs and that makes the file a little bit harder to grasp by humans. But how can we prevent the output from being over-complicated?

The problem

If we set the bar too low like: all schemas should be optimized, then the library will extract an unnecessary amount of schemas.
for example: if I have multiple instances of {type: string} then I want the optimizer to ignore it since by optimizing it, we make it complicated and hard to understand, without any gains.
I think we need to have some way of distinguishing the "optimization worthy" schemas.

Current Implementaton

After we merge #85, only the schemas that contain a type field will be considered for optimization. but that is not the perfect solution since not all schemas have a 'type' field.

Proposal

I was thinking about it and I couldn't find a perfect way to define a schema worth optimizing.
the best that I could come up with is:

a schema should be optimized if:

• it has more than one non-object or non-array field
  OR
• it has at least one object or array field.

open to suggestions.

The library should be able to support messageTraits optimization. currently, it only supports messages, parameters and schema optimization.

After we have #172 we need to think about what things will change in the implementation of the optimiser to support v3 of the spec.

Related to #168

This issue defines a list of tasks that need to be performed in this repo to make sure it's ci/cd automation works long term without any issues.

It is up to maintainers to decide if it must be addressed in one or multiple PRs.

Below are 3 different sections describing 3 different important ci/cd changes.

IMPORTANT-START
For GitHub workflows that contain This workflow is centrally managed in https://github.com/asyncapi/.github/ you do not have to perform any work. These workflows were already updated through the update in .github. The only exception is the workflows related to nodejs release. More details in Upgrade Release pipeline - in case of nodejs projects section
IMPORTANT-END

Deprecation of way data is shared between steps

Every single GitHub Action workflow that has echo "::set-output name={name}::{value}" need to be updated to follow echo "{name}={value}" >> $GITHUB_OUTPUT

We do not yet know when set-output will stop working. Previous disable date was 31.05 but now then say community needs more time.

For more details read official article from GitHub

Deprecation of node12

2nd bullet point is still relevant for you even if your projects in not nodejs project

• Every single workflow that uses setup-node action needs an update to follow v3 version of this action, and make sure minimum node 14 is used
• Now this part is more complex. Problem with node12 is that node-based GitHub Actions were using it in majority as a runtime environment. Look for example at this action.yaml file for setup-node action v2. So the job that you have to do is go through all the workflows, and verify every single action that you use, make sure you are using the latest version that is not based on node12. I already did review a lot of actions as part of this PR so maybe you will find some actions there and can copy from me. For example action/checkout needs to be updated to v3.

Node12 end of support in action is probably September 27th.

For more details read official article from GitHub

Upgrade Release pipeline - in case of nodejs projects

ignore this section if your project is not nodejs project

You have 2 options. You can:

A. choose to switch to new release pipeline using instruction from asyncapi/.github#205

B. stay with old release pipeline, and manually update GitHub workflows and actions used in it, you can inspire a lot from this PR asyncapi/.github#226

I definitely recommend going with A

Workflows related to release:

• .github/workflows/if-nodejs-release.yml
• .github/workflows/if-nodejs-version-bump.yml
• .github/workflows/bump.yml

Describe the bug

we are using the Options interface to define what options are available to be passed to getOptimizedDocument(options: Options). jsdoc2md does not appear to extract Options info from the interface and add them to API.md.

possible solution

adding jsdoc definition for Options in Optimizer class itself.

the following rules should be added to the eslint.

1. blank line before return unless it is the first line of the block.
2. blank line before and after if unless it is the first line of the block.
3. blank line before & after a set of const unless it is the first line of the block.
4. Whitespace after , when passing parameters or defining arrays

The library currently only supports YAML string files for input and output. It should be able to support JSON and JSON strings as well.

Describe the bug

The entirety of the ./lib/** folder is not published to NPM. This should contain the compiled .js and .d.ts files.

These file are referenced in the package.json:

  "main": "./lib/index.js",
  "types": "./lib/index.d.ts",

Because of this, Optimizer can't be used right now as both require and import don't work.

How to Reproduce

λ npm i @asyncapi/optimizer
...
λ node
Welcome to Node.js v16.3.0.
Type ".help" for more information.
> const { Optimizer } = require('@asyncapi/optimizer')
Uncaught:
Error: Cannot find module 'C:\Data\scratch\node_modules\@asyncapi\optimizer\lib\index.js'. Please verify that the package.json has a valid "main" entry
    at tryPackage (node:internal/modules/cjs/loader:349:19)
    at Function.Module._findPath (node:internal/modules/cjs/loader:562:18)
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:927:27)
    at Function.Module._load (node:internal/modules/cjs/loader:774:27)
    at Module.require (node:internal/modules/cjs/loader:1013:19)
    at require (node:internal/modules/cjs/helpers:93:18) {
  code: 'MODULE_NOT_FOUND',
  path: 'C:\\Data\\scratch\\node_modules\\@asyncapi\\optimizer\\package.json',
  requestPath: '@asyncapi/optimizer'
}

Same thing for import.

See also the file tree in the node_modules:

Expected behavior

Import and require work

Reason/Context

This library assumes that there is no way to disable 'traits applying' on parser but there is a flag to disable it. By disabling the applyTraits the scanning of the traits become easier and the codebase becomes simpler.

Description

First of all, we have to set applyTraits flag to false when parsing the AsyncAPI specification file.
After disabling it, the ComponentsProvider class becomes much simpler, because we don't have to figure out if the field is coming from traits or it was in the field itself. we can scan them separately.

It should be as easy as bumping the parser version and resolving the issues that comes along. we are not going to add anything extra. we just want to keep the same functionality but with v2 of parser.

Related to #168

Hi together,

I observe a strange behaviour of running the cli on Ubuntu 22 and @asyncapi/cli/0.50.0 linux-x64 node-v18.16.1.

First I bundle the main YAML with references from external local files where every schema and message is described in our system. The bundle works like we expect it and validate shows a lot of unused references, which we expect to be removed by the optimize command. But it somehow removes a lot more than expected.

The structure after bundle looks like following:

asyncapi: 2.6.0
...
channels:
  topic:
    description: ...
    parameters:
      clientId: ...
    subscribe:
      summary: ...
      operationId: ...
      message:
        oneOf:
          - $ref: '#/components/messages/message1'
          - $ref: '#/components/messages/message2'
          - $ref: '#/components/messages/message3'
components:
  messages:
      message1:
        messageId: message1
        name: message1
        title: ...
        summary: ...
        contentType: application/json
        payload:
          $ref: '#/components/schemas/message1'
      message2:
        messageId: message2
        name: message2
        title: ...
        summary: ...
        contentType: application/json
        payload:
          $ref: '#/components/schemas/message2'
      message3:
        messageId: message3
        name: message3
        title: ...
        summary: ...
        contentType: application/json
        payload:
          $ref: '#/components/schemas/message3'
  schemas:
    ...

The optimize now also removes every message in components/messages except for the last one in the "oneOf" in channels, even if the references are there and used (and validate didnt detect unused references there). If I change the order of the message refs in channels, I can reproduce this and always the last message is available after optimize. Did someone also observe this behaviour or am I misunderstanding the optimize command?

Thanks in advance, David

Reason/Context

This is to replace the old AsyncAPI logo in this repo's README with the banner attached below that represents the new branding.

Here are a few guidelines for this change as well:

1. Make sure you are using Markdown syntax only
2. Be sure to remove the old logo as well as the old title of the repo as this image will replace both elements
3. Make sure you link this image to the website: https://www.asyncapi.com
4. If there is any description text below the repo title, let's make it left-aligned if it isn't already, so as to match the left-alignment of the content in the new banner image

Download the image file:
github-repobanner-optimizer.png.zip

Banner preview

Please note that this is only a preview of the image, the contributor should download and use the above zip file

Problem

Our library is not yet compatible with AsyncAPI v3. This means that users of our library who want to use AsyncAPI v3 will be unable to do so.

Solution

We will make the necessary updates to our parser and implementation to support AsyncAPI v3. This involves three main tasks:

• #173
• #172
• #192

Once asyncapi/parser-js#780 is solved, update parser dependency (might be automated)

Rabbit Holes

Potential challenges could include unforeseen issues with the parser update.

Out of bounds

We will not be adding any new features or making any other major changes to the library as part of this bet.

Describe the bug.

The example script uses relative paths (./examples/input.yaml and ./examples/output.yaml) to handle file reading and writing operations, it will fail if executed from different directories.

Expected behavior

The script should manage file paths using the __dirname variable to ensure consistent behaviour irrespective of the current execution directory. This will ensure correct file referencing at all times.

Screenshots

How to Reproduce

1. clone the project.
2. compile it with npm run dev
3. run cd examples then node index.js

🥦 Browser

Other

👀 Have you checked for similar open issues?

• I checked and didn't find similar issue

🏢 Have you read the Contributing Guidelines?

• I have read the Contributing Guidelines

Are you willing to work on this issue ?

No, someone else can work on it

Describe the bug

semantic-release package is missing. Plugins and config is there, but the core package is not.

Expected behavior

just install it 😄

API is currently written manually. use jsdoc2md to generate the API automatically.

Reason/Context

Since we have a place for different bindings in the components section like serverBindings, channelBindings,operationBindings and messageBindings, we can use those spaces to optimize different bindings.

Clone of asyncapi/bundler#141 (comment)

When working on asyncapi/bundler#141

I tried the modified Optimizer (moveAllToComponents: true) on

asyncapi: 3.0.0
info:
  title: Comments Service
  version: 1.0.0
  description: This service is in charge of processing all the events related to comments.
servers:
  mosquitto:
    host: test.mosquitto.org
    protocol: mqtt
    tags:
      - name: env:production
        description: This environment is meant for production use case
      - name: kind:remote
        description: This server is a remote server. Not exposed by the application
      - name: visibility:public
        description: This resource is public and available to everyone
    bindings:
      mqtt:
        clientId: comment-service
channels:
  commentLiked:
    address: comment/liked
    messages:
      commentLiked:
        x-origin: ./messages.yaml#/commentLiked
        description: Message that is being sent when a comment has been liked by someone.
        payload:
          x-origin: ./schemas.yaml#/commentLikedPayload
          type: object
          title: commentLikedPayload
          additionalProperties: false
          properties:
            commentId:
              allOf:
                - type: string
                - description: Id of the comment that was liked
    description: >-
      Updates the likes count in the database and sends the new count to the
      broker.
  commentCountChange:
    address: comment/{commentId}/changed
    messages:
      commentChanged:
        x-origin: ./messages.yaml#/commentChanged
        description: Message that is being sent when a comment have been updated.
        payload:
          x-origin: ./schemas.yaml#/commentChangedPayload
          type: object
          title: commentChangedPayload
          additionalProperties: false
          properties:
            commentId:
              allOf:
                - type: string
                - description: >-
                    Id of the comment that was changed, such as when someone
                    liked it.
            likeCount:
              type: integer
              description: The new like count of how many have liked the comment.
    description: >-
      Sends the new count to the broker after it has been updated in the
      database.
    parameters:
      commentId:
        x-origin: ./parameters.yaml#/commentId
        description: ID of the comment
operations:
  receiveCommentLiked:
    action: receive
    channel:
      $ref: '#/channels/commentLiked'
    messages:
      - $ref: '#/channels/commentLiked/messages/commentLiked'
  sendCommentChange:
    action: send
    channel:
      $ref: '#/channels/commentCountChange'
    messages:
      - $ref: '#/channels/commentCountChange/messages/commentChanged'

and received

asyncapi: 3.0.0
info:
  title: Comments Service
  version: 1.0.0
  description: This service is in charge of processing all the events related to comments.
servers:
  mosquitto:
    $ref: '#/components/servers/mosquitto'
channels:
  commentLiked:
    $ref: '#/components/channels/commentLiked'
  commentCountChange:
    $ref: '#/components/channels/commentCountChange'
operations:
  receiveCommentLiked:
    $ref: '#/components/operations/receiveCommentLiked'
  sendCommentChange:
    $ref: '#/components/operations/sendCommentChange'
components:
  schemas:
    commentCountChange:
      x-origin: ./parameters.yaml#/commentId
      description: ID of the comment
    commentLiked:
      allOf:
        - $ref: '#/components/schemas/commentLiked'
        - $ref: '#/components/schemas/commentLiked'
    commentChangedPayload:
      x-origin: ./schemas.yaml#/commentChangedPayload
      type: object
      title: commentChangedPayload
      additionalProperties: false
      properties:
        commentId:
          $ref: '#/components/schemas/commentCountChange'
        likeCount:
          $ref: '#/components/schemas/commentCountChange'
    commentLikedPayload:
      x-origin: ./schemas.yaml#/commentLikedPayload
      type: object
      title: commentLikedPayload
      additionalProperties: false
      properties:
        commentId:
          $ref: '#/components/schemas/commentLiked'
  messages:
    commentChanged:
      x-origin: ./messages.yaml#/commentChanged
      description: Message that is being sent when a comment have been updated.
      payload:
        $ref: '#/components/schemas/commentChangedPayload'
    commentLiked:
      x-origin: ./messages.yaml#/commentLiked
      description: Message that is being sent when a comment has been liked by someone.
      payload:
        $ref: '#/components/schemas/commentLikedPayload'
  operations:
    receiveCommentLiked:
      action: receive
      channel:
        $ref: '#/channels/commentLiked'
      messages:
        - $ref: '#/channels/commentLiked/messages/commentLiked'
    sendCommentChange:
      action: send
      channel:
        $ref: '#/channels/commentCountChange'
      messages:
        - $ref: '#/channels/commentCountChange/messages/commentChanged'
  channels:
    commentCountChange:
      address: comment/{commentId}/changed
      messages:
        commentChanged:
          $ref: '#/components/messages/commentChanged'
      description: >-
        Sends the new count to the broker after it has been updated in the
        database.
      parameters:
        commentId:
          $ref: '#/components/schemas/commentCountChange'
    commentLiked:
      address: comment/liked
      messages:
        commentLiked:
          $ref: '#/components/messages/commentLiked'
      description: >-
        Updates the likes count in the database and sends the new count to the
        broker.
  servers:
    mosquitto:
      host: test.mosquitto.org
      protocol: mqtt
      tags:
        - name: env:production
          description: This environment is meant for production use case
        - name: kind:remote
          description: This server is a remote server. Not exposed by the application
        - name: visibility:public
          description: This resource is public and available to everyone
      bindings:
        mqtt:
          clientId: comment-service

then I turned off schemas, received

asyncapi: 3.0.0
info:
  title: Comments Service
  version: 1.0.0
  description: This service is in charge of processing all the events related to comments.
servers:
  mosquitto:
    $ref: '#/components/servers/mosquitto'
channels:
  commentLiked:
    $ref: '#/components/channels/commentLiked'
  commentCountChange:
    $ref: '#/components/channels/commentCountChange'
operations:
  receiveCommentLiked:
    $ref: '#/components/operations/receiveCommentLiked'
  sendCommentChange:
    $ref: '#/components/operations/sendCommentChange'
components:
  messages:
    commentChanged:
      x-origin: ./messages.yaml#/commentChanged
      description: Message that is being sent when a comment have been updated.
      payload:
        x-origin: ./schemas.yaml#/commentChangedPayload
        type: object
        title: commentChangedPayload
        additionalProperties: false
        properties:
          commentId:
            allOf:
              - type: string
              - description: >-
                  Id of the comment that was changed, such as when someone liked
                  it.
          likeCount:
            type: integer
            description: The new like count of how many have liked the comment.
    commentLiked:
      x-origin: ./messages.yaml#/commentLiked
      description: Message that is being sent when a comment has been liked by someone.
      payload:
        x-origin: ./schemas.yaml#/commentLikedPayload
        type: object
        title: commentLikedPayload
        additionalProperties: false
        properties:
          commentId:
            allOf:
              - type: string
              - description: Id of the comment that was liked
  operations:
    receiveCommentLiked:
      action: receive
      channel:
        $ref: '#/channels/commentLiked'
      messages:
        - $ref: '#/channels/commentLiked/messages/commentLiked'
    sendCommentChange:
      action: send
      channel:
        $ref: '#/channels/commentCountChange'
      messages:
        - $ref: '#/channels/commentCountChange/messages/commentChanged'
  channels:
    commentCountChange:
      address: comment/{commentId}/changed
      messages:
        commentChanged:
          $ref: '#/components/messages/commentChanged'
      description: >-
        Sends the new count to the broker after it has been updated in the
        database.
      parameters:
        commentId:
          x-origin: ./parameters.yaml#/commentId
          description: ID of the comment
    commentLiked:
      address: comment/liked
      messages:
        commentLiked:
          $ref: '#/components/messages/commentLiked'
      description: >-
        Updates the likes count in the database and sends the new count to the
        broker.
  servers:
    mosquitto:
      host: test.mosquitto.org
      protocol: mqtt
      tags:
        - name: env:production
          description: This environment is meant for production use case
        - name: kind:remote
          description: This server is a remote server. Not exposed by the application
        - name: visibility:public
          description: This resource is public and available to everyone
      bindings:
        mqtt:
          clientId: comment-service

and I have a feeling that users would like the latter more.

Thus, should schemas be turned off by default, and users would need to specifically ENABLE them, for example, with the command-line switch --include-schemas?
It may be the opposite way around, but I believe that the possibility of changing the appearance/invisibility of schemas should definitely exist.

Reason/Context

Fortunately, the project is initialized and we can discuss the API of this library here.

Description

Here is a draft of the API as well as the proposed Report structure.
After finalizing the API and report structure, I will open a pull request with these changes in the README.md file.

Usage

Node.js

import { Optimizer } from '@asyncapi/optimizer';
import { parse } from '@asyncapi/parser';
let asyncApiDocument = parse(`
asyncapi: 2.0.0
info:
  title: Streetlights API
  version: '1.0.0'

channels:

  smartylighting/event/{streetlightId}/lighting/measured:
    parameters:
      #this parameter is duplicated. it can be moved to components and ref-ed from here.
      streetlightId:
        schema:
          type: string
    subscribe:
      operationId: receiveLightMeasurement
      traits:
        - bindings:
            kafka:
              clientId: my-app-id
      message:
        name: lightMeasured
        title: Light measured
        contentType: application/json
        traits:
          - headers:
              type: object
              properties:
                my-app-header:
                  type: integer
                  minimum: 0
                  maximum: 100
        payload:
          type: object
          properties:
            lumens:
              type: integer
              minimum: 0
            #full form is used, we can ref it to: #/components/schemas/sentAt
            sentAt:
              type: string
              format: date-time

  smartylighting/action/{streetlightId}/turn/on:
    parameters:
      streetlightId:
        schema:
          type: string
    publish:
      operationId: turnOn
      traits:
        - bindings:
            kafka:
              clientId: my-app-id
      message:
        name: turnOnOff
        title: Turn on/off
        traits:
          - headers:
              type: object
              properties:
                my-app-header:
                  type: integer
                  minimum: 0
                  maximum: 100
        payload:
          type: object
          properties:
            sentAt:
              $ref: "#/components/schemas/sentAt"

components:
  messages:
    #libarary should be able to find and delete this message because it is not used anywhere.
    unusedMessage:
      name: unusedMessage
      title: This message is not used in any channel.
      
  schemas:
    #this schema is ref-ed in one channel and used full form in another. the library should be able to identify and ref the second channel as well.
    sentAt:
      type: string
      format: date-time`
);
let optimizer = new Optimizer(asyncApiDocument);
let report: OptimizerReport = await optimizer.getReport();
/*
the `report` value will be:
{
  ReuseComponents: [
    {
      path: '#/channels/channel1/smartylighting/event/{streetlightId}/lighting/measured/message/payload/properties/sentAt',
      action: 'ref',
      refTo: '#/components/schemas/sentAt'
    }
  ],
  RemoveComponents: [
    {
      path: '#/components/messages/unusedMessage',
      action: 'remove',
    }
  ],
  MoveToComponents: [
    {
      //move will ref the current path to the moved component as well.
      path: '#/channels/smartylighting/event/{streetlightId}/lighting/measured/parameters/streetlightId',
      action: 'move',
      moveTo: '#/components/parameters/streetlightId'
    },
    {
      path: '#/channels/smartylighting/action/{streetlightId}/turn/on/parameters/streetlightId',
      action: 'ref',
      refTo: '#/components/parameters/streetlightId'
    }
  ]
}
 */
let optimizedDocument = optimizer.getOptimizedDocument({rules: {ReuseComponents: true,RemoveComponents: true,MoveToComponents: true }})
/*
the `optimizedDocument` value will be:

asyncapi: 2.0.0
info:
  title: Streetlights API
  version: '1.0.0'

channels:

  smartylighting/event/{streetlightId}/lighting/measured:
    parameters:
      streetlightId:
        $ref: "#/components/schemas/streetlightId"
    subscribe:
      operationId: receiveLightMeasurement
      traits:
        - bindings:
            kafka:
              clientId: my-app-id
      message:
        name: lightMeasured
        title: Light measured
        contentType: application/json
        traits:
          - headers:
              type: object
              properties:
                my-app-header:
                  type: integer
                  minimum: 0
                  maximum: 100
        payload:
          type: object
          properties:
            lumens:
              type: integer
              minimum: 0
            #full form is used, we can ref it to: #/components/schemas/sentAt
            sentAt:
              $ref: "#/components/schemas/sentAt"

  smartylighting/action/{streetlightId}/turn/on:
    parameters:
      streetlightId:
        $ref: "#/components/schemas/streetlightId"
    publish:
      operationId: turnOn
      traits:
        - bindings:
            kafka:
              clientId: my-app-id
      message:
        name: turnOnOff
        title: Turn on/off
        traits:
          - headers:
              type: object
              properties:
                my-app-header:
                  type: integer
                  minimum: 0
                  maximum: 100
        payload:
          type: object
          properties:
            sentAt:
              $ref: "#/components/schemas/sentAt"

components:
  parameters:
    streetlightId:
      schema:
      type: string
  schemas:
    #this schema is ref-ed in one channel and used full form in another. library should be able to identify and ref the second channel as well.
    sentAt:
      type: string
      format: date-time`
 */

API

Constructor

new Optimizer(document)

document is a mandatory object that is parsed with @asyncapi/parser:

Methods

getReport() : OptimizerReport

getOptimizedDocument([options]) : string

• options: this is an object that contains all of the options that we can pass in getOptimizedDocument. right now we have only rules here but maybe in the future we can have other options as well.
  • rules: using rules object different optimization types can be enabled or disabled.
    • ReuseComponents if set to true, optimizer will apply all of ReuseComponents changes from report. (default: true)
    • RemoveComponents if set to true, optimizer will apply all of RemoveComponents changes from report. (default: false)
    • MoveToComponents if set to true, optimizer will apply all of MoveToComponents changes from report. (default: true)