asyncapi / converter-js Goto Github PK

It's a long term feature request.

When we will have the 3.0.0 specification stabilized we should start writing functionality for converting versions 2.X.X to 3.0.0.

One of the problems I see are the external sources (local files and http links) which we are not converting at the moment but we should (see issue #90). In order not to cause problems for people and increase DX we should figure out a way to convert also external files in which messages (and other reusable components) can be defined.

Scope

• #188
• #90
• #106
• #107

This issue is part of a longer vision for the converter that can be found here: #221

It would be great to open up how easily people can switch or try out AsyncAPI based on existing resources. One of these is the Postman Collection, which defines a bunch of HTTP requests in a grouping called collections that contain items (requests) and folders.

I think we can combine this issue with #233, so when that functionality is solved, we can use postman2openapi or postman-to-openapi to first convert it to OpenAPI 3.0, and then use the existing converter to convert it to AsyncAPI.

NOTICE, because of the difficulty of this issue, you will be able to contact me all the time through Slack if you get stuck solving it at any point and even have calls when relevant.

Reason/Context

We need to support version 2.1.0 of the spec.

Description

Changes can be found here: asyncapi/spec#536

Example reference in single quotes prior v3 -> https://github.com/asyncapi/spec/blob/master/examples/social-media/backend/asyncapi.yaml#L9

After conversion it becomes ../common/servers.yaml#/websiteWebSocketServer without quotes

Problem is that some editors, like Studio for example, interprets #/websiteWebSocketServer as a YAML comment

cc @magicmatatjahu @jonaslagoni

Context

These two if statements have duplicated part for handling oneOf

Description

Move this part to separate function in helpers.js and of course use it instead of duplicating code.

Describe the bug

When you provide the following AsyncAPI in JSON format:

{
    "asyncapi": "2.2.0",
    "info": {
        "title": "Rust processor",
        "version": "0.2.0"
    },
    "defaultContentType": "json",
    "channels": {
        "test": {
            "publish": {
                "message": {
                    "payload": true
                }
            }
        }
    }
}

It returns the following when running asyncapi-converter test.json > test2.json

asyncapi: 2.3.0
info:
  title: Rust processor
  version: 0.2.0
defaultContentType: json
channels:
  test:
    publish:
      message:
        payload: true

Expected behavior

Expected it to return JSON.

as described in https://github.com/asyncapi/converter-js#conversion-2xx-to-3xx

If the operation does not have an operationId field defined, the unique identifier of the operation will be defined as a combination of the identifier of the channel on which the operation was defined + the type of operation, publish or subscribe. Identical situation is with messages. However, here the priority is the messageId field and then the concatenation {publish|subscribe}.messages.{optional index of oneOf messages}.

Why we do not take into account name of the message, because it can be duplicated across operations (not unique like messageId)? If we see duplication, we can have the same approach as described above, with index.

Or am I something missing

Currently in case of this 2.x example:

channels:
  smartylighting/streetlights/1/0/event/{streetlightId}/lighting/measured:
    parameters:
      streetlightId:
        $ref: '#/components/parameters/streetlightId'
    publish:
      summary: Inform about environmental lighting conditions of a particular streetlight.
      operationId: receiveLightMeasurement
      message:
        $ref: '#/components/messages/lightMeasured'

components:
  messages:
    lightMeasured:
      name: lightMeasured
      title: Light measured
      summary: Inform about environmental lighting conditions of a particular streetlight.

resulting message name in a new v3 channel is:

messages:
      receiveLightMeasurement.message:
        $ref: '#/components/messages/lightMeasured'

and better would be

messages:
      lightMeasured:
        $ref: '#/components/messages/lightMeasured'

or in some cases:

messages:
      lightMeasured.1:
        $ref: '#/components/messages/lightMeasured'

cc @jonaslagoni @magicmatatjahu

Reason/Context

It would be nice to show how to use the library in different workflows.

For example, how to incorporate it into a GH action workflow.

Example usage:

name: Convert the AsyncAPI document
env:
  GH_USER: jonaslagoni
  GH_EMAIL: <[email protected]>
on:
  workflow_dispatch: 
jobs:
  convert:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: '14'
      - name: Install converter
        run: npm i -g @asyncapi/converter 
      - name: Convert AsyncAPI file
        run: asyncapi-converter ./asyncapi.json > ./asyncapi-temp.json && mv ./asyncapi-temp.json ./asyncapi.json
      - name: Create Pull Request with updated document
        uses: peter-evans/create-pull-request@v3
        with:
          token: '${{ secrets.GH_TOKEN }}'
          commit-message: 'feat: convert to newest AsyncAPI version'
          committer: '${{env.GH_USER}} ${{env.GH_EMAIL}}'
          author: '${{env.GH_USER}} ${{env.GH_EMAIL}}'
          title: 'feat: convert to newest AsyncAPI version'
          body: Update to newest AsyncAPI version
          branch: 'convert/asyncapi_file'

So here we have securituSchemes.supportedOauthFlows -> https://github.com/asyncapi/spec/blob/next-major-spec/examples/correlation-id.yml#L150 but it was already inlined in https://github.com/asyncapi/spec/blob/next-major-spec/examples/correlation-id.yml#L24

so we kinda duplicate info, imho securituSchemes.supportedOauthFlows should be removed by the converter.

also, why Security requirements that use scopes are defined in the appropriate places inline?

cc @magicmatatjahu @jonaslagoni

In most of our projects we use jest as a testing framework. In this project (which was one of the first) we use mocha. It is not a bad choice but we should be consistent. This should be done after doing this #106

Reason/Context

Parser-JS requires now Node.js 14 and npm7 or higher.
This means package-lock.json version should be always set to 2.

Description

In order to unify development requirements, I suggest all repositories to stick with that requirements. In order to do that, the following is needed:

• Clarify this requirement under Development section on this repository README file.
• To update package-lock.json to version 2. This is automatically done when running npm install if the npm version used is 7 or higher.

Related:

• asyncapi/.github#177

Description

Hello folks,

I'm working on a project where we use converter-js, and our monitoring tools caught that there are some vulnerabilities in Mocha, up until the latest major release.
I've checked the breaking changes and it looks like it won't be a problem to bump the version directly from ^6.1.4 to ^9.1.3 (the tests are passing locally as well).
The only concerning thing seems to be the drop of Node.js v10.x support, however the vulnerability is not fully addressed in the latest v8 release.

Details

The particular vulnerability is CVE-2021-3807, and it's part of ansi-regex, which is a transitive dependency of mocha. Here are the dependency graphs:

Context

1. We want to make sure we have healthy community and people respec each other in GitHub. We want to monitor all comments in every repository and analyze sentiments to identify negative ones and get proper notification. Idea is to use GitHub Actions for it, to check sentiments with Google Natural Language API and post negative comments to Slack for review.

2. We manage community files in global .github repo, and we have GitHub Funding config file there too. As a result we need to cleanup all the repos to remove .github/FUNDING.yml

3. Start supporting new GitHub pull request event that allows gives access to GITHUB_TOKEN with write access. Fix welcome contributors workflow is needed.

4. We want to automatically merge Pull Requests that are submitted by Dependanbot. New workflow needs to be added, or if already existins it needs to be modified

Description

1. Add this workflow file called sentiment-analysis.yml to .github/workflows (create it if it doesn't exist yet) directory: https://gist.github.com/derberg/ab362e4b37f542e7e1813e67b7cb11ee
   Proper secrets are already added to GitHub organization so it is as simple as adding above file to workflows dir.
2. Remove .github/FUNDING.yml if it exists
3. If not done already rename pull_request from line 4 to pull_request_target in .github/workflows/welcome-first-time-contrib.yml file
4. Create new file called automerge.yml with the following content: https://gist.github.com/derberg/024814a26959d54f683e7bd68d68f007
   If the repository already has a file called automerge-release-pr-bump.yml than rename it and adjust if statements to check github.actor == 'asyncapi-bot' || github.actor == 'dependabot[bot]' || github.actor == 'dependabot-preview[bot]' instead of github.actor == 'asyncapi-bot'

This issue is part of a longer vision for the converter that can be found here: #221

It would be great to open up how easily people can switch or try out AsyncAPI based on existing resources. One of these is OpenAPI 3.0: https://spec.openapis.org/oas/v3.0.3. With the new AsyncAPI v3, we should be able to map this 100% from OpenAPI to AsyncAPI.

In order to solve this it requires a lot of conversion that might not be straightforward. While this issue is tagged as a good first issue it requires a lot of effort to push through, on the bright side, solving this issue will help you gain knowledge about OpenAPI and AsyncAPI, and how they relate and differentiate. Of course alongside a huge deal of TS, testing, documentation, and working in Open Source in general.

To make it easier to digest, this issue can be split up into multiple sections to simplify the process of solving this:

1. Step: Convert the following OpenAPI keywords to form a basic AsyncAPI document with no channels nor operations: openapi, info, servers, tags, externalDocs, and security.
2. Step: Convert the OpenAPI paths keyword into channels and operations in AsyncAPI
3. Step: Convert the OpenAPI components into AsyncAPI components

Potential next steps that should NOT be focused on right now

4. Step: Enable multiple options to customize the conversion.
5. Step: Enable the split of the OpenAPI document into one or more AsyncAPI documents.
6. Step: Integrate it into the AsyncAPI CLI to enable faster conversions for users.
7. Write a blog post about this new conversion

NOTICE, because of the difficulty of this issue, you will be able to contact me all the time through Slack if you get stuck solving it at any point and even have calls when relevant.

Reason/Context

As messageId now will be removed, we need to also remove it from the conversion to v3.

This can be achieved by altering the convertMessages function to now more generically also remove messageId along side the schema union changes:

Feel free to reach out if you have any questions

Related spec issue: asyncapi/spec#978

We are very close to adding a convert command to our main CLI which works identically as the CLI in this project. Around April when we release version 2.4.0 of AsyncAPI we should release version 1.0.0 of that package in which we will remove the CLI from this project and refer to the CLI project.

Pipe https://github.com/asyncapi/asyncapi/blob/master/examples/1.2.0/streetlights.yml to converter and then to parser-js, and see such an error:

Converter does not delete "name" field of components/parameters, which leads to such kind of problem

What Dev Docs changes are you proposing?

• add banner like in another repos with AsyncAPI logo
• add all-contributors section
• add TOC
• add development/contributing section

Code of Conduct

• I agree to follow this project's Code of Conduct

Reason/Context

I would like to throw my name into the hat of consideration as a new maintainer for the converter.

With the AsyncAPI spec 3.0 release I did a bunch of stuff around the converter and know the code base pretty well by now.

I would like to help steer the converter into the future as I believe it can become a core tool used to integrate with different ecosystems.

cc current code owners @fmvilas @magicmatatjahu @derberg

Reason/Context

The sole purpose of the current converter is converting AsyncAPI versions from one version to the next.

However, it does not really create a strong community around it nor enable further development of the tool because it is so narrowly focused on one specific area.

So I want to propose that we open up the tool to encompass convert to AsyncAPI meaning that we can have an implementation that converts from OpenAPI, Protobuf, CloudEvents, Postman collections, etc, into AsyncAPI documents. This way we open up a bunch of integration options for people who already use existing standards to switch over or try AsyncAPI.

Each of these implementations can have its own subset of code owners, in a similar fashion as we have achieved with Modelina: https://github.com/asyncapi/modelina/blob/master/CODEOWNERS

How does that idea sound @derberg (even though you leaving 😆), @fmvilas, @magicmatatjahu?

As most of our tools in which we use converter are written in TS I think it would be a good idea to rewrite converter to TS. The library itself is quite small so it shouldn't be a problem. On the plus side, we will have types for Studio/ServerAPI/CLI.

Related to #110

Reason/Context

I think often you want to run the upgrade on a file and see what has changed in git. In such a case you want the converter to directly modify that file instead of creating a new file.

I executed the command like this:
asyncapi-converter asyncapi.yaml > asyncapi.yaml

but this resulted in an empty file.

As a workaround I run it like this:

asyncapi-converter asyncapi.yaml > asyncapi_updated.yaml
mv asyncapi_updated.yaml asyncapi.yaml

this worked but it would be nice to do it in one step.

Description

To support in-place update, you could add a command line switch -i, e. g.
asyncapi-converter -i asyncapi.yaml

Related code is in lib/index.js

Details abotu template literals -> https://sonarcloud.io/project/issues?id=asyncapi_converter-js&issues=AXh_YR_Lghu_tM7IAP9M&open=AXh_YR_Lghu_tM7IAP9M

More details about issue related with the same line of code -> https://sonarcloud.io/project/issues?id=asyncapi_converter-js&issues=AXSXXHJEI_W3FC5up5d1&open=AXSXXHJEI_W3FC5up5d1

This issue defines a list of tasks that need to be performed in this repo to make sure it's ci/cd automation works long term without any issues.

It is up to maintainers to decide if it must be addressed in one or multiple PRs.

Below are 3 different sections describing 3 different important ci/cd changes.

IMPORTANT-START
For GitHub workflows that contain This workflow is centrally managed in https://github.com/asyncapi/.github/ you do not have to perform any work. These workflows were already updated through the update in .github. The only exception is the workflows related to nodejs release. More details in Upgrade Release pipeline - in case of nodejs projects section
IMPORTANT-END

Deprecation of way data is shared between steps

Every single GitHub Action workflow that has echo "::set-output name={name}::{value}" need to be updated to follow echo "{name}={value}" >> $GITHUB_OUTPUT

We do not yet know when set-output will stop working. Previous disable date was 31.05 but now then say community needs more time.

For more details read official article from GitHub

Deprecation of node12

2nd bullet point is still relevant for you even if your projects in not nodejs project

• Every single workflow that uses setup-node action needs an update to follow v3 version of this action, and make sure minimum node 14 is used
• Now this part is more complex. Problem with node12 is that node-based GitHub Actions were using it in majority as a runtime environment. Look for example at this action.yaml file for setup-node action v2. So the job that you have to do is go through all the workflows, and verify every single action that you use, make sure you are using the latest version that is not based on node12. I already did review a lot of actions as part of this PR so maybe you will find some actions there and can copy from me. For example action/checkout needs to be updated to v3.

Node12 end of support in action is probably September 27th.

For more details read official article from GitHub

Upgrade Release pipeline - in case of nodejs projects

ignore this section if your project is not nodejs project

You have 2 options. You can:

A. choose to switch to new release pipeline using instruction from asyncapi/.github#205

B. stay with old release pipeline, and manually update GitHub workflows and actions used in it, you can inspire a lot from this PR asyncapi/.github#226

I definitely recommend going with A

Workflows related to release:

• .github/workflows/if-nodejs-release.yml
• .github/workflows/if-nodejs-version-bump.yml
• .github/workflows/bump.yml

Reason/Context

For v3, there are bindings that need to change i.e. 0.2.0 needs to be upgraded to 0.3.0 to be compatible with v3.

See the following changes:

• asyncapi/bindings#213
• asyncapi/bindings#203
• asyncapi/bindings#205
• asyncapi/bindings#204

It would be nice if the converter could handle these as well.

🚨 The automated release from the master branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you could benefit from your bug fixes and new features.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can resolve this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the master branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here is some links that can help you:

• Usage documentation
• Frequently Asked Questions
• Support channels

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.

Invalid npm token.

The npm token configured in the NPM_TOKEN environment variable must be a valid token allowing to publish to the registry https://registry.npmjs.org/.

If you are using Two-Factor Authentication, make configure the auth-only level is supported. semantic-release cannot publish with the default auth-and-writes level.

Please make sure to set the NPM_TOKEN environment variable in your CI with the exact value of the npm token.

Good luck with your project ✨

Your semantic-release bot 📦🚀

Reason/Context

If you want to do reusability in AsyncAPI, you cannot use the converter as it only changes the provided AsyncAPI file.

This renders the library more or less "useless" in those cases as we heavily encourage reuseability.

This is a notification for code owners of this repository (@fmvilas @magicmatatjahu @derberg) to be aware of AsyncAPI 2.5.0 pre-releases:

• https://github.com/asyncapi/spec/releases/tag/v2.5.0-next-spec.2
• https://github.com/asyncapi/spec/releases/tag/v2.5.0-next-spec.3
• https://github.com/asyncapi/spec/releases/tag/v2.5.0-next-spec.4
• https://github.com/asyncapi/spec/releases/tag/v2.5.0
• https://github.com/asyncapi/spec-json-schemas/releases/tag/v2.15.0-next-spec.2
• https://github.com/asyncapi/spec-json-schemas/releases/tag/v2.15.0-next-spec.3
• https://github.com/asyncapi/spec-json-schemas/releases/tag/v3.2.0
• https://github.com/asyncapi/parser-js/releases/tag/v1.17.0-next-spec.1
• https://github.com/asyncapi/parser-js/releases/tag/v1.17.0-next-spec.1
• https://github.com/asyncapi/parser-js/releases/tag/v1.17.0

Note: if there will be new pre-releases I'll update this issue and add them to above list.