asyncapi / bundler Goto Github PK

This issue is for https://www.openforce.tech/ participants. If you do not participate in this even please pick up other issue from this repo, there is plenty of help needed 😄

Reason/Context

Bundler like all other tools we have does not need to perform validation (have a look at this description of diff https://github.com/asyncapi/diff#usage). Bundler like others had the main goal to be used by CLI and Studio, where validation/parsing/dereferencing is already provided or should be as dependencies are there already.

Description

• remove parser dependency, it can also be a devdependency in case of tests
• bundler should get already parsed (validated) and dereferenced document as input
• update docs

This issue is for https://www.openforce.tech/ participants. If you do not participate in this even please pick up other issue from this repo, there is plenty of help needed 😄

@Souvikns pinging you so you are aware

Last release is not usable. Package is published without code

👇🏼 is what we see after package installation

I think it might be because of https://github.com/asyncapi/bundler/blob/master/package.json#L37

I think it should say "/lib" and not lib/
I do not see any other reason why it is not really published to npm.

"main": "lib/index.js" entrypoint is correct. It's just that the index file do not exist in the package

I wanted to have two things discussed for the bundler

1. Should we move to typescript?
2. Should we remove the CLI from this project and make it a standalone library?

@derberg

The package is still not properly released.
It is because after migration to TS we forgot that we need to make sure proper lib/index.js code is produced during the release. In other words in current release script we do not run npm run build

Fix is to add below script to package.json:
"prepublishOnly": "npm run build"

Reason/Context

Seems like there are no releases yet, is that on purpose or? 🤔

Describe the bug.

When referenced documents have a lot of indirection some of the $refs are not resolved as I'm expecting. I might be missing something or not using the tooling correctly ... but as far as I could tell this might be a bug.

when i bundle the following files a $ref in commonTypes.json is not resolved correctly. The input files look like this:

main.yml:

asyncapi: 2.6.0
info:
  version: 1.4.2
  title: "demo"
defaultContentType: application/json
channels:
  'task.v1':
    publish:
      operationId: publishTaskEvent
      message:
        $ref: '#/components/messages/task.v1'
components:
  messages:
    task.v1:
      name: task.v1
      contentType: application/cloudevents+json; charset=utf-8
      payload:
        $ref: '#/components/schemas/event-task.v1'
  schemas:
    event-task.v1:
      properties:
        data:
          $ref: '#/components/schemas/task.v1'
    task.v1:
      $ref: 'schema/v1/task.json'

schema/v1/task.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "customerOids": {
      "$ref": "commonTypes.json#/definitions/customerOidArrayType",
      "description": "A list of customer Oids"
    }
  }
}

schema/v1/commonTypes.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "definitions": {
    "customerOidType": {
      "type": "string",
      "minLength": 1,
      "maxLength": 128
    },
    "customerOidArrayType": {
      "type": "array",
      "items": {
        "$ref": "#/definitions/customerOidType"
      }
    }
  }
}

I bundle and validate the files as follows in bundle.js:

const { readFileSync, writeFileSync } = require('fs');
const bundle = require('@asyncapi/bundler');
const { DiagnosticSeverity, Parser } = require('@asyncapi/parser');


async function main() {
  // bundle
  const YAMLDocument = readFileSync('main.yml', 'utf-8')
  const bundledDocument = await bundle([YAMLDocument], { referenceIntoComponents: true})
  writeFileSync('bundled.yaml', bundledDocument.yml());

  // validate the document
  const asyncApiDocument = readFileSync('bundled.yaml', 'utf-8')
  const parser = new Parser();
  const parse = await parser.parse(asyncApiDocument);
  const parseErrors = parse.diagnostics.filter((el) => {
    return el.severity == DiagnosticSeverity.Error;
  });
  if (parseErrors.length > 0) {
    const msg = parseErrors.map((err) => {
      return `${err.code}: ${err.message} [${err.path.join(' > ')}]`;
    });
    throw new Error(msg.join('\n'));
  }
}

main().catch((e) => console.error(e));

The output file looks like this bundled.yml:

asyncapi: 2.6.0
info:
  version: 1.4.2
  title: demo
defaultContentType: application/json
channels:
  task.v1:
    publish:
      operationId: publishTaskEvent
      message:
        $ref: '#/components/messages/task.v1'
components:
  messages:
    task.v1:
      name: task.v1
      contentType: application/cloudevents+json; charset=utf-8
      payload:
        $ref: '#/components/schemas/event-task.v1'
  schemas:
    event-task.v1:
      properties:
        data:
          $ref: '#/components/schemas/task.v1'
    task.v1:
      $schema: http://json-schema.org/draft-07/schema#
      type: object
      properties:
        customerOids:
          description: A list of customer Oids
          type: array
          items:
            $ref: '#/definitions/customerOidType'

The issue is in the final line, the $ref #/definitions/customerOidType is not resolved and thus invalid. The validation outputs:

Error: invalid-ref: '#/definitions/customerOidType' does not exist [components > schemas > task.v1 > properties > customerOids > items > $ref]
    at main (./broken/bundle.js:23:11)

I put the failing example in a repo at https://github.com/deiferni/asyncapibundlerbrokenexample for your convenience.

Expected behavior

I would expect the schema to be included in components.schemas and be referenced correctly in the bundled file, something like this:

asyncapi: 2.6.0
info:
  version: 1.4.2
  title: demo
defaultContentType: application/json
channels:
  task.v1:
    publish:
      operationId: publishTaskEvent
      message:
        $ref: '#/components/messages/task.v1'
components:
  messages:
    task.v1:
      name: task.v1
      contentType: application/cloudevents+json; charset=utf-8
      payload:
        $ref: '#/components/schemas/event-task.v1'
  schemas:
    event-task.v1:
      properties:
        data:
          $ref: '#/components/schemas/task.v1'
    customerOidType:
      type: string
      minLength: 1
      maxLength: 128
    task.v1:
      $schema: http://json-schema.org/draft-07/schema#
      type: object
      properties:
        customerOids:
          description: A list of customer Oids
          type: array
          items:
            $ref: '#/components/schemas/customerOidType'

Screenshots

IMO a screenshot is not very useful :)

How to Reproduce

1. check out the broken example from https://github.com/deiferni/asyncapibundlerbrokenexample (or create files above)
2. run
   • either: node bundle.js
   • or:

   npx asyncapi bundle main.yml > bundled.yaml
   npx asyncapi validate bundled.yaml
   

3. You can see the error on your command line and by inspecting bundled.yaml

🥦 Browser

None

👀 Have you checked for similar open issues?

• I checked and didn't find similar issue

#151 might be related, but it explicitly mentions v3. We are using an older version.
#141 might also be related, but I'm not completely sure either.

🏢 Have you read the Contributing Guidelines?

• I have read the Contributing Guidelines

Are you willing to work on this issue ?

No, someone else can work on it

Reason/Context

All other libraries all publish under AsyncAPI published under @asyncapi/xxx however this repository does not 🤔 Is there a specific reason for it?

Reason/Context

Add an option to specify the base file's directory so we can correctly check for relative references of separate files in the specifications.

I created an asyncapi.yml in the docs folder and in that file I referenced other files with the relative path. And this causes unresolved reference to the file.

Description

Here is an example of the asyncapi.yml file in docs directory

asyncapi: 2.6.0
info:
  title: Test app
  description: This document describes the REST API endpoints and SOCKET events
  version: 0.0.1
servers:
  $ref: "servers.yml#/servers"

Here is the serves.yml file in docs directory

servers:
  production:
    url: test.com/{basePath}
    description: Production server
    protocol: https
    security:
      - JWT: []
    variables:
      basePath:
        default: api/v2

Here is the app.js code in root directory

async function exposeApiDocumentation() {
  try {
    const generator = new Generator('@asyncapi/html-template', path.resolve('public/docs/ui'), { forceWrite: true, install: true })
    generator.generateFromFile('docs/asyncapi.yml')
    const filePaths = [];
    readdirSync(path.resolve('docs')).forEach(item => {
      var filePath = path.join(path.resolve('docs'), item);
      var stat = statSync(filePath);
      if (stat.isFile() && item !== "asyncapi.yml") {
        filePaths.push(filePath)
      } else if (stat.isDirectory()) {
        readdirSync(filePath).forEach(file => {
          var filePath2 = path.join(filePath, file);
          var stat2 = statSync(filePath2);
          if (stat2.isFile())
            filePaths.push(filePath2)
        })
      }
    })
    const pathAsyncapi = path.join(path.resolve('docs'), 'asyncapi.yml')
    const document = await bundle(
      filePaths.map(filePath => {
        return readFileSync(filePath, { encoding: 'utf-8' })
      }),
      {
        base: readFileSync(pathAsyncapi, { encoding: 'utf-8' }),
        referenceIntoComponents: true,
      }
    )
    writeFileSync('public/docs/asyncapi.yml', document.yml());
    app.use('/public', express.static('public'))
  } catch (e) {
    console.log(e)
    getLogger().error(`Error generating api documentation : ${e}`)
  }
}

exposeApiDocumentation()

This is the error that I get

{
  stack: 'ResolverError: Error opening file "XXXX/servers.yml" \n' +
    "ENOENT: no such file or directory, open 'XXXX/servers.yml'\n" +
    '    at ReadFileContext.callback (XXXX/node_modules/@apidevtools/json-schema-ref-parser/lib/resolvers/file.js:52:20)\n' +
    '    at FSReqCallback.readFileAfterOpen [as oncomplete] (node:fs:327:13)\n' +
    '    at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)',
  code: 'ERESOLVER',
  message: 'Error opening file "XXXX/servers.yml" \n' +
    "ENOENT: no such file or directory, open 'XXXX/servers.yml'",
  source: 'XXXX/servers.yml',
  path: null,
  toJSON: [Function: toJSON],
  ioErrorCode: 'ENOENT',
  name: 'ResolverError',
  footprint: 'null+XXXX/servers.yml+ERESOLVER+Error opening file "XXXX/servers.yml" \n' +
    "ENOENT: no such file or directory, open 'XXXX/servers.yml'",
  toString: [Function: toString]
}

Let me know if I could help

Describe the bug

Before I describe the bug I would like to talk about what the base option does:
The base option lets us specify some fields that should not change during the merge process. For example ⬇️

// File 1
asyncapi: 2.5.0
info:
  title: Camera
  version: 2.6.3
  description: An API to connect with the camera 
channels:
  camera/connect:
    subscribe:
      message:
        payload:
          $ref: './messages.yaml#components/ConnectCamera'

// File 2
asyncapi: 2.5.0
info:
  title: Lights
  version: 2.2.3
  description: An API to connect with the Lights 
channels:
  Lights/connect:
    subscribe:
      message:
        payload:
          $ref: './messages.yaml#components/ConnectLight'

// After Combining 

asyncapi: 2.5.0
info:
  title: Smart Home    
  version: 2.6.3
  description: An API to interact with smart home devices      -> base option lets you overwrite specific fields 
channels:
  Lights/connect:
    subscribe:
      message:
        payload:
          $ref: './messages.yaml#components/ConnectLight'
  camera/connect:
    subscribe:
      message:
        payload:
          $ref: './messages.yaml#components/ConnectCamera'

Currently the base file is not being resolved so all the resolved message refs are getting overwritten

asyncapi: 2.5.0
info:
  title: Account Service
  version: 1.0.0
  description: This service is in charge of processing user signups
channels:
  user/signedup:
    subscribe:
      message:
        $ref: ./test/commands/bundle/messages.yaml#/messages/UserSignedUp -> This should be '#/components/messages/UserSignedUp'
  user/loggedOut:
    subcribe:
      message:
        $ref: '#/components/messages/UserLoggedOut'
components:
  messages:
    UserSignedUp:
      payload:
        type: object
        properties:
          displayName:
            type: string
            description: Name of the user
          email:
            type: string
            format: email
            description: Email of the user
    UserLoggedOut:
      payload:
        type: object
        properties:
          displayName:
            type: string
            description: Name of the user
          userId:
            type: string
            description: Id the user
          timestamp:
            type: number
            description: Time stamp when the user logged out

It is working fine if am not using the base option.

How to Reproduce

Steps to reproduce the issue. Attach all resources that can help us understand the issue:

I found this bug while writing asyncapi/cli#391 so building and running the command would help reproduce the bug

./bin/run bundle ./test/commands/bundle/asyncapi.yaml ./test/commands/bundle/spec.yaml --reference-into-components -b ./test/commands/bundle/asyncapi.yaml

Expected behavior

asyncapi: 2.5.0
info:
  title: Account Service
  version: 1.0.0
  description: This service is in charge of processing user logouts
channels:
  user/signedup:
    subscribe:
      message:
        $ref: '#/components/messages/UserSignedUp'
  user/loggedOut:
    subcribe:
      message:
        $ref: '#/components/messages/UserLoggedOut'
components:
  messages:
    UserSignedUp:
      payload:
        type: object
        properties:
          displayName:
            type: string
            description: Name of the user
          email:
            type: string
            format: email
            description: Email of the user
    UserLoggedOut:
      payload:
        type: object
        properties:
          displayName:
            type: string
            description: Name of the user
          userId:
            type: string
            description: Id the user
          timestamp:
            type: number
            description: Time stamp when the user logged out

asyncapi: "2.5.0"
info:
  title: Account Service
  version: 1.0.0
  description: This service is in charge of processing user signups
channels:
  user/signedup:
    subscribe:
      message:
        $ref: "messages.yaml#/messages/UserSignedUp"

asyncapi: "2.5.0"
info:
  title: Account Service
  version: 1.0.0
  description: This service is in charge of processing user logouts
channels:
  user/loggedOut:
    subcribe:
      message:
        $ref: 'messages.yaml#/messages/UserLoggedOut'

messages:
  UserSignedUp:
    payload:
      type: object
      properties:
        displayName:
          type: string
          description: Name of the user
        email:
          type: string
          format: email
          description: Email of the user
  UserLoggedOut:
    payload:
      type: object
      properties:
        displayName:
          type: string
          description: Name of the user
        userId:
          type: string
          description: Id the user
        timestamp:
          type: number
          description: Time stamp when the user logged out

Reason/Context

Currently, the referenceIntoComponents option only seems to work for message components. However, it can also be useful for all other component types (see https://www.asyncapi.com/docs/reference/specification/v2.5.0#componentsObject).

Description

Changes:

• Extend https://github.com/asyncapi/bundler/blob/master/src/parser.ts so that it detects all kinds of references.

Although the bundled spec will change, I would not consider this feature as a breaking change. Implementing this feature would only mean the correct implementation of the current documentation of the referenceIntoComponents flag:

Pass true to resolve external references to components.

Describe the bug

Exception: invalid-yaml: The provided yaml is not valid.
Stack: invalid-yaml: The provided yaml is not valid.

How to Reproduce

I've converted the streetlight example to json and am trying to run the bundler on it.

  const filePaths = ['./spec.json'];
  const document = await bundle(
    filePaths.map((filePath) =>
      fs.readFileSync(path.resolve(path.join(__dirname, filePath)), 'utf-8')
    )
  );
  console.log(document.json());

{
  "asyncapi": "2.4.0",
  "info": {
    "title": "Streetlights Kafka API",
    "version": "1.0.0",
    "description": "The Smartylighting Streetlights API allows you to remotely manage the city lights.\n\n### Check out its awesome features:\n\n* Turn a specific streetlight on/off 🌃\n* Dim a specific streetlight 😎\n* Receive real-time information about environmental lighting conditions 📈\n",
    "license": {
      "name": "Apache 2.0",
      "url": "https://www.apache.org/licenses/LICENSE-2.0"
    }
  },
  "servers": {
    "test": {
      "url": "test.mykafkacluster.org:8092",
      "protocol": "kafka-secure",
      "description": "Test broker",
      "security": [
        {
          "saslScram": []
        }
      ]
    }
  },
  "defaultContentType": "application/json",
  "channels": {
    "smartylighting.streetlights.1.0.event.{streetlightId}.lighting.measured": {
      "description": "The topic on which measured values may be produced and consumed.",
      "parameters": {
        "streetlightId": {
          "$ref": "#/components/parameters/streetlightId"
        }
      },
      "publish": {
        "summary": "Inform about environmental lighting conditions of a particular streetlight.",
        "operationId": "receiveLightMeasurement",
        "traits": [
          {
            "$ref": "#/components/operationTraits/kafka"
          }
        ],
        "message": {
          "$ref": "#/components/messages/lightMeasured"
        }
      }
    },
    "smartylighting.streetlights.1.0.action.{streetlightId}.turn.on": {
      "parameters": {
        "streetlightId": {
          "$ref": "#/components/parameters/streetlightId"
        }
      },
      "subscribe": {
        "operationId": "turnOn",
        "traits": [
          {
            "$ref": "#/components/operationTraits/kafka"
          }
        ],
        "message": {
          "$ref": "#/components/messages/turnOnOff"
        }
      }
    },
    "smartylighting.streetlights.1.0.action.{streetlightId}.turn.off": {
      "parameters": {
        "streetlightId": {
          "$ref": "#/components/parameters/streetlightId"
        }
      },
      "subscribe": {
        "operationId": "turnOff",
        "traits": [
          {
            "$ref": "#/components/operationTraits/kafka"
          }
        ],
        "message": {
          "$ref": "#/components/messages/turnOnOff"
        }
      }
    },
    "smartylighting.streetlights.1.0.action.{streetlightId}.dim": {
      "parameters": {
        "streetlightId": {
          "$ref": "#/components/parameters/streetlightId"
        }
      },
      "subscribe": {
        "operationId": "dimLight",
        "traits": [
          {
            "$ref": "#/components/operationTraits/kafka"
          }
        ],
        "message": {
          "$ref": "#/components/messages/dimLight"
        }
      }
    }
  },
  "components": {
    "messages": {
      "lightMeasured": {
        "name": "lightMeasured",
        "title": "Light measured",
        "summary": "Inform about environmental lighting conditions of a particular streetlight.",
        "contentType": "application/json",
        "traits": [
          {
            "$ref": "#/components/messageTraits/commonHeaders"
          }
        ],
        "payload": {
          "$ref": "#/components/schemas/lightMeasuredPayload"
        }
      },
      "turnOnOff": {
        "name": "turnOnOff",
        "title": "Turn on/off",
        "summary": "Command a particular streetlight to turn the lights on or off.",
        "traits": [
          {
            "$ref": "#/components/messageTraits/commonHeaders"
          }
        ],
        "payload": {
          "$ref": "#/components/schemas/turnOnOffPayload"
        }
      },
      "dimLight": {
        "name": "dimLight",
        "title": "Dim light",
        "summary": "Command a particular streetlight to dim the lights.",
        "traits": [
          {
            "$ref": "#/components/messageTraits/commonHeaders"
          }
        ],
        "payload": {
          "$ref": "#/components/schemas/dimLightPayload"
        }
      }
    },
    "schemas": {
      "lightMeasuredPayload": {
        "type": "object",
        "properties": {
          "lumens": {
            "type": "integer",
            "minimum": 0,
            "description": "Light intensity measured in lumens."
          },
          "sentAt": {
            "$ref": "#/components/schemas/sentAt"
          }
        }
      },
      "turnOnOffPayload": {
        "type": "object",
        "properties": {
          "command": {
            "type": "string",
            "enum": [
              "on",
              "off"
            ],
            "description": "Whether to turn on or off the light."
          },
          "sentAt": {
            "$ref": "#/components/schemas/sentAt"
          }
        }
      },
      "dimLightPayload": {
        "type": "object",
        "properties": {
          "percentage": {
            "type": "integer",
            "description": "Percentage to which the light should be dimmed to.",
            "minimum": 0,
            "maximum": 100
          },
          "sentAt": {
            "$ref": "#/components/schemas/sentAt"
          }
        }
      },
      "sentAt": {
        "type": "string",
        "format": "date-time",
        "description": "Date and time when the message was sent."
      }
    },
    "securitySchemes": {
      "saslScram": {
        "type": "scramSha256",
        "description": "Provide your username and password for SASL/SCRAM authentication"
      }
    },
    "parameters": {
      "streetlightId": {
        "description": "The ID of the streetlight.",
        "schema": {
          "type": "string"
        }
      }
    },
    "messageTraits": {
      "commonHeaders": {
        "headers": {
          "type": "object",
          "properties": {
            "my-app-header": {
              "type": "integer",
              "minimum": 0,
              "maximum": 100
            }
          }
        }
      }
    },
    "operationTraits": {
      "kafka": {
        "bindings": {
          "kafka": {
            "clientId": "my-app-id"
          }
        }
      }
    }
  }
}

Expected behavior

It should basically just return what i've passed in. I planned to introduce external references next but couldn't get past this step.

Instead I get an error:

Exception: invalid-yaml: The provided yaml is not valid.
Stack: invalid-yaml: The provided yaml is not valid.
    at exports.toJS (\node_modules\@asyncapi\bundler\lib\util.js:32:11)
    at \node_modules\@asyncapi\bundler\lib\index.js:38:41
    at Array.map (<anonymous>)
    at bundle (\node_modules\@asyncapi\bundler\lib\index.js:38:29)
    at Object.<anonymous> (\dist\asyncapi\index.js:58:54)
    at Generator.next (<anonymous>)
    at \dist\asyncapi\index.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (\dist\asyncapi\index.js:4:12)       
    at Object.httpTrigger [as default] (\dist\asyncapi\index.js:55:12).

I think commander and meow can be removed from the project as it no longer have CLI.

Reason/Context

Currently, we inline the references inside the AsyncAPI document. However, I think the references should be kept, and relocate it inside components instead, or provide an option to do so, although I dont know why both would be needed. This also fixes a problem with circular references.

For example:

# asyncapi.yaml
asyncapi: '2.2.0'
info:
  title: Account Service
  version: 1.0.0
  description: This service is in charge of processing user signups
channels:
  user/signup:
    subscribe:
      message:
        $ref: './messages.yaml#/messages/UserSignedUp'

#messages.yaml
messages:
  UserSignedUp:
    payload:
      type: object
      properties:
        displayName:
          type: string
          description: Name of the user
        email:
          type: string
          format: email
          description: Email of the user

Should resolve to:

# After combining 
asyncapi: 2.2.0
info:
  title: Account Service
  version: 1.0.0
  description: This service is in charge of processing user signups
channels:
  user/signedup:
    subscribe:
      message:
        $ref: '#/components/messages/UserSignedUp'
components: 
  messages:
    UserSignedUp:
      payload:
        type: object
        properties:
          displayName:
            type: string
            description: Name of the user
          email:
            type: string
            format: email
            description: Email of the user

Describe the bug

I have the current folder setup:

definitions
  asyncapi.json
  components
    messages
       ...
bundler
  bundlerScript.js

And standing inside the bundler dir executing bundlerScript.js it is unable to locate the references for example messages as ./components/messages/myMessage.json that asyncapi.json uses.

Expected behavior

Expected the setup to work.

I dont quite understand why there is a custom dereference in place before parsing the documents? Otherwise, I could utilize the following custom parser. Although a bit of a hack as there is no real docs what is expected from parser option.

  const bundledDocument = await bundler(
    fileContents,
    {
      parser: {
        parse: (fileContent) => {
          const p = path.resolve(__dirname, '../../definitions')
          return parse(fileContent, {path: p})
        }
      }
    }
  );

Why do we need this improvement?

Resolving relative paths from asyncapi documents located in different folders is currently not possible in a way that would keep the scope of the single files locally.

For example i am trying to destructure our asyncapi spec into smaller documents to reduce complexity.
To do this i tried to use the following folder structure:

docs
├── common
│   └── metadata.yaml
├── index.yaml
├── receive
│   └── event-zero
│       ├── asyncapi.yaml
│       └── schema.yaml
└── send
    └── event-one
        ├── asyncapi.yaml
        └── schema.yaml

index.yaml would be the base file and our different operations would be split into separate files.
Now when trying to bundle this into a single asyncapi spec i am running into a problem when trying to reference schema.yaml from any asyncapi.yaml via ./schema.yaml.

By default it will take the current working directory which is try to resolve it to docs/schema.yaml.
With the help of the --baseDir arg I would only be able to set the baseDir to docs.
I would still have to reference schema.yaml via a path that is not relative to the files that define each operation (event-zero, event-one): receive/event-zero/schema.yaml.

While that works, it is easy to make a mistake when creating a new operation and it also makes it impossible to validate each document separately, since validating always assumes $refs relative to the document's directory.

How will this change help?

• I am able to structure each split asyncapi document i want to bundle into separate folders and each file can only be aware of its own context.
• Creating new asyncapi documents for additional operations is less prone to copy paste errors due to be relative to their own directory
• Validating/Generating based on each asyncapi document separately is possible
  • One thing I personally want to do is to generate a markdown for each operation separately but an html documentation with all operations of the service combined. This is currently not possible like this because asyncapi generate fromTemplate parsing expects the paths to be relative to the document without the possibility to specify a "baseDir"
• It is less confusing to read the documents because it is clear, what the path in $refs are relative to

Screenshots

No response

How could it be implemented/designed?

Change directory to the folder of each asyncapi document when dereferencing any $refs, unless a --baseDir option is specified (to not break the functionality of it).

Alternatively it is also possible to implement this in a non-breaking way in which it would only apply when a new option like --path-relative-to-document is specified in case a breaking change by changing default behaviour is undesired.

🚧 Breaking changes

Yes

👀 Have you checked for similar open issues?

• I checked and didn't find a similar issue

🏢 Have you read the Contributing Guidelines?

• I have read the Contributing Guidelines

Are you willing to work on this issue?

Yes I am willing to submit a PR!

After migration to TS in readme we focus only on examples with import.

We need examples with require. Especially that before migration const bundle = require('@asyncapi/bundler'); worked well before 0.3 and now it doesn't.

I think we can consider it also a breaking change 🤔

const bundle = require('@asyncapi/bundler');
console.log(bundle)
//{ Document: [Getter], default: [AsyncFunction: bundle] }
//TypeError: bundle is not a function

Referencing the linked line - is there a reason the bundler is not being used? As-is, resolving a v3 document only resolves references found within channels and operations. I might be missing something?

asyncapi: 3.0.0
info:
  $ref: './info.yaml#/info'
...

ends up outputting $ref: './info.yaml#/info' instead of resolving the value of the reference into the final document. The same goes for other parts of the v3 document, including tags, etc.

Switching the spec version to 2.6.0 does result in correctly resolved references (ignoring differences between 2.x and 3.x specs).

main.yaml:

asyncapi: 3.0.0
info:
  $ref: './info.yaml#/info'
channels:
  userSignedup:
    address: 'user/signedup'
    messages:
      userSignedUpMessage:
        $ref: './messages.yaml#/messages/UserSignedUp'
  test:
    address: '/test'
    messages:
      testMessage:
        $ref: '#/components/messages/TestMessage'
operations:
  UserSignedUp:
    action: send
    channel:
      $ref: '#/channels/userSignedup'
    messages:
      - $ref: './messages.yaml#/messages/UserSignedUp'
  TestOpp:
    action: send
    channel:
      $ref: '#/channels/test'
    messages:
      - $ref: '#/components/messages/TestMessage'
components:
  messages:
    TestMessage:
      payload:
        type: string

messages.yaml:

messages:
  UserSignedUp:
    payload:
      type: object
      properties:
        displayName:
          type: string
          description: Name of the user
        email:
          type: string
          format: email
          description: Email of the user
  UserLoggedIn:
    payload:
      type: object
      properties:
        id: string

info.yaml:

info:
  title: Account Service
  version: 1.0.0
  description: This service is in charge of processing user signupsA

expected output:

asyncapi: 3.0.0
info:
  title: Account Service
  version: 1.0.0
  description: This service is in charge of processing user signupsA
channels:
  userSignedup:
    address: user/signedup
    messages:
      userSignedUpMessage:
        $ref: '#/components/messages/UserSignedUp'
  test:
    address: /test
    messages:
      testMessage:
        $ref: '#/components/messages/TestMessage'
operations:
  UserSignedUp:
    action: send
    channel:
      $ref: '#/channels/userSignedup'
    messages:
      - $ref: '#/components/messages/UserSignedUp'
  TestOpp:
    action: send
    channel:
      $ref: '#/channels/test'
    messages:
      - $ref: '#/components/messages/TestMessage'
components:
  messages:
    TestMessage:
      payload:
        type: string
    UserSignedUp:
      payload:
        type: object
        properties:
          displayName:
            type: string
            description: Name of the user
          email:
            type: string
            format: email
            description: Email of the user

actual output:

asyncapi: 3.0.0
info:
  $ref: ./info.yaml#/info    # <-- unresolved ref
channels:
  userSignedup:
    address: user/signedup
    messages:
      userSignedUpMessage:
        $ref: '#/components/messages/UserSignedUp'
  test:
    address: /test
    messages:
      testMessage:
        $ref: '#/components/messages/TestMessage'
operations:
  UserSignedUp:
    action: send
    channel:
      $ref: '#/channels/userSignedup'
    messages:
      - $ref: '#/components/messages/UserSignedUp'
  TestOpp:
    action: send
    channel:
      $ref: '#/channels/test'
    messages:
      - $ref: '#/components/messages/TestMessage'
components:
  messages:
    TestMessage:
      payload:
        type: string
    UserSignedUp:
      payload:
        type: object
        properties:
          displayName:
            type: string
            description: Name of the user
          email:
            type: string
            format: email
            description: Email of the user

the following test fails when run against the above files:

test('should be able to bundle v3 files', async () => {
    const files = ['./main.yaml']
    const response = await bundle(
        files.map(file => fs.readFileSync(path.resolve(process.cwd(), file), 'utf-8'))
    )
    const expected = fs.readFileSync(path.resolve(process.cwd(), './bundled.yaml'), 'utf-8');
    expect(response.yml()).toBe(expected);
  });

@Souvikns states in PR
#46
I also think this would be the right time to move to typescript

I would like to take on this issue also.
@derberg @magicmatatjahu @jonaslagoni, there's a need to decide who of us two will migrate the bundler's codebase to TypeScript.

Reason/Context

The parser has already done this change because @apidevtools/json-schema-ref-parser does not have a maintainer.

• To successfully bundle, we need different spec files which we would take as input. We can load these files by URL path where we will read all the yml or JSON files, or we would take a list of file paths.
• we also need an output file path where we will dump the complete bundled spec document.

Describe the bug

When you go to https://github.com/asyncapi/bundler/blob/master/API.md#functions you see some internal functions in the API that users can use. But we should expose only bundle

Expected behavior

All functions other than bundle should be annotated as private, so they do not show up in the API

Reason/Context

To facilitate seamless integration with other tools that implement bundler, we can enhance compatibility by exporting a designated flag or configuration option. This will effectively communicate the supported spec version, enabling smoother interactions between systems.

We can use the Problem library to define an error code that could be reused in other tools as well. Any tool implementing bundler can check if the spec bundler being used on, is supported or not and throw appropriate warnings and errors.

Issue is reported by user in
https://asyncapi.slack.com/archives/CQVJXFNQL/p1666700935030749

Shani Alaluf Shamah
We have an issue in bundle version 0.3.4 using parser 1.17.1
When adding the feature referenceIntoComponents: true
we get the following error:

TypeError: Cannot read property 'startsWith' of undefined
    at isExternalReference (/Users/shani.shamah/IdeaProjects/data-model-schemas/node_modules/@asyncapi/bundler/lib/parser.js:41:17)
    at /Users/shani.shamah/IdeaProjects/data-model-schemas/node_modules/@asyncapi/bundler/lib/parser.js:58:13
    at Array.forEach (<anonymous>)
    at resolveExternalRefs (/Users/shani.shamah/IdeaProjects/data-model-schemas/node_modules/@asyncapi/bundler/lib/parser.js:56:8)
    at parse (/Users/shani.shamah/IdeaProjects/data-model-schemas/node_modules/@asyncapi/bundler/lib/parser.js:80:43)
    at async resolve (/Users/shani.shamah/IdeaProjects/data-model-schemas/node_modules/@asyncapi/bundler/lib/util.js:62:13)
    at async bundle (/Users/shani.shamah/IdeaProjects/data-model-schemas/node_modules/@asyncapi/bundler/lib/index.js:42:27)
    at async createApiSpecforService (/Users/shani.shamah/IdeaProjects/data-model-schemas/scripts/generate-api-spec.js:12:15) {stack: 'TypeError: Cannot read property 'startsWith' …l-schemas/scripts/generate-api-spec.js:12:15)', message: 'Cannot read property 'startsWith' of undefined'}

it doesn’t work with 0.3.3 either

With 0.2.3 I am getting the following error:

TypeError: bundle is not a function

I’m using JS, require (CJS) syntax:

const bundle = require('@asyncapi/bundler').default;

switched to 0.2.3 with

const bundle = require("@asyncapi/bundler")

but still getting the Cannot read property 'startsWith' of undefined error.

Maciej Urbańczyk
https://github.com/asyncapi/bundler/blob/master/src/parser.ts#L44 we should check if $ref exists and then check if starts with #
in old code this same https://github.com/Souvikns/bundler/blob/ef574b8620f12c075d89bfc742070748941a45e0/lib/parser.js#L51 😅

Lukasz Gornicki
@shani Alaluf Shamah
ok so mistery solved
imho change is to just extend condition from https://github.com/asyncapi/bundler/blob/master/src/parser.ts#L64, to if (ref && isExternalReference(ref))

Maciej Urbańczyk
we use isExternalReference in several places, we should check if $ref exist inside isExternalReference function:

function isExternalReference(ref?: string): boolean {
  return typeof ref === 'string' && !ref.startsWith('#');
}

Lukasz Gornicki
makes sense 👍:skin-tone-2:

Describe the bug.

Invoking the bundle() function with a baseDir option provided will have the working directory changed, but will not be changed back on function exit. Successive calls to this method where each call provides a relative path baseDir option will fail, specifically on the 2nd call, since the original relative path no longer makes sense in the CWD.

Error:

Error: ENOENT: no such file or directory, chdir '<project>/src' -> './src'
    at process.wrappedChdir (node:internal/bootstrap/switches/does_own_process_state:130:14)
    at process.chdir (<project>/node_modules/graceful-fs/polyfills.js:22:11)
    at bundle (<project>/node_modules/@asyncapi/bundler/lib/index.js:85:17)
    at build (<project>/gulpfile.js:37:30)

The lines in question are:

• bundler/src/index.ts

  Line 90 in c9ae036

  process.chdir(options.baseDir);

• bundler/src/index.ts

  Line 92 in c9ae036

  process.chdir(String(options.baseDir[0])); // guard against passing an array

The function will need to copy the CWD, and restore it prior to exiting. In cases where the function body throws, the exception should most likely be caught, the CWD reverted, and the exception be propagated as per normal.

Expected behavior

Invoking the bundle restores the CWD.

Screenshots

N/A

How to Reproduce

  // Bundle server AsyncAPI files
  const serverMerged = await bundle(["common.yaml"], {
    base: "server.yaml",
    baseDir: "./src/",
    xOrigin: false,
  });

  // Merge client AsyncAPI files
  // Fails here due to ./src/ relative path no longer correct, as the cwd IS NOW ./src/. 
  const clientMerged = await bundle(["common.yaml"], {
    base: "client.yaml",
    baseDir: "./src/",
    xOrigin: false,
  });

🥦 Browser

None

👀 Have you checked for similar open issues?

• I checked and didn't find similar issue

🏢 Have you read the Contributing Guidelines?

• I have read the Contributing Guidelines

Are you willing to work on this issue ?

Yes I am willing to submit a PR!

This issue defines a list of tasks that need to be performed in this repo to make sure it's ci/cd automation works long term without any issues.

It is up to maintainers to decide if it must be addressed in one or multiple PRs.

Below are 3 different sections describing 3 different important ci/cd changes.

IMPORTANT-START
For GitHub workflows that contain This workflow is centrally managed in https://github.com/asyncapi/.github/ you do not have to perform any work. These workflows were already updated through the update in .github. The only exception is the workflows related to nodejs release. More details in Upgrade Release pipeline - in case of nodejs projects section
IMPORTANT-END

Deprecation of way data is shared between steps

Every single GitHub Action workflow that has echo "::set-output name={name}::{value}" need to be updated to follow echo "{name}={value}" >> $GITHUB_OUTPUT

We do not yet know when set-output will stop working. Previous disable date was 31.05 but now then say community needs more time.

For more details read official article from GitHub

Deprecation of node12

2nd bullet point is still relevant for you even if your projects in not nodejs project

• Every single workflow that uses setup-node action needs an update to follow v3 version of this action, and make sure minimum node 14 is used
• Now this part is more complex. Problem with node12 is that node-based GitHub Actions were using it in majority as a runtime environment. Look for example at this action.yaml file for setup-node action v2. So the job that you have to do is go through all the workflows, and verify every single action that you use, make sure you are using the latest version that is not based on node12. I already did review a lot of actions as part of this PR so maybe you will find some actions there and can copy from me. For example action/checkout needs to be updated to v3.

Node12 end of support in action is probably September 27th.

For more details read official article from GitHub

Upgrade Release pipeline - in case of nodejs projects

ignore this section if your project is not nodejs project

You have 2 options. You can:

A. choose to switch to new release pipeline using instruction from asyncapi/.github#205

B. stay with old release pipeline, and manually update GitHub workflows and actions used in it, you can inspire a lot from this PR asyncapi/.github#226

I definitely recommend going with A

Workflows related to release:

• .github/workflows/if-nodejs-release.yml
• .github/workflows/if-nodejs-version-bump.yml
• .github/workflows/bump.yml

Reason/Context

This Issue is used to track changes needed to support AsyncAPI v3. As a code owner, please edit this list of TODO tasks in order to properly track the progress 🙂 Once this issue is closed it means that v3 is now fully supported in this library.

Remaining tasks:

• Update to support v3 structures for bundling

Describe the bug you found in AsyncAPI Docs.

Usage example has an old example where the bundle function was not exported as a default export. Since now the it is default export we need to change the example

Now the example would look something like this

const bundle = require('@asyncapi/bundler');
const fs = require('fs');
const path = require('path');

const filePaths = ['./camera.yml','./audio.yml']
const document = await bundle(
  filePaths.map(filePath => fs.readFileSync(path.resolve(filePaths), 'utf-8')),
  {
    base: fs.readFileSync(path.resolve('./base.yml'), 'utf-8')
  }
);

console.log(document.json()); // the complete bundled asyncapi document.

Attach any resources that can help us understand the issue.

Code of Conduct

• I agree to follow this project's Code of Conduct

Reason/Context

This is to replace the old AsyncAPI logo in this repo's README with the banner attached below that represents the new branding.

Here are a few guidelines for this change as well:

1. Make sure you are using Markdown syntax only
2. Be sure to remove the old logo as well as the old title of the repo as this image will replace both elements
3. Make sure you link this image to the website: https://www.asyncapi.com
4. If there is any description text below the repo title, let's make it left-aligned if it isn't already, so as to match the left-alignment of the content in the new banner image

Download the image file:
github-repobanner-bundler.png.zip

Banner preview

Please note that this is only a preview of the image, the contributor should download and use the above zip file

Context

#97

Summary:

Integrating the optimizer and bundler to enhance user experience requires a few modifications. These include removing redundant features, tracing the origins of references, and improving naming conventions within the bundled output.

Detailed Steps:

1) Simplify the Bundler by Removing referenceIntoComponents:

The referenceIntoComponents feature in the bundler seems to belong to the optimizer. Its removal should be straightforward:

• Identify the areas of code where the referenceIntoComponents flag is utilized.
• Ensure that the feature is decoupled without affecting other functionalities.
• Remove the feature and its flag entirely from the codebase.

2) Traceability of $ref Origins Post-Bundling:

To maintain a clear lineage of $ref components post-bundling, it's necessary to include an x-origin property. This will annotate where the original $ref was located in the source files.

For example, transform this:

...
  message:
    $ref: ./messages.yaml#/messages/UserSignedUp
...

Into this:

...
  message:
    x-origin: ./messages.yaml#/messages/UserSignedUp
    payload:
      type: object
      properties:
        displayName:
          type: string
          description: Name of the user
...

3) Introduction of a New Flag in the optimizer:

We propose adding a flag to the optimizer to centralize all components under the components section of an AsyncAPI document. The proposed flag is moveAllComponents. Alternative suggestions for the flag name are welcome for better intuitiveness.

4) Intelligent Component Naming in the optimizer:

With the x-origin in place, the optimizer should leverage it to assign meaningful names to components, falling back to a standard naming convention (e.g., message-1) only when a better name isn't discernible from the context.

For instance, utilizing UserSignedUp derived from x-origin instead of a generic message-1.