ovotech / castle Goto Github PK

Hi!

my build currently says:

error Couldn't find package "@ovotech/ts-compose@^0.16.0" required by "@ovotech/avro-ts" on the "npm" registry.

Looks like @ovotech/ts-compose was unpublished from the npm registry.

Is there a workaround or a fix for this?

We are using blaise to generating kafka messages. In this case the schema is a union.

Code used:

    const message =
      blaise<ComOvoenergyKafkaHomemovesEvent.ChangeOfTenancyEvent>({
        eachMessage: {topic: KAFKA_TOPICS.HOME_MOVES_COT_V1},
        avro: {schema: homeMovesEventSchema},
      }).message({
        value: {
          event: {
            'com.ovoenergy.kafka.homemoves.event.ChangeOfTenancyCancelled': {
              incomingAccountId: 'someIncomingAccount1',
              supplyPoint: '123',
            },
          },
          metadata: {
            createdAt: now.getTime(),
            eventId: 'someEventId1',
            traceToken: 'someTraceToken1',
          },
        },
      });

Nothing weird about this, the output:

if you notice blaise adds ChangeOfTenancyCreate with some weird data, where we have never passed. This looks like a memory leak, or data from before

The code currently makes the wrong assumption that there can be only a single schema in a topic, see here:

This makes it impossible to produce messages of multiple schemas in a single topic.

Hi, in your current api you have getSubjectVersionSchema which maps to GET /subjects/:subject/versions/:version/schema but I'm missing a getSubjectVersion which maps to GET /subjects/:subject/versions/:version, this gets back the schema id (in addition to the schema string) which will be very helpful to me. Thanks.

Hello 👋

I think there is something wrong when using Avro enums with namespaces and trying to generate corresponding typescript types.

I'm using these two schemas:

{
    "type": "enum",
    "name": "AvroJobStatus",
    "namespace": "com.zenaton.jobManager.data",
    "symbols": [
        "RUNNING_OK",
        "RUNNING_WARNING",
        "RUNNING_ERROR",
        "TERMINATED_COMPLETED",
        "TERMINATED_CANCELED"
    ]
}

{
    "type": "record",
    "name": "AvroJobStatusUpdated",
    "namespace": "com.zenaton.jobManager.messages",
    "fields": [
        {
            "name": "jobId",
            "type": "string",
            "logicalType": "uuid"
        },
        {
            "name": "jobName",
            "type": "string"
        },
        {
            "name": "oldStatus",
            "type": [
                "null",
                "com.zenaton.jobManager.data.AvroJobStatus"
            ]
        },
        {
            "name": "newStatus",
            "type": "com.zenaton.jobManager.data.AvroJobStatus"
        }
    ]
}

Generating types give the following results:

// AvroJobStatus enum

export type AvroType = "RUNNING_OK" | "RUNNING_WARNING" | "RUNNING_ERROR" | "TERMINATED_COMPLETED" | "TERMINATED_CANCELED";

// AvroJobStatusUpdated record

export type AvroJobStatusUpdated = ComZenatonJobManagerMessages.AvroJobStatusUpdated;

export namespace ComZenatonJobManagerMessages {
    export const AvroJobStatusUpdatedName = "com.zenaton.jobManager.messages.AvroJobStatusUpdated";
    export interface AvroJobStatusUpdated {
        jobId: string;
        jobName: string;
        oldStatus: null | ComZenatonJobManagerData.AvroJobStatus;
        newStatus: ComZenatonJobManagerData.AvroJobStatus;
    }
}

The issues here are:

• The enum is not honoring the namespace instruction. It does not have a export namespace instruction. It also probably miss a type definition to define its type outside the namespace as the record does.
• The enum is not named correctly. It is named AvroType instead of AvroJobStatus.
• The record is missing an import of the (not yet existing) enum namespace: import { ComZenatonJobManagerData as ComZenatonJobManagerDataAvroJobStatus } from "./AvroSerializedData.avsc";
• The record should use the type ComZenatonJobManagerDataAvroJobStatus.AvroJobStatus for the oldStatus and newStatus properties.

I believe the generated results of these two schemas should be the following (if I did not do any mistake):

// AvroJobStatus enum
export type AvroJobStatus = ComZenatonJobManagerData.AvroJobStatus;

export namespace ComZenatonJobManagerData {
  export type AvroJobStatus =
    | 'RUNNING_OK'
    | 'RUNNING_WARNING'
    | 'RUNNING_ERROR'
    | 'TERMINATED_COMPLETED'
    | 'TERMINATED_CANCELED';
}

// AvroJobStatus record
/* eslint-disable @typescript-eslint/no-namespace */

import { ComZenatonJobManagerData as ComZenatonJobManagerDataAvroJobStatus } from './AvroJobStatus.avsc';

export type AvroJobStatusUpdated = ComZenatonJobManagerMessages.AvroJobStatusUpdated;

export namespace ComZenatonJobManagerMessages {
  export const AvroJobStatusUpdatedName =
    'com.zenaton.jobManager.messages.AvroJobStatusUpdated';
  export interface AvroJobStatusUpdated {
    jobId: string;
    jobName: string;
    oldStatus: null | ComZenatonJobManagerDataAvroJobStatus.AvroJobStatus;
    newStatus: ComZenatonJobManagerDataAvroJobStatus.AvroJobStatus;
  }
}

These changes will lead to BC breaks though.

What do you think?

Hi,

I have stumbled upon very strange error – Cannot work out type [object Object].

Little bit of digging brought me to following schema:

{
  "type" : "error",
  "name" : "EpicFailure",
  "namespace" : "namespace",
  "fields" : [ {
    "name" : "code",
    "type" : {
      "type" : "enum",
      "name" : "ErrorCode",
      "symbols" : [ "ERROR" ]
    }
  }, {
    "name" : "message",
    "type" : "string"
  } ]
}

Which is absolutely correct schema, but apparently we get issue with error type. Here is piece of my initial avdl:

  enum ErrorCode {
    ERROR
  }

  error EpicFailure {
    ErrorCode code;
    string message;
  }

Is possible to add support for error types too?

Hey Ivan,

Opening a discussion on a recent issue we've at in the BIT team - if you like it I'll open a PR.

We discovered we were producing records without keys due to our limited knowledge of kafka. Kafka brokers block null keys on a log-compacted topic but in a way that doesn't immediately fail the production so it didn't show up in the logs for a while.

It's a silly mistake too easy to happen. Strengthening the TS types on castle to make this mistake harder to happen: CastleSender can block undefined keys, forcing a producer to explicitly define key: null in case they want a null key.

In essence, this will block sendFunction(producer, [{value}]) and force to pass a key sendFunction(producer, [{value, key: null}]), removing risks of making this mistake again.

export declare type CastleSender<
  TValue = unknown,
  TKey = AvroMessage['key']
> = (producer: AvroProducer, messages: AvroMessage<TValue, TKey>[]) => Promise<RecordMetadata[]>;

into

export declare type CastleSender<
  TValue = unknown,
  TKey = Exclude<AvroMessage['key'], undefined>
> = (producer: AvroProducer, messages: AvroMessage<TValue, TKey>[]) => Promise<RecordMetadata[]>;

Cheers

I'm currently working with the external example in the repository.

Upon executing the script, I encountered the following output:

DeprecationWarning: 'createPropertySignature' has been deprecated since v4.0.0. Use the appropriate method on 'ts.factory' or the 'factory' supplied by your transformation context instead.
DeprecationWarning: 'createInterfaceDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createTypeAliasDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createModuleDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createImportClause' has been deprecated since v4.0.0. Use the appropriate method on 'ts.factory' or the 'factory' supplied by your transformation context instead.
DeprecationWarning: 'createImportDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.

/* eslint-disable @typescript-eslint/no-namespace */

import { type MyNamespaceDataAddress as  } from "./external-Address";

export type CreateUser = MyNamespaceMessages.CreateUser;

export namespace MyNamespaceMessages {
    export const CreateUserSchema = "{\"type\":\"record\",\"name\":\"CreateUser\",\"namespace\":\"my.namespace.messages\",\"fields\":[{\"name\":\"userId\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"address\",\"type\":\"my.namespace.data.Address\"}]}";
    export const CreateUserName = "my.namespace.messages.CreateUser";
    export interface CreateUser {
        userId: string;
        name: string;
        address: MyNamespaceDataAddress.Address;
    }
}

There are two issues identified in the output:

1. The import statement import { type MyNamespaceDataAddress as } from "./external-Address"; is incorrect and should be replaced with import { type MyNamespaceData } from "./external-Address";.
2. The type of address in the CreateUser interface is declared as MyNamespaceDataAddress.Address, but it should be MyNamespaceData.Address.

/* eslint-disable @typescript-eslint/no-namespace */

import { type MyNamespaceData } from "./external-Address";

export type CreateUser = MyNamespaceMessages.CreateUser;

export namespace MyNamespaceMessages {
    export const CreateUserSchema = "{\"type\":\"record\",\"name\":\"CreateUser\",\"namespace\":\"my.namespace.messages\",\"fields\":[{\"name\":\"userId\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"address\",\"type\":\"my.namespace.data.Address\"}]}";
    export const CreateUserName = "my.namespace.messages.CreateUser";
    export interface CreateUser {
        userId: string;
        name: string;
        address: MyNamespaceData.Address;
    }
}

These corrections would ensure the code functions as intended.

Node version: v18.19.0
Typescript version: 5.4.2
Using default tsconfig

Context

Using @ovotech/[email protected] and executing

npx avro-ts address.avsc.json create-user.avsc.json

where both the schemas were copied from avro-ts/examples

Expected

Generated TypeScript files should contain no errors in their imports of external references

Actual

I got these generated output with errors in imports of external reference to my.namespace.data.Address

create-user.avsc.json.ts has

1. import which wasn't exported from address.avsc.json.ts; and
2. import is missing alias identifier after the as; and
3. broken CreateUser.address reference with an undefined namespace.

/* eslint-disable @typescript-eslint/no-namespace */

import { type MyNamespaceDataAddress as  } from "./address.avsc.json";     (1), (2)

export type CreateUser = MyNamespaceMessages.CreateUser;

export namespace MyNamespaceMessages {
    export const CreateUserName = "my.namespace.messages.CreateUser";
    export interface CreateUser {
        userId: string;
        name: string;
        address: MyNamespaceDataAddress.Address;                           (3)
    }
}

address.avsc.json looks ok and the namespace exported is actually MyNamespaceData rather than MyNamespaceDataAddress which is imported by the create-user.avsc.json.ts above.

/* eslint-disable @typescript-eslint/no-namespace */

export type Address = MyNamespaceData.Address;

export namespace MyNamespaceData {
    export const AddressName = "my.namespace.data.Address";
    export interface Address {
        street: string;
        zipcode: string;
        country: string;
    }
}

It seems, that record field types are not properly nested with a record namespace. Namely having a.avsc schema:

{
  "type": "enum",
  "name": "A",
  "namespace": "com.example",
  "symbols": [
    "foo",
    "bar"
  ]
}

and b.avsc schema:

{
  "type": "record",
  "name": "B",
  "namespace": "com.example",
  "fields": [
    {
      "name": "a",
      "type": "A"
    }
  ]
}

Doesn't work - when avro-ts a.avsc b.avsc is executed, the resulting b.avsc.ts file has no import. If, however, "type": "A" is replaced with "type": "com.example.A", import shows up. This seems to be conflicting with Avro specification and also with how avsc package handles this.

We're using KafkaJS Schema Registry to decode avro messages. It would be nice if avro-ts included the schema used to generate the types so we can decode to the appropriate version of the schema. e.g.

// without schema, have to be explicit about version if the consumer wants a different version from the producer
const registryId = await registry.getRegistryId(MyEvents.MyEventName, schemaVersion);
const schema = await registry.getSchema(registryId);
const decoded = await registry.decode(buffer, { AVRO: { readerSchema: schema } });

// with schema
const decoded = await registry.decode(buffer, { AVRO: { readerSchema: MyEvents.MyEventSchema } });

The Schema type generated could be equivalent to the following interface:

export interface RawAvroSchema {
    name: string;
    namespace?: string;
    type: 'record';
    fields: any[];
}

This would also be handy for registering the schema with Schema Registry in the first place if you were publishing using generated types.

Generators for other languages such as the .NET apache.avro.tools package do this and it's really helpful.

In avro-ts AVRO enums are only converted to string unions. I'm not sure the exact reason for this, but I was looking for a way to use Typescript enums instead.

For example

{
  "name": "com.example.avro.MyEnum",
  "type": "enum",
  "symbols": ["ACTIVE", "INACTIVE"]
}

will be converted to

export type ComExampleAvroMyEnum = "ACTIVE" | "INACTIVE";

With Typescript enums it will be converted to

export enum ComExampleAvroMyEnum {
    ACTIVE = "ACTIVE",
    INACTIVE = "INACTIVE"
}

          @ivank, let's merge this. It'll be a step to fix https://github.com/ovotech/castle/issues/107.

After this we should update @ovotech/avro-ts dependency for @ovotech/avro-ts-cli, and bug will be fixed.

Originally posted by @gwer in #105 (comment)

Is it possible to merge this PR please ?

I'm looking to use the capabilities added in the PR #139, but see that a new NPM package has not been published.

Can you please make a new release with the latest changes available?

Hi,
Is there any option or workaround to support BigInt? number won't support 64bit id which is used as snowflake id

Avro:
union {null,long} orderId = null;

will generate
orderId?: null | undefined | number

expected
orderId?: null | undefined | BigInt

given the schema

{
  "type": "record",
  "name": "test",
  "namespace": "com.ovoenergy.kafka.test.event",
  "fields": [
    {
      "name": "event",
      "type": [
        {
          "type": "record",
          "name": "A",
          "fields": [
            {
              "name": "foo",
              "type": "string"
            },
            {
              "name": "bar",
              "type": {
                "type": "int",
                "logicalType": "date"
              }
            }
          ]
        },
        {
          "type": "record",
          "name": "B",
          "fields": [
            {
              "name": "fuzz",
              "type": "boolean",
              "default": true
            }
          ]
        },
        {
          "type": "record",
          "name": "C",
          "fields": [
            {
              "name": "foo",
              "type": "string"
            },
            {
              "name": "bar",
              "type": {
                "type": "int",
                "logicalType": "date"
              }
            }
          ]
        }
      ]
    }
  ]
}

Running the library produces (correctly) the types

/* eslint-disable @typescript-eslint/no-namespace */

export type Test = ComOvoenergyKafkaTestEvent.Test;

export namespace ComOvoenergyKafkaTestEvent {
    export const AName = "com.ovoenergy.kafka.test.event.A";
    export interface A {
        foo: string;
        bar: number;
    }
    export const BName = "com.ovoenergy.kafka.test.event.B";
    export interface B {
        /**
         * Default: true
         */
        fuzz: boolean;
    }
    export const CName = "com.ovoenergy.kafka.test.event.C";
    export interface C {
        foo: string;
        bar: number;
    }
    export const TestName = "com.ovoenergy.kafka.test.event.test";
    export interface Test {
        event: {
            "com.ovoenergy.kafka.test.event.A": ComOvoenergyKafkaTestEvent.A;
            "com.ovoenergy.kafka.test.event.B"?: never;
            "com.ovoenergy.kafka.test.event.C"?: never;
        } | {
            "com.ovoenergy.kafka.test.event.A"?: never;
            "com.ovoenergy.kafka.test.event.B": ComOvoenergyKafkaTestEvent.B;
            "com.ovoenergy.kafka.test.event.C"?: never;
        } | {
            "com.ovoenergy.kafka.test.event.A"?: never;
            "com.ovoenergy.kafka.test.event.B"?: never;
            "com.ovoenergy.kafka.test.event.C": ComOvoenergyKafkaTestEvent.C;
        };
    }
}

Meaning you can access the union members like event.A, event.B etc

Changing the schema (not addition of null into union:

{
  "type": "record",
  "name": "test",
  "namespace": "com.ovoenergy.kafka.test.event",
  "fields": [
    {
      "name": "event",
      "default": "null",
      "type": [
        "null",
        {
          "type": "record",
          "name": "A",
          "fields": [
            {
              "name": "foo",
              "type": "string"
            },
            {
              "name": "bar",
              "type": {
                "type": "int",
                "logicalType": "date"
              }
            }
          ]
        },
        {
          "type": "record",
          "name": "B",
          "fields": [
            {
              "name": "fuzz",
              "type": "boolean",
              "default": true
            }
          ]
        },
        {
          "type": "record",
          "name": "C",
          "fields": [
            {
              "name": "foo",
              "type": "string"
            },
            {
              "name": "bar",
              "type": {
                "type": "int",
                "logicalType": "date"
              }
            }
          ]
        }
      ]
    }
  ]
}

Now produces types:

/* eslint-disable @typescript-eslint/no-namespace */

export type Test = ComOvoenergyKafkaTestEvent.Test;

export namespace ComOvoenergyKafkaTestEvent {
    export const AName = "com.ovoenergy.kafka.test.event.A";
    export interface A {
        foo: string;
        bar: number;
    }
    export const BName = "com.ovoenergy.kafka.test.event.B";
    export interface B {
        /**
         * Default: true
         */
        fuzz: boolean;
    }
    export const CName = "com.ovoenergy.kafka.test.event.C";
    export interface C {
        foo: string;
        bar: number;
    }
    export const TestName = "com.ovoenergy.kafka.test.event.test";
    export interface Test {
        /**
         * Default: "null"
         */
        event: null | ComOvoenergyKafkaTestEvent.A | ComOvoenergyKafkaTestEvent.B | ComOvoenergyKafkaTestEvent.C;
    }
}

Which means event.A is no longer available. We would have event.foo and event.bar available which doesn't represent the schema any more!

Hello 👋

Sorry this is going to be a long text but I prefer giving as much information as possible.
I created a small repository to reproduce the issue here: https://github.com/pylebecq/avro-ts-referenced-types-issue

I have a Java application receiving messages (from an Apache Pulsar cluster, but it does not matter). The data expected in those messages is some Avro encoded data. Some of the schema files reference types defined in other schema files, and schema files contain namespaces.
This is supported when using Avro in Java as we have multiple Java application being able to use these schema files properly.

I need a typescript application to be able to send messages to the previously mentioned Java application.
So I'm trying to use this library to generate typescript definitions from the schema files. This way, when there are some schema change, I can get a quick typescript compilation error if something is not correct.

However, when generating the typescript definitions, some namespaces import statement are missing in typescript, resulting in compilation errors:

I'm using these 4 schema files:

{
  "type": "record",
  "name": "Message",
  "namespace": "my.namespace",
  "fields": [
    {
      "name": "type",
      "type": {
        "type": "enum",
        "name": "MessageType",
        "symbols": ["CreateUser", "UpdateAddress"]
      }
    },
    {
      "name": "CreateUser",
      "type": ["null", "my.namespace.messages.CreateUser"],
      "default": null
    },
    {
      "name": "UpdateAddress",
      "type": ["null", "my.namespace.messages.UpdateAddress"],
      "default": null
    }
  ]
}

{
  "type": "record",
  "name": "CreateUser",
  "namespace": "my.namespace.messages",
  "fields": [
    {
      "name": "userId",
      "type": "string",
      "logicalType": "uuid"
    },
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "address",
      "type": "my.namespace.data.Address"
    }
  ]
}

{
  "type": "record",
  "name": "UpdateAddress",
  "namespace": "my.namespace.messages",
  "fields": [
    {
      "name": "userId",
      "type": "string",
      "logicalType": "uuid"
    },
    {
      "name": "address",
      "type": "my.namespace.data.Address"
    }
  ]
}

{
  "type": "record",
  "name": "Address",
  "namespace": "my.namespace.data",
  "fields": [
    {
      "name": "street",
      "type": "string"
    },
    {
      "name": "zipcode",
      "type": "string"
    },
    {
      "name": "country",
      "type": "string"
    }
  ]
}

If I generated the definitions and try to use them in my project I get these errors during the typescript compilation:

yarn run v1.22.4
$ tsc
src/__generated__/CreateUser.avsc.ts(10,18): error TS2503: Cannot find namespace 'MyNamespaceData'.
src/__generated__/CreateUser.avsc.ts(10,18): error TS4033: Property 'address' of exported interface has or is using private name 'MyNamespaceData'.
src/__generated__/Message.avsc.ts(9,28): error TS2503: Cannot find namespace 'MyNamespaceMessages'.
src/__generated__/Message.avsc.ts(9,28): error TS4033: Property 'CreateUser' of exported interface has or is using private name 'MyNamespaceMessages'.
src/__generated__/Message.avsc.ts(10,31): error TS2503: Cannot find namespace 'MyNamespaceMessages'.
src/__generated__/Message.avsc.ts(10,31): error TS4033: Property 'UpdateAddress' of exported interface has or is using private name 'MyNamespaceMessages'.
src/__generated__/UpdateAddress.avsc.ts(9,18): error TS2503: Cannot find namespace 'MyNamespaceData'.
src/__generated__/UpdateAddress.avsc.ts(9,18): error TS4033: Property 'address' of exported interface has or is using private name 'MyNamespaceData'.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

If we take a look at the generated definitions for CreateUser for example, we get this:

/* eslint-disable @typescript-eslint/no-namespace */

export type CreateUser = MyNamespaceMessages.CreateUser;

export namespace MyNamespaceMessages {
    export const CreateUserName = "my.namespace.messages.CreateUser";
    export interface CreateUser {
        userId: string;
        name: string;
        address: MyNamespaceData.Address;
    }
}

So, this file is referencing type MyNamespaceData.Address but no import is present in this file, resulting in the src/__generated__/CreateUser.avsc.ts(10,18): error TS2503: Cannot find namespace 'MyNamespaceData'. error.

This could be fixed if the generated output contained an import line: import { MyNamespaceData } from './Address.avsc';

However, if we take a look at the generated file for Message, the issue is even worse:

/* eslint-disable @typescript-eslint/no-namespace */

export type Message = MyNamespace.Message;

export namespace MyNamespace {
    export const MessageName = "my.namespace.Message";
    export interface Message {
        type: "CreateUser" | "UpdateAddress";
        CreateUser: null | MyNamespaceMessages.CreateUser;
        UpdateAddress: null | MyNamespaceMessages.UpdateAddress;
    }
}

Here, MyNamespaceMessages is referenced twice with types in two different files. If we add an import line e.g. import { MyNamespaceMessages } from './CreateUser.avsc';, we get another error:

src/__generated__/Message.avsc.ts(12,47): error TS2694: Namespace '"/Users/pierre-yves/dev/avro-ts-referenced-types-issue/src/__generated__/CreateUser.avsc".MyNamespaceMessages' has no exported member 'UpdateAddress'.

Of course, we cannot simply add a second import line for the namespace referencing the second file, otherwise we get this:

src/__generated__/Message.avsc.ts(3,10): error TS2300: Duplicate identifier 'MyNamespaceMessages'.
src/__generated__/Message.avsc.ts(4,10): error TS2300: Duplicate identifier 'MyNamespaceMessages'.

If you have any solution to make this work, please let me know.

If there is definitely no workaround this atm, I'm not sure yet about the best way to fix it but here is an idea:
I noticed the types are exported also without the namespace in their name:

export type Address = MyNamespaceData.Address;

// [...]

If CreateUser was referencing the non-namespaced type Address, maybe imports could be easy to generate:

/* eslint-disable @typescript-eslint/no-namespace */

import { Address } from "./Address.avsc";

export type CreateUser = MyNamespaceMessages.CreateUser;

export namespace MyNamespaceMessages {
  export const CreateUserName = "my.namespace.messages.CreateUser";
  export interface CreateUser {
    userId: string;
    name: string;
    address: Address;
  }
}

And it would also work for the Message type:

/* eslint-disable @typescript-eslint/no-namespace */

import { CreateUser } from "./CreateUser.avsc";
import { UpdateAddress } from "./UpdateAddress.avsc";

export type Message = MyNamespace.Message;

export namespace MyNamespace {
  export const MessageName = "my.namespace.Message";
  export interface Message {
    type: "CreateUser" | "UpdateAddress";
    CreateUser: null | CreateUser;
    UpdateAddress: null | UpdateAddress;
  }
}

However, this might open some issues of name collisions if there are two different namespaces using a same type name.

We use the avro-ts package and we love it, but we want to have the possibility to generate flat typescript structure
without namespaces.
i can submit a pr if needed.

I have the following schema:

cat-millis.avsc

{"logicalType":"timestamp-millis","type":"long"}

Running npx avro-ts cat-millis.avsc gives me

cat-millis.avsc.ts

export type AvroType = number;

Same goes with timestamp-micros. Would it be possible to create something more immediately usable as a TypeScript Date?

See code sandbox:
https://codesandbox.io/s/javascript-forked-k99d2s

If I have union of null and int, the integer value is wrapped within an object. But avro-ts-cli does not seem to generate it properly. It generates null | int (which would be ideal of course, but it does not match Avro).

Ideas?

Hey!

We are implementing the changes to support the Reader Schema in our code but we're stuck. Comparing the schema that we got from Aiven with the generated types, it looks like the type for the event uses the namespace (instead of the name) to differ between the different branches in the union.

export interface Event {
   event:
     | {
         'namespace.obfuscated.this.is.a.public.package.event.CreatedOrUpdate': NAMESPACE.CreatedOrUpdate;
         'namespace.obfuscated.this.is.a.public.package.event.Deleted'?: never;
       }
     | {
         'namespace.obfuscated.this.is.a.public.package.event.CreatedOrUpdate'?: never;
         'namespace.obfuscated.this.is.a.public.package.event.Deleted': NAMESPACE.Deleted;
       };
 }

{
   "name": "event",
   "type": [
        {
              "type": "record",
              "name": "CreatedOrUpdate",
              "namespace": "different.namespace.obfuscated.this.is.a.public.package.Event",
              "fields": ["MANY_FIELDS"]
        },
        {
              "type": "record",
              "name": "Deleted",
              "namespace": "different.namespace.obfuscated.this.is.a.public.package.Event",
              "fields": ["JUST_AN_ID"]
       }
   ]
}

You can notice that if the generated types used the name, if the namespace changed, everything would be fine. But because it is using the namespace (concatenated with the name), if a message is produced with a different namespace we'll never be able to access the correct branch of the union.

A schema definition with hyphens in the name produces invalid TypeScript output, e.g

// Hyphen is in var name invalid
export type Record-value = ComAcmeMyapp.Record-value;

Would be produced from this avsc file:

{
   "type" : "record",
   "name" : "record-value",
   "namespace": "com.acme.myapp",
   "fields" : [
      { "name" : "name" , "type" : "string" }
   ]
}

I believe this is a perfectly valid schema, so perhaps the solution is to remove the invalid characters?

Hey Team,

I've encountered an issue in @ovotech/[email protected] where it doesn't seem to support providing an instance of an Type from the [email protected] library

Here is a minimal reproduction, I would expect the below to work based on the signature of toTypeScript

const x = toTypeScript(
  avro.Type.forSchema({
    type: 'record',
    name: 'ExampleEvent',
    fields: [
      { name: 'id', type: 'string' },
      { name: 'mobile', type: ['null', 'int'], default: null },
      { name: 'name', type: ['null', 'string'], default: null },
    ],
  }),
);

I do have a workaround which is just to convert my avro schema to JSON then parse back to a plain ol JS object

const ts = toTypeScript(
  JSON.parse(
    JSON.stringify(
      avro.Type.forSchema({
        type: 'record',
        name: 'ExampleEvent',
        fields: [
          { name: 'id', type: 'string' },
          { name: 'mobile', type: ['null', 'int'], default: null },
          { name: 'name', type: ['null', 'string'], default: null },
        ],
      }),
    ),
  ),
);

From to the README of avro-ts

Avro Ts attempts to generate the types of the "auto" setting for wrapped unions.

It does this with the following simple logic on these lines of code:

But avsc seem to have a more complex approach as seen here. More specifically in the isAmbiguous() function.

This sometimes leads to failure when trying to encode such message from the types generated by avro-ts

It is not possible to run the CLI without manually installing 'ansi-regex'.
This applies both when running through npx or when installing it.

npx @ovotech/avro-ts-cli ./*.avsc --output-dir .\typescript\
npx: installed 12 in 8.231s
Cannot find module 'ansi-regex'

In avro-ts, if you have an enum type with a fully qualified name (namespace + name), it leaves all of the periods in the namespace when converting to TypeScript, which results in invalid syntax.

e.g.

const MyEnum = {
  name: 'com.foo.bar.MyEnum',
  type: 'enum',
  symbols: ['A', 'B'],
};

const ts = toTypeScript(MyEnum);

console.log(ts);

will return:

export type AvroType = ComFooBarMyEnum;

export type Com.foo.bar.MyEnum = "A" | "B";

What we'd like to see here is ComFooBarMyEnum, which is how this enum's name will get converted when it's referenced in other schemas.

It appears that this behavior originates from this line, and I believe this could be fixed by simply changing it to:

 name: convertName(firstUpperCase(schema.name)),

similar to how the names for Record types are generated.

I'm using a wildcard pattern to generate typescript from my avro files, and have some folder structuring to organize these files. I would like to be able to output to a particular directory but then maintain that structure underneath the output directory. From what I can tell, this isn't possible.

Example:

/contracts/events/example1.avsc
/contracts/events/example2.avsc
/contracts/commands/example3.avsc
/contracts/commands/example4.avsc

avsc-ts /contracts/**/*.avsc --o /dist

Output:

/dist/example1.avsc.ts
/dist/example2.avsc.ts
/dist/example3.avsc.ts
/dist/example4.avsc.ts

It would be nice to have some way of maintaining the source structure and ending up with something like this:

/dist/events/example1.avsc.ts
/dist/events/example2.avsc.ts
/dist/commands/example3.avsc.ts
/dist/commands/example4.avsc.ts

I had to change these lines to do something like:

const fields = mapWithContext(context, schema.fields, (fieldContext, { name, type, doc, ...rest }) => {
    const fullType = ['enum', 'array'].includes(type) ? { name, type, doc, ...rest } : type;
    const converted = convertType({ ...fieldContext, namespace }, fullType);
    return document(converted.context, Type.Prop({ name, type: converted.type, jsDoc: doc }));
  });

Otherwise, the enum/array fields inside my records do not get generated properly

A schema with a union of record types internally serializes to a WrappedUnion type. When passing an object, which conforms to the Avro schema, the library will throw an error, because of the ambiguity.

Pseudo code would be something like:

Avro schema:

{
  "type": "record",
  "namespace": "co.uk.ovotech",
  "fields": [{
    "name": "ambiguous",
    "type": [{
      "name": "record_1",
      "fields": [{"name": "record_field", "type": "int"}]
    }, {
      "name": "record_2",
      "fields": [{"name": "record_field", "type": "int"}, {"name": "record_2_another_field", "type": "int"}]
    }]
  }]
}

The following json will throw, even though it is valid:

{
"ambiguous": {"record_field": 1, "record_2_another_field": 2}
}

The way to work around this is include the type as a key on the level above, i.e. this will not throw:

{
  "co.uk.ovotech.record_2": {
    "ambiguous": {"record_field": 1, "record_2_another_field": 2}
  }
}

However this is not a 1-to-1 mapping of the actual avro schema.

If this is not considered an issue, I think at least it should be documented (if it isn't already).

Assume we have the following code

import { toTypeScript } from '@ovotech/avro-ts';

const RecordWithEnum = {
  type: 'record',
  name: 'Status',
  namespace: 'com.example.avro',
  fields: [
    {
      name: 'statusName',
      type: {
        type: 'enum',
        name: 'StatusName',
        symbols: ['ACTIVE', 'INACTIVE'],
      },
    },
  ],
};

const result = toTypeScript(RecordWithEnum);
console.log(result);

We would expect this to produce the following TypeScript types:

export type Status = ComExampleAvro.Status;

export namespace ComExampleAvro {
    export const StatusNameName = "com.example.avro.StatusName";
    export type StatusName = "ACTIVE" | "INACTIVE";
    export interface Status {
        statusName: ComExampleAvro.StatusName;
    }
}

However, we actually get this:

export type Status = ComExampleAvro.Status;

export namespace ComExampleAvro {
    export const StatusNameName = "com.example.avro.StatusName";
    export const StatusName = "com.example.avro.Status";
    export interface Status {
        statusName: ComExampleAvro.StatusName;
    }
}

The type for StatusName ("ACTIVE" | "INACTIVE") actually disappears and is replaced by a string constant ("com.example.avro.Status").

This happens because the conversion functions for both record and enum types end in a call to namedType, which takes an existing and generates a name constant for it by appending Name to the existing type name and storing it on the namespaces key of the context object.

Normally, this works fine, but (as demonstrated above) there is potential for a naming collision here. If you have two types whose names are identical save for the fact that one of them has Name tacked on to the end (e.g. two types named Foo and FooName), and they are both record or enum types (and will therefore get run through namedType), then the type for FooName will get erased when namedType generates the new name constant for Foo (e.g. const FooName = "com.avro.Foo). More specifically, this will happen because withIdentifier simply merges its new key into the existing namespaces object without checking if the key already exists.

That said, I'm unclear what actually should happen in the case of a naming collision here. It would be easy enough to intervene inside the body of namedType and detect when there's about to be a naming collision, but I don't yet have a good idea how to handle it in a way that won't break some other part of the library.

Hi there,
more a question than an issue. Do you have anyone using this with AWS Kinesis? Is it compatible as far as you know ? Also could we use it with a JSON schema (and without the schema registry) instead of Avro? Any code examples I could look at ?

Thanks 😉

I think this schema

{
  type: 'record',
  name: 'Event',
  namespace: 'com.example.avro',
  fields: [
    {
      name: 'field',
      type: [
        {
          type: 'record',
          name: 'FieldTypeA',
          fields: [
            { name: 'id', type: 'string' },
            {
              name: 'nested',
              type: {
                name: 'NestedFieldType',
                type: 'record',
                fields: [
                  {name: 'id', type: 'string'}
                ]
              }
            }
          ],
        },
        {
          type: 'record',
          name: 'FieldTypeB',
          fields: [
            { name: 'id', type: 'string' },
          ],
        },
        'Field.NestedFieldType'      // <-- Nested type referred to by name
      ],
    }
  ],
}

should result in this Event:

export interface Event {
        field: {
            "com.example.avro.FieldTypeA": ComExampleAvro.FieldTypeA;
            "com.example.avro.FieldTypeB"?: never;
            "com.example.avro.NestedFieldType"?: never;
        } | {
            "com.example.avro.FieldTypeA"?: never;
            "com.example.avro.FieldTypeB": ComExampleAvro.FieldTypeB;
            "com.example.avro.NestedFieldType"?: never;
        } | {
            "com.example.avro.FieldTypeA"?: never;
            "com.example.avro.FieldTypeB"?: never;
            "com.example.avro.NestedFieldType": ComExampleAvro.NestedFieldType;
        };
 }

whereas instead, it results in this:

export interface Event {
        field: ComExampleAvro.FieldTypeA | ComExampleAvro.FieldTypeB | Field.NestedFieldType;
}

If we copy the nested type instead of referring to it by its type name, as per the schema below, we get the expected types

{
  type: 'record',
  name: 'Event',
  namespace: 'com.example.avro',
  fields: [
    {
      name: 'field',
      type: [
        {
          type: 'record',
          name: 'FieldTypeA',
          fields: [
            { name: 'id', type: 'string' },
            {
              name: 'nested',
              type: {
                name: 'NestedFieldType',
                type: 'record',
                fields: [
                  {name: 'id', type: 'string'}
                ]
              }
            }
          ],
        },
        {
          type: 'record',
          name: 'FieldTypeB',
          fields: [
            { name: 'id', type: 'string' },
          ],
        },
        {                                                 // <-- copied
          name: 'NestedFieldType',
          type: 'record',
          fields: [
            {name: 'id', type: 'string'}
          ]
        }
      ],
    }
  ],
}

This is an issue when generating types for the comms_communication_update_v1-value schema which uses a wrapped union for the deliverTo property

We are working with really big avro record with a lot of nested objects, sometimes we want to build the
record from little building blocks, the thing is that the generator exports only the top level record as alias we want to have aliases to all of our building blocks. can i submit a pr that add and option to alias all nested records?

When running the CLI using wildcard for finding avro files, it fails.

$ avro-ts avro-files/*.avsc
Converting Avro to TypeScript

internal/fs/utils.js:230
    throw err;
    ^

Error: ENOENT: no such file or directory, open 'avro-files/*.avsc'
    at Object.openSync (fs.js:457:3)
    at Object.readFileSync (fs.js:359:35)
    at %AppData%\Roaming\npm\node_modules\@ovotech\avro-ts-cli\dist\convert.js:72:55
    at Array.map (<anonymous>)

If I take one file at a time it works fine.

I'm running on Windows.

Im trying to use this library along with Confluent cloud and for that auth is needed.

As far as i can tell you guys dont support any form of auth. Or did i miss something

Hi there,

As i understand it, the message value is nullable

meaning that I've had to do { value: null as unknown as undefined} for now 🤢

Hello 👋

I noticed something that I believe can be improved with default values.
When using this Avro schema:

{
  "type": "record",
  "name": "CreateUser",
  "namespace": "my.namespace.messages",
  "fields": [
    {
      "name": "userId",
      "type": "string",
      "logicalType": "uuid"
    },
    {
      "name": "firstname",
      "type": ["string", "null"],
      "default": "John"
    },
    {
      "name": "lastname",
      "type": ["null", "string"],
      "default": null
    },
    {
      "name": "email",
      "type": "string",
      "default": "[email protected]"
    }
  ]
}

If I try to use it with avsc, I can supply only a userId for this object and this is valid because all other fields have default values:

import avsc from "avsc";
import path from "path";
import { readFileSync } from "fs";

const type = avsc.Type.forSchema(
  JSON.parse(
    readFileSync(path.resolve(__dirname, "../avro/CreateUser.avsc")).toString()
  )
);

// This is valid using avsc because all other fields have default values
const buffer = type.toBuffer({ userId: "123456789" });
console.log(buffer);

However, generating typescript definitions from this schema result of the following:

/* eslint-disable @typescript-eslint/no-namespace */

export type CreateUser = MyNamespaceMessages.CreateUser;

export namespace MyNamespaceMessages {
  export const CreateUserName = "my.namespace.messages.CreateUser";
  export interface CreateUser {
    userId: string;
    name: string;
    email: null | string;
  }
}

This means I must provide all keys when trying to create such an object. Otherwise, if I supply only the userId as I did with avsc:

import { CreateUser } from "./__generated__/CreateUser.avsc";

const createUser: CreateUser = { userId: "123456789" };

I get the following error:

src/index.ts(17,7): error TS2739: Type '{ userId: string; }' is missing the following properties from type 'CreateUser': name, email

Is there any reason why the fields with default values are not generated as optional using ?.

I got this error when trying to build a typescript project using @ovotech/castle:

node_modules/@ovotech/castle/node_modules/@ovotech/avro-kafkajs/dist/AvroTransformBatch.d.ts:4:23 - error TS2497: This module can only be referenced with ECMAScript imports/exports by turning on the 'esModuleInterop' flag and referencing its default export.
4 import * as Long from 'long';

Using node 20.5.1 and typescript 5.1.6 with esModuleInterop enabled in tsconfig.json.

I think its related to this issue. If I replace the offending line:

import * as Long from 'long';

with

import Long = require('long');

The build completes

I've been trying to make example code (https://github.com/ovotech/castle/blob/main/packages/avro-decimal/examples/simple.ts) working and it was throwing an error.

After looking into it it seem that avro-decimal does not work with versions of decimal.js bigger than 10.2.1. Seems like they've changed something in exports and the value no longer recognized as instanceof Decimal.
Downgrading it fixes the issue.

Examples:

• Install [email protected]
• Run following code:

import avro from "avsc";
import { AvroDecimal } from "@ovotech/avro-decimal";
import decimal from "decimal.js";

const { Decimal } = decimal;

const decimalSchema = {
    type: "bytes",
    logicalType: "decimal",
    precision: 16,
    scale: 8,
};

export const DecimalType = avro.Type.forSchema(decimalSchema, {
    logicalTypes: { decimal: AvroDecimal },
});

const encoded = DecimalType.toBuffer(new Decimal("100.01"));
const decoded = DecimalType.fromBuffer(encoded);

console.log("decoded", decoded); // output: decoded 100.01

• Install [email protected]
• Run following code (the only difference is import of decimal.js):

import avro from "avsc";
import { AvroDecimal } from "@ovotech/avro-decimal";
import { Decimal } from "decimal.js";

const decimalSchema = {
    type: "bytes",
    logicalType: "decimal",
    precision: 16,
    scale: 8,
};

export const DecimalType = avro.Type.forSchema(decimalSchema, {
    logicalTypes: { decimal: AvroDecimal },
});

const encoded = DecimalType.toBuffer(new Decimal("100.01"));
const decoded = DecimalType.fromBuffer(encoded);

console.log("decoded", decoded);

That throws this error:

/Users/tim.shilov/IdeaProjects/affiliate-etl/node_modules/.pnpm/@[email protected][email protected]/node_modules/@ovotech/avro-decimal/dist/index.js:63
             throw new Error('expecting Decimal type');
                   ^

Error: expecting Decimal type
    at AvroDecimal._toValue (/Users/tim.shilov/IdeaProjects/affiliate-etl/node_modules/.pnpm/@[email protected][email protected]/node_modules/@ovotech/avro-decimal/dist/index.js:63:20)
    at LogicalType._write (/Users/tim.shilov/IdeaProjects/affiliate-etl/node_modules/.pnpm/[email protected]/node_modules/avsc/lib/types.js:2627:40)
    at Type.toBuffer (/Users/tim.shilov/IdeaProjects/affiliate-etl/node_modules/.pnpm/[email protected]/node_modules/avsc/lib/types.js:658:8)
    at file:///Users/tim.shilov/IdeaProjects/affiliate-etl/apps/etl-engine/src/decimal-test.js:16:29
    at ModuleJob.run (node:internal/modules/esm/module_job:222:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:323:24)
    at async loadESM (node:internal/process/esm_loader:28:7)
    at async handleMainPromise (node:internal/modules/run_main:113:12)

Node.js v20.12.1

This project is super useful for us! Thank you for providing it.

However there is some deprecation warnings being logged when calling toTypeScript which would be nice to get rid of.

DeprecationWarning: 'createPropertySignature' has been deprecated since v4.0.0. Use the appropriate method on 'ts.factory' or the 'factory' supplied by your transformation context instead.
DeprecationWarning: 'createInterfaceDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createTypeAliasDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createModuleDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.

@kafkajs/confluent-schema-registry supports AVRO encoded key but doesn't support extra options for avro.Type.forSchema() function, @ovotech/avro-kafkajs supports extra options but no AVRO encoded key :-(

Hi, I'm very new to Avro. I'm fairly even new in building gradle. I'm trying to learn it. It seems I'm having an issue with generating a Java class from my loanApplication.avsc file. Here's the sample of my avsc file:

{
    "type" : "record",
    "name" : "LoanApplication",
    "doc" : "Data model for loan application connector",
    "fields" : [
        { "name" : "name", "type" : "string" },
        { "name" : "title", "type" : "string" },
        { "name" : "address", "type" : "string" },
        { "name" : "name", "phone" : "string" },
        { "name" : "email", "type" : "string" },
        { "name" : "amount", "type" : [ 
                {
                    "type": "string",
                    "java-class": "java.math.BigDecimal"
                } 
            ]
        }
    ]
}

All seems good in my gradle configuration but it's not generating my Java class. Here's the ./gradlew build --info details, maybe you can help me check what is going on.

> Task :generateAvroProtocol NO-SOURCE
Skipping task ':generateAvroProtocol' as it has no source files and no previous output files.
:generateAvroProtocol (Thread[included builds,5,main]) completed. Took 0.012 secs.
Resolve mutations for :generateAvroJava (Thread[included builds,5,main]) started.
Resolve mutations for :generateAvroJava (Thread[included builds,5,main]) completed. Took 0.0 secs.
:generateAvroJava (Thread[included builds,5,main]) started.

> Task :generateAvroJava FAILED
Build cache key for task ':generateAvroJava' is ffd348588289801380f428a7bb768e1b
Task ':generateAvroJava' is not up-to-date because:
  Task has failed previously.
file or directory '<I REDACTED THIS FILE PATH>\build\generated-main-avro-protocol', not found
Found 1 files
file or directory '<I REDACTED THIS FILE PATH>\build\generated-main-avro-protocol', not found
file or directory '<I REDACTED THIS FILE PATH>\build\generated-main-avro-protocol', not found
file or directory '<I REDACTED THIS FILE PATH>\build\generated-main-avro-protocol', not found
:generateAvroJava (Thread[included builds,5,main]) completed. Took 0.05 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':generateAvroJava'.
> Failed to resolve schema definition file src\main\avro\loanApplicationModel.avsc

Please do tell me if this is an avsc file issue or a gradle issue. Thank you very much.