ovotech / castle Goto Github PK
View Code? Open in Web Editor NEWFramework for building Kafka and avro based apps in typescript
License: Other
Framework for building Kafka and avro based apps in typescript
License: Other
Hi!
my build currently says:
error Couldn't find package "@ovotech/ts-compose@^0.16.0" required by "@ovotech/avro-ts" on the "npm" registry.
Looks like @ovotech/ts-compose was unpublished from the npm registry.
Is there a workaround or a fix for this?
We are using blaise to generating kafka messages. In this case the schema is a union.
Code used:
const message =
blaise<ComOvoenergyKafkaHomemovesEvent.ChangeOfTenancyEvent>({
eachMessage: {topic: KAFKA_TOPICS.HOME_MOVES_COT_V1},
avro: {schema: homeMovesEventSchema},
}).message({
value: {
event: {
'com.ovoenergy.kafka.homemoves.event.ChangeOfTenancyCancelled': {
incomingAccountId: 'someIncomingAccount1',
supplyPoint: '123',
},
},
metadata: {
createdAt: now.getTime(),
eventId: 'someEventId1',
traceToken: 'someTraceToken1',
},
},
});
Nothing weird about this, the output:
if you notice blaise adds ChangeOfTenancyCreate with some weird data, where we have never passed. This looks like a memory leak, or data from before
The code currently makes the wrong assumption that there can be only a single schema in a topic, see here:
This makes it impossible to produce messages of multiple schemas in a single topic.
Hi, in your current api you have getSubjectVersionSchema
which maps to GET /subjects/:subject/versions/:version/schema
but I'm missing a getSubjectVersion
which maps to GET /subjects/:subject/versions/:version
, this gets back the schema id (in addition to the schema string) which will be very helpful to me. Thanks.
Hello ๐
I think there is something wrong when using Avro enums with namespaces and trying to generate corresponding typescript types.
I'm using these two schemas:
{
"type": "enum",
"name": "AvroJobStatus",
"namespace": "com.zenaton.jobManager.data",
"symbols": [
"RUNNING_OK",
"RUNNING_WARNING",
"RUNNING_ERROR",
"TERMINATED_COMPLETED",
"TERMINATED_CANCELED"
]
}
{
"type": "record",
"name": "AvroJobStatusUpdated",
"namespace": "com.zenaton.jobManager.messages",
"fields": [
{
"name": "jobId",
"type": "string",
"logicalType": "uuid"
},
{
"name": "jobName",
"type": "string"
},
{
"name": "oldStatus",
"type": [
"null",
"com.zenaton.jobManager.data.AvroJobStatus"
]
},
{
"name": "newStatus",
"type": "com.zenaton.jobManager.data.AvroJobStatus"
}
]
}
Generating types give the following results:
// AvroJobStatus enum
export type AvroType = "RUNNING_OK" | "RUNNING_WARNING" | "RUNNING_ERROR" | "TERMINATED_COMPLETED" | "TERMINATED_CANCELED";
// AvroJobStatusUpdated record
export type AvroJobStatusUpdated = ComZenatonJobManagerMessages.AvroJobStatusUpdated;
export namespace ComZenatonJobManagerMessages {
export const AvroJobStatusUpdatedName = "com.zenaton.jobManager.messages.AvroJobStatusUpdated";
export interface AvroJobStatusUpdated {
jobId: string;
jobName: string;
oldStatus: null | ComZenatonJobManagerData.AvroJobStatus;
newStatus: ComZenatonJobManagerData.AvroJobStatus;
}
}
The issues here are:
export namespace instruction
. It also probably miss a type definition to define its type outside the namespace as the record does.AvroType
instead of AvroJobStatus
.import { ComZenatonJobManagerData as ComZenatonJobManagerDataAvroJobStatus } from "./AvroSerializedData.avsc";
ComZenatonJobManagerDataAvroJobStatus.AvroJobStatus
for the oldStatus
and newStatus
properties.I believe the generated results of these two schemas should be the following (if I did not do any mistake):
// AvroJobStatus enum
export type AvroJobStatus = ComZenatonJobManagerData.AvroJobStatus;
export namespace ComZenatonJobManagerData {
export type AvroJobStatus =
| 'RUNNING_OK'
| 'RUNNING_WARNING'
| 'RUNNING_ERROR'
| 'TERMINATED_COMPLETED'
| 'TERMINATED_CANCELED';
}
// AvroJobStatus record
/* eslint-disable @typescript-eslint/no-namespace */
import { ComZenatonJobManagerData as ComZenatonJobManagerDataAvroJobStatus } from './AvroJobStatus.avsc';
export type AvroJobStatusUpdated = ComZenatonJobManagerMessages.AvroJobStatusUpdated;
export namespace ComZenatonJobManagerMessages {
export const AvroJobStatusUpdatedName =
'com.zenaton.jobManager.messages.AvroJobStatusUpdated';
export interface AvroJobStatusUpdated {
jobId: string;
jobName: string;
oldStatus: null | ComZenatonJobManagerDataAvroJobStatus.AvroJobStatus;
newStatus: ComZenatonJobManagerDataAvroJobStatus.AvroJobStatus;
}
}
These changes will lead to BC breaks though.
What do you think?
Hi,
I have stumbled upon very strange error โ Cannot work out type [object Object]
.
Little bit of digging brought me to following schema:
{
"type" : "error",
"name" : "EpicFailure",
"namespace" : "namespace",
"fields" : [ {
"name" : "code",
"type" : {
"type" : "enum",
"name" : "ErrorCode",
"symbols" : [ "ERROR" ]
}
}, {
"name" : "message",
"type" : "string"
} ]
}
Which is absolutely correct schema, but apparently we get issue with error
type. Here is piece of my initial avdl
:
enum ErrorCode {
ERROR
}
error EpicFailure {
ErrorCode code;
string message;
}
Is possible to add support for error
types too?
Hey Ivan,
Opening a discussion on a recent issue we've at in the BIT team - if you like it I'll open a PR.
We discovered we were producing records without keys due to our limited knowledge of kafka. Kafka brokers block null
keys on a log-compacted topic but in a way that doesn't immediately fail the production so it didn't show up in the logs for a while.
It's a silly mistake too easy to happen. Strengthening the TS types on castle to make this mistake harder to happen: CastleSender
can block undefined
keys, forcing a producer to explicitly define key: null
in case they want a null
key.
In essence, this will block sendFunction(producer, [{value}])
and force to pass a key sendFunction(producer, [{value, key: null}])
, removing risks of making this mistake again.
export declare type CastleSender<
TValue = unknown,
TKey = AvroMessage['key']
> = (producer: AvroProducer, messages: AvroMessage<TValue, TKey>[]) => Promise<RecordMetadata[]>;
into
export declare type CastleSender<
TValue = unknown,
TKey = Exclude<AvroMessage['key'], undefined>
> = (producer: AvroProducer, messages: AvroMessage<TValue, TKey>[]) => Promise<RecordMetadata[]>;
Cheers
I'm currently working with the external example in the repository.
Upon executing the script, I encountered the following output:
DeprecationWarning: 'createPropertySignature' has been deprecated since v4.0.0. Use the appropriate method on 'ts.factory' or the 'factory' supplied by your transformation context instead.
DeprecationWarning: 'createInterfaceDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createTypeAliasDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createModuleDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createImportClause' has been deprecated since v4.0.0. Use the appropriate method on 'ts.factory' or the 'factory' supplied by your transformation context instead.
DeprecationWarning: 'createImportDeclaration' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
/* eslint-disable @typescript-eslint/no-namespace */
import { type MyNamespaceDataAddress as } from "./external-Address";
export type CreateUser = MyNamespaceMessages.CreateUser;
export namespace MyNamespaceMessages {
export const CreateUserSchema = "{\"type\":\"record\",\"name\":\"CreateUser\",\"namespace\":\"my.namespace.messages\",\"fields\":[{\"name\":\"userId\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"address\",\"type\":\"my.namespace.data.Address\"}]}";
export const CreateUserName = "my.namespace.messages.CreateUser";
export interface CreateUser {
userId: string;
name: string;
address: MyNamespaceDataAddress.Address;
}
}
There are two issues identified in the output:
import { type MyNamespaceDataAddress as } from "./external-Address";
is incorrect and should be replaced with import { type MyNamespaceData } from "./external-Address";
.address
in the CreateUser
interface is declared as MyNamespaceDataAddress.Address
, but it should be MyNamespaceData.Address
./* eslint-disable @typescript-eslint/no-namespace */
import { type MyNamespaceData } from "./external-Address";
export type CreateUser = MyNamespaceMessages.CreateUser;
export namespace MyNamespaceMessages {
export const CreateUserSchema = "{\"type\":\"record\",\"name\":\"CreateUser\",\"namespace\":\"my.namespace.messages\",\"fields\":[{\"name\":\"userId\",\"type\":\"string\",\"logicalType\":\"uuid\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"address\",\"type\":\"my.namespace.data.Address\"}]}";
export const CreateUserName = "my.namespace.messages.CreateUser";
export interface CreateUser {
userId: string;
name: string;
address: MyNamespaceData.Address;
}
}
These corrections would ensure the code functions as intended.
Node version: v18.19.0
Typescript version: 5.4.2
Using default tsconfig
Using @ovotech/[email protected]
and executing
npx avro-ts address.avsc.json create-user.avsc.json
where both the schemas were copied from avro-ts/examples
Generated TypeScript files should contain no errors in their imports of external references
I got these generated output with errors in imports of external reference to my.namespace.data.Address
create-user.avsc.json.ts
has
import
which wasn't exported from address.avsc.json.ts
; andimport
is missing alias identifier after the as
; andCreateUser.address
reference with an undefined namespace./* eslint-disable @typescript-eslint/no-namespace */
import { type MyNamespaceDataAddress as } from "./address.avsc.json"; (1), (2)
export type CreateUser = MyNamespaceMessages.CreateUser;
export namespace MyNamespaceMessages {
export const CreateUserName = "my.namespace.messages.CreateUser";
export interface CreateUser {
userId: string;
name: string;
address: MyNamespaceDataAddress.Address; (3)
}
}
address.avsc.json
looks ok and the namespace exported is actually MyNamespaceData
rather than MyNamespaceDataAddress
which is imported by the create-user.avsc.json.ts
above.
/* eslint-disable @typescript-eslint/no-namespace */
export type Address = MyNamespaceData.Address;
export namespace MyNamespaceData {
export const AddressName = "my.namespace.data.Address";
export interface Address {
street: string;
zipcode: string;
country: string;
}
}
It seems, that record field types are not properly nested with a record namespace. Namely having a.avsc
schema:
{
"type": "enum",
"name": "A",
"namespace": "com.example",
"symbols": [
"foo",
"bar"
]
}
and b.avsc
schema:
{
"type": "record",
"name": "B",
"namespace": "com.example",
"fields": [
{
"name": "a",
"type": "A"
}
]
}
Doesn't work - when avro-ts a.avsc b.avsc
is executed, the resulting b.avsc.ts
file has no import
. If, however, "type": "A"
is replaced with "type": "com.example.A"
, import shows up. This seems to be conflicting with Avro specification and also with how avsc
package handles this.
We're using KafkaJS Schema Registry to decode avro messages. It would be nice if avro-ts included the schema used to generate the types so we can decode to the appropriate version of the schema. e.g.
// without schema, have to be explicit about version if the consumer wants a different version from the producer
const registryId = await registry.getRegistryId(MyEvents.MyEventName, schemaVersion);
const schema = await registry.getSchema(registryId);
const decoded = await registry.decode(buffer, { AVRO: { readerSchema: schema } });
// with schema
const decoded = await registry.decode(buffer, { AVRO: { readerSchema: MyEvents.MyEventSchema } });
The Schema
type generated could be equivalent to the following interface:
export interface RawAvroSchema {
name: string;
namespace?: string;
type: 'record';
fields: any[];
}
This would also be handy for registering the schema with Schema Registry in the first place if you were publishing using generated types.
Generators for other languages such as the .NET apache.avro.tools
package do this and it's really helpful.
In avro-ts AVRO enums are only converted to string unions. I'm not sure the exact reason for this, but I was looking for a way to use Typescript enums instead.
For example
{
"name": "com.example.avro.MyEnum",
"type": "enum",
"symbols": ["ACTIVE", "INACTIVE"]
}
will be converted to
export type ComExampleAvroMyEnum = "ACTIVE" | "INACTIVE";
With Typescript enums it will be converted to
export enum ComExampleAvroMyEnum {
ACTIVE = "ACTIVE",
INACTIVE = "INACTIVE"
}
@ivank, let's merge this. It'll be a step to fix https://github.com/ovotech/castle/issues/107.
After this we should update @ovotech/avro-ts dependency for @ovotech/avro-ts-cli, and bug will be fixed.
Originally posted by @gwer in #105 (comment)
Is it possible to merge this PR please ?
I'm looking to use the capabilities added in the PR #139, but see that a new NPM package has not been published.
Can you please make a new release with the latest changes available?
Hi,
Is there any option or workaround to support BigInt? number won't support 64bit id which is used as snowflake id
Avro:
union {null,long} orderId = null;
will generate
orderId?: null | undefined | number
expected
orderId?: null | undefined | BigInt
given the schema
{
"type": "record",
"name": "test",
"namespace": "com.ovoenergy.kafka.test.event",
"fields": [
{
"name": "event",
"type": [
{
"type": "record",
"name": "A",
"fields": [
{
"name": "foo",
"type": "string"
},
{
"name": "bar",
"type": {
"type": "int",
"logicalType": "date"
}
}
]
},
{
"type": "record",
"name": "B",
"fields": [
{
"name": "fuzz",
"type": "boolean",
"default": true
}
]
},
{
"type": "record",
"name": "C",
"fields": [
{
"name": "foo",
"type": "string"
},
{
"name": "bar",
"type": {
"type": "int",
"logicalType": "date"
}
}
]
}
]
}
]
}
Running the library produces (correctly) the types
/* eslint-disable @typescript-eslint/no-namespace */
export type Test = ComOvoenergyKafkaTestEvent.Test;
export namespace ComOvoenergyKafkaTestEvent {
export const AName = "com.ovoenergy.kafka.test.event.A";
export interface A {
foo: string;
bar: number;
}
export const BName = "com.ovoenergy.kafka.test.event.B";
export interface B {
/**
* Default: true
*/
fuzz: boolean;
}
export const CName = "com.ovoenergy.kafka.test.event.C";
export interface C {
foo: string;
bar: number;
}
export const TestName = "com.ovoenergy.kafka.test.event.test";
export interface Test {
event: {
"com.ovoenergy.kafka.test.event.A": ComOvoenergyKafkaTestEvent.A;
"com.ovoenergy.kafka.test.event.B"?: never;
"com.ovoenergy.kafka.test.event.C"?: never;
} | {
"com.ovoenergy.kafka.test.event.A"?: never;
"com.ovoenergy.kafka.test.event.B": ComOvoenergyKafkaTestEvent.B;
"com.ovoenergy.kafka.test.event.C"?: never;
} | {
"com.ovoenergy.kafka.test.event.A"?: never;
"com.ovoenergy.kafka.test.event.B"?: never;
"com.ovoenergy.kafka.test.event.C": ComOvoenergyKafkaTestEvent.C;
};
}
}
Meaning you can access the union members like event.A
, event.B
etc
Changing the schema (not addition of null
into union:
{
"type": "record",
"name": "test",
"namespace": "com.ovoenergy.kafka.test.event",
"fields": [
{
"name": "event",
"default": "null",
"type": [
"null",
{
"type": "record",
"name": "A",
"fields": [
{
"name": "foo",
"type": "string"
},
{
"name": "bar",
"type": {
"type": "int",
"logicalType": "date"
}
}
]
},
{
"type": "record",
"name": "B",
"fields": [
{
"name": "fuzz",
"type": "boolean",
"default": true
}
]
},
{
"type": "record",
"name": "C",
"fields": [
{
"name": "foo",
"type": "string"
},
{
"name": "bar",
"type": {
"type": "int",
"logicalType": "date"
}
}
]
}
]
}
]
}
Now produces types:
/* eslint-disable @typescript-eslint/no-namespace */
export type Test = ComOvoenergyKafkaTestEvent.Test;
export namespace ComOvoenergyKafkaTestEvent {
export const AName = "com.ovoenergy.kafka.test.event.A";
export interface A {
foo: string;
bar: number;
}
export const BName = "com.ovoenergy.kafka.test.event.B";
export interface B {
/**
* Default: true
*/
fuzz: boolean;
}
export const CName = "com.ovoenergy.kafka.test.event.C";
export interface C {
foo: string;
bar: number;
}
export const TestName = "com.ovoenergy.kafka.test.event.test";
export interface Test {
/**
* Default: "null"
*/
event: null | ComOvoenergyKafkaTestEvent.A | ComOvoenergyKafkaTestEvent.B | ComOvoenergyKafkaTestEvent.C;
}
}
Which means event.A
is no longer available. We would have event.foo
and event.bar
available which doesn't represent the schema any more!
Hello ๐
Sorry this is going to be a long text but I prefer giving as much information as possible.
I created a small repository to reproduce the issue here: https://github.com/pylebecq/avro-ts-referenced-types-issue
I have a Java application receiving messages (from an Apache Pulsar cluster, but it does not matter). The data expected in those messages is some Avro encoded data. Some of the schema files reference types defined in other schema files, and schema files contain namespaces.
This is supported when using Avro in Java as we have multiple Java application being able to use these schema files properly.
I need a typescript application to be able to send messages to the previously mentioned Java application.
So I'm trying to use this library to generate typescript definitions from the schema files. This way, when there are some schema change, I can get a quick typescript compilation error if something is not correct.
However, when generating the typescript definitions, some namespaces import statement are missing in typescript, resulting in compilation errors:
I'm using these 4 schema files:
{
"type": "record",
"name": "Message",
"namespace": "my.namespace",
"fields": [
{
"name": "type",
"type": {
"type": "enum",
"name": "MessageType",
"symbols": ["CreateUser", "UpdateAddress"]
}
},
{
"name": "CreateUser",
"type": ["null", "my.namespace.messages.CreateUser"],
"default": null
},
{
"name": "UpdateAddress",
"type": ["null", "my.namespace.messages.UpdateAddress"],
"default": null
}
]
}
{
"type": "record",
"name": "CreateUser",
"namespace": "my.namespace.messages",
"fields": [
{
"name": "userId",
"type": "string",
"logicalType": "uuid"
},
{
"name": "name",
"type": "string"
},
{
"name": "address",
"type": "my.namespace.data.Address"
}
]
}
{
"type": "record",
"name": "UpdateAddress",
"namespace": "my.namespace.messages",
"fields": [
{
"name": "userId",
"type": "string",
"logicalType": "uuid"
},
{
"name": "address",
"type": "my.namespace.data.Address"
}
]
}
{
"type": "record",
"name": "Address",
"namespace": "my.namespace.data",
"fields": [
{
"name": "street",
"type": "string"
},
{
"name": "zipcode",
"type": "string"
},
{
"name": "country",
"type": "string"
}
]
}
If I generated the definitions and try to use them in my project I get these errors during the typescript compilation:
yarn run v1.22.4
$ tsc
src/__generated__/CreateUser.avsc.ts(10,18): error TS2503: Cannot find namespace 'MyNamespaceData'.
src/__generated__/CreateUser.avsc.ts(10,18): error TS4033: Property 'address' of exported interface has or is using private name 'MyNamespaceData'.
src/__generated__/Message.avsc.ts(9,28): error TS2503: Cannot find namespace 'MyNamespaceMessages'.
src/__generated__/Message.avsc.ts(9,28): error TS4033: Property 'CreateUser' of exported interface has or is using private name 'MyNamespaceMessages'.
src/__generated__/Message.avsc.ts(10,31): error TS2503: Cannot find namespace 'MyNamespaceMessages'.
src/__generated__/Message.avsc.ts(10,31): error TS4033: Property 'UpdateAddress' of exported interface has or is using private name 'MyNamespaceMessages'.
src/__generated__/UpdateAddress.avsc.ts(9,18): error TS2503: Cannot find namespace 'MyNamespaceData'.
src/__generated__/UpdateAddress.avsc.ts(9,18): error TS4033: Property 'address' of exported interface has or is using private name 'MyNamespaceData'.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
If we take a look at the generated definitions for CreateUser
for example, we get this:
/* eslint-disable @typescript-eslint/no-namespace */
export type CreateUser = MyNamespaceMessages.CreateUser;
export namespace MyNamespaceMessages {
export const CreateUserName = "my.namespace.messages.CreateUser";
export interface CreateUser {
userId: string;
name: string;
address: MyNamespaceData.Address;
}
}
So, this file is referencing type MyNamespaceData.Address
but no import is present in this file, resulting in the src/__generated__/CreateUser.avsc.ts(10,18): error TS2503: Cannot find namespace 'MyNamespaceData'.
error.
This could be fixed if the generated output contained an import line: import { MyNamespaceData } from './Address.avsc';
However, if we take a look at the generated file for Message
, the issue is even worse:
/* eslint-disable @typescript-eslint/no-namespace */
export type Message = MyNamespace.Message;
export namespace MyNamespace {
export const MessageName = "my.namespace.Message";
export interface Message {
type: "CreateUser" | "UpdateAddress";
CreateUser: null | MyNamespaceMessages.CreateUser;
UpdateAddress: null | MyNamespaceMessages.UpdateAddress;
}
}
Here, MyNamespaceMessages
is referenced twice with types in two different files. If we add an import line e.g. import { MyNamespaceMessages } from './CreateUser.avsc';
, we get another error:
src/__generated__/Message.avsc.ts(12,47): error TS2694: Namespace '"/Users/pierre-yves/dev/avro-ts-referenced-types-issue/src/__generated__/CreateUser.avsc".MyNamespaceMessages' has no exported member 'UpdateAddress'.
Of course, we cannot simply add a second import line for the namespace referencing the second file, otherwise we get this:
src/__generated__/Message.avsc.ts(3,10): error TS2300: Duplicate identifier 'MyNamespaceMessages'.
src/__generated__/Message.avsc.ts(4,10): error TS2300: Duplicate identifier 'MyNamespaceMessages'.
If you have any solution to make this work, please let me know.
If there is definitely no workaround this atm, I'm not sure yet about the best way to fix it but here is an idea:
I noticed the types are exported also without the namespace in their name:
export type Address = MyNamespaceData.Address;
// [...]
If CreateUser
was referencing the non-namespaced type Address, maybe imports could be easy to generate:
/* eslint-disable @typescript-eslint/no-namespace */
import { Address } from "./Address.avsc";
export type CreateUser = MyNamespaceMessages.CreateUser;
export namespace MyNamespaceMessages {
export const CreateUserName = "my.namespace.messages.CreateUser";
export interface CreateUser {
userId: string;
name: string;
address: Address;
}
}
And it would also work for the Message type:
/* eslint-disable @typescript-eslint/no-namespace */
import { CreateUser } from "./CreateUser.avsc";
import { UpdateAddress } from "./UpdateAddress.avsc";
export type Message = MyNamespace.Message;
export namespace MyNamespace {
export const MessageName = "my.namespace.Message";
export interface Message {
type: "CreateUser" | "UpdateAddress";
CreateUser: null | CreateUser;
UpdateAddress: null | UpdateAddress;
}
}
However, this might open some issues of name collisions if there are two different namespaces using a same type name.
We use the avro-ts package and we love it, but we want to have the possibility to generate flat typescript structure
without namespaces.
i can submit a pr if needed.
I have the following schema:
cat-millis.avsc
{"logicalType":"timestamp-millis","type":"long"}
Running npx avro-ts cat-millis.avsc
gives me
cat-millis.avsc.ts
export type AvroType = number;
Same goes with timestamp-micros
. Would it be possible to create something more immediately usable as a TypeScript Date
?
See code sandbox:
https://codesandbox.io/s/javascript-forked-k99d2s
If I have union of null
and int
, the integer value is wrapped within an object. But avro-ts-cli
does not seem to generate it properly. It generates null | int
(which would be ideal of course, but it does not match Avro).
Ideas?
Hey!
We are implementing the changes to support the Reader Schema in our code but we're stuck. Comparing the schema that we got from Aiven with the generated types, it looks like the type for the event uses the namespace (instead of the name) to differ between the different branches in the union.
export interface Event {
event:
| {
'namespace.obfuscated.this.is.a.public.package.event.CreatedOrUpdate': NAMESPACE.CreatedOrUpdate;
'namespace.obfuscated.this.is.a.public.package.event.Deleted'?: never;
}
| {
'namespace.obfuscated.this.is.a.public.package.event.CreatedOrUpdate'?: never;
'namespace.obfuscated.this.is.a.public.package.event.Deleted': NAMESPACE.Deleted;
};
}
{
"name": "event",
"type": [
{
"type": "record",
"name": "CreatedOrUpdate",
"namespace": "different.namespace.obfuscated.this.is.a.public.package.Event",
"fields": ["MANY_FIELDS"]
},
{
"type": "record",
"name": "Deleted",
"namespace": "different.namespace.obfuscated.this.is.a.public.package.Event",
"fields": ["JUST_AN_ID"]
}
]
}
You can notice that if the generated types used the name, if the namespace changed, everything would be fine. But because it is using the namespace (concatenated with the name), if a message is produced with a different namespace we'll never be able to access the correct branch of the union.
A schema definition with hyphens in the name produces invalid TypeScript output, e.g
// Hyphen is in var name invalid
export type Record-value = ComAcmeMyapp.Record-value;
Would be produced from this avsc file:
{
"type" : "record",
"name" : "record-value",
"namespace": "com.acme.myapp",
"fields" : [
{ "name" : "name" , "type" : "string" }
]
}
I believe this is a perfectly valid schema, so perhaps the solution is to remove the invalid characters?
Hey Team,
I've encountered an issue in @ovotech/[email protected]
where it doesn't seem to support providing an instance of an Type
from the [email protected]
library
Here is a minimal reproduction, I would expect the below to work based on the signature of toTypeScript
const x = toTypeScript(
avro.Type.forSchema({
type: 'record',
name: 'ExampleEvent',
fields: [
{ name: 'id', type: 'string' },
{ name: 'mobile', type: ['null', 'int'], default: null },
{ name: 'name', type: ['null', 'string'], default: null },
],
}),
);
I do have a workaround which is just to convert my avro schema to JSON then parse back to a plain ol JS object
const ts = toTypeScript(
JSON.parse(
JSON.stringify(
avro.Type.forSchema({
type: 'record',
name: 'ExampleEvent',
fields: [
{ name: 'id', type: 'string' },
{ name: 'mobile', type: ['null', 'int'], default: null },
{ name: 'name', type: ['null', 'string'], default: null },
],
}),
),
),
);
From to the README of avro-ts
Avro Ts attempts to generate the types of the "auto" setting for wrapped unions.
It does this with the following simple logic on these lines of code:
castle/packages/avro-ts/src/types/wrapped-union.ts
Lines 14 to 17 in 192599e
But avsc
seem to have a more complex approach as seen here. More specifically in the isAmbiguous()
function.
This sometimes leads to failure when trying to encode such message from the types generated by avro-ts
It is not possible to run the CLI without manually installing 'ansi-regex'.
This applies both when running through npx or when installing it.
npx @ovotech/avro-ts-cli ./*.avsc --output-dir .\typescript\
npx: installed 12 in 8.231s
Cannot find module 'ansi-regex'
In avro-ts, if you have an enum type with a fully qualified name (namespace + name), it leaves all of the periods in the namespace when converting to TypeScript, which results in invalid syntax.
e.g.
const MyEnum = {
name: 'com.foo.bar.MyEnum',
type: 'enum',
symbols: ['A', 'B'],
};
const ts = toTypeScript(MyEnum);
console.log(ts);
will return:
export type AvroType = ComFooBarMyEnum;
export type Com.foo.bar.MyEnum = "A" | "B";
What we'd like to see here is ComFooBarMyEnum
, which is how this enum's name will get converted when it's referenced in other schemas.
It appears that this behavior originates from this line, and I believe this could be fixed by simply changing it to:
name: convertName(firstUpperCase(schema.name)),
similar to how the names for Record
types are generated.
I'm using a wildcard pattern to generate typescript from my avro files, and have some folder structuring to organize these files. I would like to be able to output to a particular directory but then maintain that structure underneath the output directory. From what I can tell, this isn't possible.
Example:
/contracts/events/example1.avsc
/contracts/events/example2.avsc
/contracts/commands/example3.avsc
/contracts/commands/example4.avsc
avsc-ts /contracts/**/*.avsc --o /dist
Output:
/dist/example1.avsc.ts
/dist/example2.avsc.ts
/dist/example3.avsc.ts
/dist/example4.avsc.ts
It would be nice to have some way of maintaining the source structure and ending up with something like this:
/dist/events/example1.avsc.ts
/dist/events/example2.avsc.ts
/dist/commands/example3.avsc.ts
/dist/commands/example4.avsc.ts
I had to change these lines to do something like:
const fields = mapWithContext(context, schema.fields, (fieldContext, { name, type, doc, ...rest }) => {
const fullType = ['enum', 'array'].includes(type) ? { name, type, doc, ...rest } : type;
const converted = convertType({ ...fieldContext, namespace }, fullType);
return document(converted.context, Type.Prop({ name, type: converted.type, jsDoc: doc }));
});
Otherwise, the enum/array fields inside my records do not get generated properly
A schema with a union of record types internally serializes to a WrappedUnion
type. When passing an object, which conforms to the Avro schema, the library will throw an error, because of the ambiguity.
Pseudo code would be something like:
Avro schema:
{
"type": "record",
"namespace": "co.uk.ovotech",
"fields": [{
"name": "ambiguous",
"type": [{
"name": "record_1",
"fields": [{"name": "record_field", "type": "int"}]
}, {
"name": "record_2",
"fields": [{"name": "record_field", "type": "int"}, {"name": "record_2_another_field", "type": "int"}]
}]
}]
}
The following json will throw, even though it is valid:
{
"ambiguous": {"record_field": 1, "record_2_another_field": 2}
}
The way to work around this is include the type as a key on the level above, i.e. this will not throw:
{
"co.uk.ovotech.record_2": {
"ambiguous": {"record_field": 1, "record_2_another_field": 2}
}
}
However this is not a 1-to-1 mapping of the actual avro schema.
If this is not considered an issue, I think at least it should be documented (if it isn't already).
Assume we have the following code
import { toTypeScript } from '@ovotech/avro-ts';
const RecordWithEnum = {
type: 'record',
name: 'Status',
namespace: 'com.example.avro',
fields: [
{
name: 'statusName',
type: {
type: 'enum',
name: 'StatusName',
symbols: ['ACTIVE', 'INACTIVE'],
},
},
],
};
const result = toTypeScript(RecordWithEnum);
console.log(result);
We would expect this to produce the following TypeScript types:
export type Status = ComExampleAvro.Status;
export namespace ComExampleAvro {
export const StatusNameName = "com.example.avro.StatusName";
export type StatusName = "ACTIVE" | "INACTIVE";
export interface Status {
statusName: ComExampleAvro.StatusName;
}
}
However, we actually get this:
export type Status = ComExampleAvro.Status;
export namespace ComExampleAvro {
export const StatusNameName = "com.example.avro.StatusName";
export const StatusName = "com.example.avro.Status";
export interface Status {
statusName: ComExampleAvro.StatusName;
}
}
The type for StatusName
("ACTIVE" | "INACTIVE"
) actually disappears and is replaced by a string constant ("com.example.avro.Status"
).
This happens because the conversion functions for both record
and enum
types end in a call to namedType
, which takes an existing and generates a name constant for it by appending Name
to the existing type name and storing it on the namespaces
key of the context object.
Normally, this works fine, but (as demonstrated above) there is potential for a naming collision here. If you have two types whose names are identical save for the fact that one of them has Name
tacked on to the end (e.g. two types named Foo
and FooName
), and they are both record
or enum
types (and will therefore get run through namedType
), then the type for FooName
will get erased when namedType
generates the new name constant for Foo
(e.g. const FooName = "com.avro.Foo
). More specifically, this will happen because withIdentifier
simply merges its new key into the existing namespaces
object without checking if the key already exists.
That said, I'm unclear what actually should happen in the case of a naming collision here. It would be easy enough to intervene inside the body of namedType
and detect when there's about to be a naming collision, but I don't yet have a good idea how to handle it in a way that won't break some other part of the library.
Hi there,
more a question than an issue. Do you have anyone using this with AWS Kinesis? Is it compatible as far as you know ? Also could we use it with a JSON schema (and without the schema registry) instead of Avro? Any code examples I could look at ?
Thanks ๐
I think this schema
{
type: 'record',
name: 'Event',
namespace: 'com.example.avro',
fields: [
{
name: 'field',
type: [
{
type: 'record',
name: 'FieldTypeA',
fields: [
{ name: 'id', type: 'string' },
{
name: 'nested',
type: {
name: 'NestedFieldType',
type: 'record',
fields: [
{name: 'id', type: 'string'}
]
}
}
],
},
{
type: 'record',
name: 'FieldTypeB',
fields: [
{ name: 'id', type: 'string' },
],
},
'Field.NestedFieldType' // <-- Nested type referred to by name
],
}
],
}
should result in this Event:
export interface Event {
field: {
"com.example.avro.FieldTypeA": ComExampleAvro.FieldTypeA;
"com.example.avro.FieldTypeB"?: never;
"com.example.avro.NestedFieldType"?: never;
} | {
"com.example.avro.FieldTypeA"?: never;
"com.example.avro.FieldTypeB": ComExampleAvro.FieldTypeB;
"com.example.avro.NestedFieldType"?: never;
} | {
"com.example.avro.FieldTypeA"?: never;
"com.example.avro.FieldTypeB"?: never;
"com.example.avro.NestedFieldType": ComExampleAvro.NestedFieldType;
};
}
whereas instead, it results in this:
export interface Event {
field: ComExampleAvro.FieldTypeA | ComExampleAvro.FieldTypeB | Field.NestedFieldType;
}
If we copy the nested type instead of referring to it by its type name, as per the schema below, we get the expected types
{
type: 'record',
name: 'Event',
namespace: 'com.example.avro',
fields: [
{
name: 'field',
type: [
{
type: 'record',
name: 'FieldTypeA',
fields: [
{ name: 'id', type: 'string' },
{
name: 'nested',
type: {
name: 'NestedFieldType',
type: 'record',
fields: [
{name: 'id', type: 'string'}
]
}
}
],
},
{
type: 'record',
name: 'FieldTypeB',
fields: [
{ name: 'id', type: 'string' },
],
},
{ // <-- copied
name: 'NestedFieldType',
type: 'record',
fields: [
{name: 'id', type: 'string'}
]
}
],
}
],
}
This is an issue when generating types for the comms_communication_update_v1-value
schema which uses a wrapped union for the deliverTo
property
We are working with really big avro record with a lot of nested objects, sometimes we want to build the
record from little building blocks, the thing is that the generator exports only the top level record as alias we want to have aliases to all of our building blocks. can i submit a pr that add and option to alias all nested records?
When running the CLI using wildcard for finding avro files, it fails.
$ avro-ts avro-files/*.avsc
Converting Avro to TypeScript
internal/fs/utils.js:230
throw err;
^
Error: ENOENT: no such file or directory, open 'avro-files/*.avsc'
at Object.openSync (fs.js:457:3)
at Object.readFileSync (fs.js:359:35)
at %AppData%\Roaming\npm\node_modules\@ovotech\avro-ts-cli\dist\convert.js:72:55
at Array.map (<anonymous>)
If I take one file at a time it works fine.
I'm running on Windows.
Im trying to use this library along with Confluent cloud and for that auth is needed.
As far as i can tell you guys dont support any form of auth. Or did i miss something
Hi there,
As i understand it, the message value is nullable
castle/packages/avro-kafkajs/src/types.ts
Line 37 in 193347c
meaning that I've had to do { value: null as unknown as undefined}
for now ๐คข
Hello ๐
I noticed something that I believe can be improved with default values.
When using this Avro schema:
{
"type": "record",
"name": "CreateUser",
"namespace": "my.namespace.messages",
"fields": [
{
"name": "userId",
"type": "string",
"logicalType": "uuid"
},
{
"name": "firstname",
"type": ["string", "null"],
"default": "John"
},
{
"name": "lastname",
"type": ["null", "string"],
"default": null
},
{
"name": "email",
"type": "string",
"default": "[email protected]"
}
]
}
If I try to use it with avsc, I can supply only a userId
for this object and this is valid because all other fields have default values:
import avsc from "avsc";
import path from "path";
import { readFileSync } from "fs";
const type = avsc.Type.forSchema(
JSON.parse(
readFileSync(path.resolve(__dirname, "../avro/CreateUser.avsc")).toString()
)
);
// This is valid using avsc because all other fields have default values
const buffer = type.toBuffer({ userId: "123456789" });
console.log(buffer);
However, generating typescript definitions from this schema result of the following:
/* eslint-disable @typescript-eslint/no-namespace */
export type CreateUser = MyNamespaceMessages.CreateUser;
export namespace MyNamespaceMessages {
export const CreateUserName = "my.namespace.messages.CreateUser";
export interface CreateUser {
userId: string;
name: string;
email: null | string;
}
}
This means I must provide all keys when trying to create such an object. Otherwise, if I supply only the userId as I did with avsc:
import { CreateUser } from "./__generated__/CreateUser.avsc";
const createUser: CreateUser = { userId: "123456789" };
I get the following error:
src/index.ts(17,7): error TS2739: Type '{ userId: string; }' is missing the following properties from type 'CreateUser': name, email
Is there any reason why the fields with default values are not generated as optional using ?
.
I got this error when trying to build a typescript project using @ovotech/castle
:
node_modules/@ovotech/castle/node_modules/@ovotech/avro-kafkajs/dist/AvroTransformBatch.d.ts:4:23 - error TS2497: This module can only be referenced with ECMAScript imports/exports by turning on the 'esModuleInterop' flag and referencing its default export.
4 import * as Long from 'long';
Using node 20.5.1 and typescript 5.1.6 with esModuleInterop
enabled in tsconfig.json
.
I think its related to this issue. If I replace the offending line:
import * as Long from 'long';
with
import Long = require('long');
The build completes
I've been trying to make example code (https://github.com/ovotech/castle/blob/main/packages/avro-decimal/examples/simple.ts) working and it was throwing an error.
After looking into it it seem that avro-decimal
does not work with versions of decimal.js
bigger than 10.2.1
. Seems like they've changed something in exports and the value no longer recognized as instanceof Decimal
.
Downgrading it fixes the issue.
Examples:
[email protected]
import avro from "avsc";
import { AvroDecimal } from "@ovotech/avro-decimal";
import decimal from "decimal.js";
const { Decimal } = decimal;
const decimalSchema = {
type: "bytes",
logicalType: "decimal",
precision: 16,
scale: 8,
};
export const DecimalType = avro.Type.forSchema(decimalSchema, {
logicalTypes: { decimal: AvroDecimal },
});
const encoded = DecimalType.toBuffer(new Decimal("100.01"));
const decoded = DecimalType.fromBuffer(encoded);
console.log("decoded", decoded); // output: decoded 100.01
[email protected]
decimal.js
):import avro from "avsc";
import { AvroDecimal } from "@ovotech/avro-decimal";
import { Decimal } from "decimal.js";
const decimalSchema = {
type: "bytes",
logicalType: "decimal",
precision: 16,
scale: 8,
};
export const DecimalType = avro.Type.forSchema(decimalSchema, {
logicalTypes: { decimal: AvroDecimal },
});
const encoded = DecimalType.toBuffer(new Decimal("100.01"));
const decoded = DecimalType.fromBuffer(encoded);
console.log("decoded", decoded);
That throws this error:
/Users/tim.shilov/IdeaProjects/affiliate-etl/node_modules/.pnpm/@[email protected][email protected]/node_modules/@ovotech/avro-decimal/dist/index.js:63
throw new Error('expecting Decimal type');
^
Error: expecting Decimal type
at AvroDecimal._toValue (/Users/tim.shilov/IdeaProjects/affiliate-etl/node_modules/.pnpm/@[email protected][email protected]/node_modules/@ovotech/avro-decimal/dist/index.js:63:20)
at LogicalType._write (/Users/tim.shilov/IdeaProjects/affiliate-etl/node_modules/.pnpm/[email protected]/node_modules/avsc/lib/types.js:2627:40)
at Type.toBuffer (/Users/tim.shilov/IdeaProjects/affiliate-etl/node_modules/.pnpm/[email protected]/node_modules/avsc/lib/types.js:658:8)
at file:///Users/tim.shilov/IdeaProjects/affiliate-etl/apps/etl-engine/src/decimal-test.js:16:29
at ModuleJob.run (node:internal/modules/esm/module_job:222:25)
at async ModuleLoader.import (node:internal/modules/esm/loader:323:24)
at async loadESM (node:internal/process/esm_loader:28:7)
at async handleMainPromise (node:internal/modules/run_main:113:12)
Node.js v20.12.1
This project is super useful for us! Thank you for providing it.
However there is some deprecation warnings being logged when calling toTypeScript
which would be nice to get rid of.
DeprecationWarning: '
createPropertySignature
' has been deprecated since v4.0.0. Use the appropriate method on 'ts.factory' or the 'factory' supplied by your transformation context instead.
DeprecationWarning: 'createInterfaceDeclaration
' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createTypeAliasDeclaration
' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
DeprecationWarning: 'createModuleDeclaration
' has been deprecated since v4.8.0. Decorators are no longer supported for this function. Callers should switch to an overload that does not accept a 'decorators' parameter.
@kafkajs/confluent-schema-registry supports AVRO encoded key but doesn't support extra options for avro.Type.forSchema() function, @ovotech/avro-kafkajs supports extra options but no AVRO encoded key :-(
Hi, I'm very new to Avro. I'm fairly even new in building gradle. I'm trying to learn it. It seems I'm having an issue with generating a Java class from my loanApplication.avsc
file. Here's the sample of my avsc file:
{
"type" : "record",
"name" : "LoanApplication",
"doc" : "Data model for loan application connector",
"fields" : [
{ "name" : "name", "type" : "string" },
{ "name" : "title", "type" : "string" },
{ "name" : "address", "type" : "string" },
{ "name" : "name", "phone" : "string" },
{ "name" : "email", "type" : "string" },
{ "name" : "amount", "type" : [
{
"type": "string",
"java-class": "java.math.BigDecimal"
}
]
}
]
}
All seems good in my gradle configuration but it's not generating my Java class. Here's the ./gradlew build --info
details, maybe you can help me check what is going on.
> Task :generateAvroProtocol NO-SOURCE
Skipping task ':generateAvroProtocol' as it has no source files and no previous output files.
:generateAvroProtocol (Thread[included builds,5,main]) completed. Took 0.012 secs.
Resolve mutations for :generateAvroJava (Thread[included builds,5,main]) started.
Resolve mutations for :generateAvroJava (Thread[included builds,5,main]) completed. Took 0.0 secs.
:generateAvroJava (Thread[included builds,5,main]) started.
> Task :generateAvroJava FAILED
Build cache key for task ':generateAvroJava' is ffd348588289801380f428a7bb768e1b
Task ':generateAvroJava' is not up-to-date because:
Task has failed previously.
file or directory '<I REDACTED THIS FILE PATH>\build\generated-main-avro-protocol', not found
Found 1 files
file or directory '<I REDACTED THIS FILE PATH>\build\generated-main-avro-protocol', not found
file or directory '<I REDACTED THIS FILE PATH>\build\generated-main-avro-protocol', not found
file or directory '<I REDACTED THIS FILE PATH>\build\generated-main-avro-protocol', not found
:generateAvroJava (Thread[included builds,5,main]) completed. Took 0.05 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':generateAvroJava'.
> Failed to resolve schema definition file src\main\avro\loanApplicationModel.avsc
Please do tell me if this is an avsc file issue or a gradle issue. Thank you very much.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.