Serialization and deserialization about io-ts HOT 24 CLOSED

Re. mixed, FWIW in Runtypes I call this type always, since it is dual to never. There is also an open TypeScript issue to make a "native" top type called unknown.

btw isn't mixed = {} | null | undefined

In the absence of void that would be a top type, but since void isn't assignable to that I'd suggest {} | null | void (undefined is assignable to void).

from io-ts.

but I still need to do serialization manually

Isn't a custom toJSON method or the replacer argument of JSON.stringify supposed to solve this problem?

from io-ts.

Kind of, but that's not typed. For example, I want to use the API like this:

// on client
const api = makeClient(Api);
console.log(await api.addObject({name: "test"}); // {id: 123, created: 2017-06-26...}
// on server
app.use(makeServer({
   addObject: async ({name}) => ({id: counter++, created: new Date()})
}));
// communication internally happens with AJAX calls

On both sides, Date types should be compile-time and runtime type-checked that they are supplied as a Date object, and converted from / to something that can be stored in JSON (i.e. an ISO string).

Also the JSON.stringify replacer argument only receives the key and the value within the current object, not the full path to the value or the type I declared it to be within io-ts.

from io-ts.

I basically need a way to define a type with a SerializedType (string), a DeserializedType (Date), a validation/deserialization function (new Date(string), as io-ts can already do) and a serialization function (date.toISOString())

@phiresky Let me understand, are you proposing to add a serialize function to Type?

+export type Serialize<T> = (value: T) => any

// law: serialize . validate = identity
export interface Type<A> {
  readonly _A: A
  readonly name: string
  readonly validate: Validate<A>
+ readonly serialize?: Serialize<A> // this is optional for backward compatibility
}

+export const serialize = <T>(value: T, type: Type<T>): any =>
+  typeof type.serialize !== 'undefined' ? type.serialize(value) : value

export class InterfaceType<P extends Props> implements Type<InterfaceOf<P>> {
  ...
+ this.serialize = value => {
+   const t: { [x: string]: any } = {}
+   for (let k in props) {
+     t[k] = serialize(value[k], props[k])
+   }
+   return t
+ }
}

and then

const DateFromNumber: t.Type<Date> = {
  _A: t._A,
  name: 'DateFromNumber',
  validate: (v, c) =>
    t.number.validate(v, c).chain(n => {
      const d = new Date(n)
      return isNaN(d.getTime()) ? t.failure(n, c) : t.success(d)
    }),
  serialize: v => v.getTime()
}

const Person = t.interface({
  name: t.string,
  birth: DateFromNumber
})

console.log(t.serialize({ name: 'Giulio', birth: new Date(0) }, Person))
// => { name: 'Giulio', birth: 0 }

from io-ts.

@gcanti I see it as dangerous because nesting a type with serialize belong a type without it can generate silent errors.
Restricting those cases by construction (with a different set of interfaces/classes may fix it).

from io-ts.

@sledorze Yeah, I'm just trying to understand what we could do for now. For example, which one do we want to support?

1. a type must be deserializable and may be serializable
2. a type must be deserializable and must be serializable

Do we want the following laws?

1. serialize . deserialize = identity
2. deserialize . serialize = identity

from io-ts.

@phiresky @sledorze I wrote a POC (branch 53). There are some tests too.

I assumed the following properties

1. a type must be both deserializable and serializable (i.e. the serialize function is mandatory)

Laws:

2. deserialize(x).fold(() => x, serialize) = x
3. deserialize(serialize(x)) = Right(x)

(basically Type<T> is similar to Prism<any, T>)

Note that I tried to optimize serialize. When only perfect types (*) are involved is just identity.

Implementation aside, I'd love to hear what you think about the "theory" above.

Here's the list of breaking changes so far

• remove t.map and t.mapWithName (in general doesn't look serializable, needs more investigation)
• remove t.prism (in general doesn't look serializable, needs more investigation)
• remove domain type from t.dictionary (doesn't play nicely with strictFunctionType)
• change Type from interface to class (since TypeScript is structural shouldn't be a breaking change though)
  • remove t._A
• add Type#serialize
• add Type#is (in order to serialize unions and, while we're at it, looks useful anyway)
• remove t.is (now that there's Type#is is misleading)

(*) a "perfect type" (a term I just made off) is a type that just validates (i.e. doesn't change the type of the input like DateFromString does)

from io-ts.

In general, I think that "validate" is not really an accurate name for what the function does: A validator shouldn't transform the data, only check it's correctness. It's really validation and deserialization in one. The way you name it "DateFromNumber" already shows that your t.Type type isn't really a type but a transformer.

So what I was thinking of is similar to this:

validate: (any, Type) → (success | error(...))
deserialize<S extends JSONable, T>: S → T
serialize<T, S extends JSONable>: T → S

With S extends JSONable I mean S is some type that survives JSON.parse(JSON.stringify(s)) without change (only plain objects and primitives). I've almost managed to describe this type recursively in typescript, but not completely.

You would then run validate<> only on the JSONable type, and deserialization could never fail. The validation function should not be able to return any value, instead the input value is returned unmodified (just with a type level cast).

The problem with completely seperating these functions is that validation and deserialization probably often have a lot of common code (as seen in DateFromNumber)

I'm not sure yet if what I'm thinking of can not also be accomplished using separate types, for example
DeserializeDate = DateFromNumber

and

SerializeDate = {validate: (d: any) => d instanceof Date ? success(d.getTime()) : errors(...)}

One immediate problem with this is that this library has any as the input parameter of validate, so there would be no compile-time checking for correctness when using (de)serialization. (And the "validate" function is still not named very well, maybe "parse" would be better)

Generally in other languages (e.g. python) deserialization is a huge can of worms. I like plain objects where it's not a huge problem, but if someone starts to do class instantiation or similar it might get more complicated, possibly out of scope for this project.

from io-ts.

Yeah, names are always a source of confusion :) we could choose any of the follwing pairs

• validate/serialize
• decode/encode
• parse/format

That's why you often see generic names in functional programming: the underline abstraction is Prism (excerpt from monocle-ts)

export class Prism<S, A> {
  constructor(readonly getOption: (s: S) => Option<A>, readonly reverseGet: (a: A) => S) {}
}

Note that S and A are generic so you can have transformations both in getOption and in reverseGet.

However we need better error messages than None so we can change the model to

type E = Array<string> // should be t.ValidationError, I'm using string just to keep things simple

export class Prism<S, A> {
  constructor(readonly getEither: (s: S) => Either<E, A>, readonly reverseGet: (a: A) => S) {}
}

Also we want to be able to extract the handled types, we can use some phantom fields for that

export type ExtractS<T extends Prism<any, any>> = T['_S']
export type ExtractA<T extends Prism<any, any>> = T['_A']

export class Prism<S, A> {
  readonly _S: S
  readonly _A: A
  constructor(readonly getEither: (s: S) => Either<E, A>, readonly reverseGet: (a: A) => S) {}
}

Now that's a model I like!

export const string = new Prism<any, string>(s => (typeof s === 'string' ? right(s) : left(['not a string'])), a => a)

export const number = new Prism<any, number>(s => (typeof s === 'number' ? right(s) : left(['not a number'])), a => a)

export type Props = { [key: string]: Prism<any, any> }

export declare function type<P extends Props>(
  props: P
): Prism<{ [K in keyof P]: ExtractS<P[K]> }, { [K in keyof P]: ExtractA<P[K]> }>


/*
const Person: Prism<{
    name: any;
    age: any;
}, {
    name: string;
    age: number;
}>
*/
const Person = type({
  name: string,
  age: number
})

export type PersonT = ExtractA<typeof Person>
/* same as
type PersonT = {
    name: string;
    age: number;
}
*/

etc... for the other types and combinators.

I have a problem with TypeScript though: Flow has a type named mixed which is perfect for managing untrusted data. TypeScript has just any.

Let's see what happens if I misuse the defined types

Person.getEither(1) // error
Person.getEither({ foo: 'Giulio', bar: 43 }) // error
Person.getEither({ name: 'Giulio', age: 43 }) // ok

Person.getEither(JSON.parse('1')) // NO STATIC ERROR! You will get a runtime error though!

This is so dangerous that I ended up with modeling S = any. Always.

export class Type<A> extends Prism<any, A> {}

Is there a way to preserve the Prism<S, A> model (which I really like) without incurring in unsafe behaviors?

from io-ts.

Another example

export const DateFromNumber = new Prism<number, Date>(s => right(new Date(s)), a => a.getTime())

// prisms compose
export function compose<S, A, B>(ab: Prism<A, B>, sa: Prism<S, A>): Prism<S, B> {
  return new Prism(s => sa.getEither(s).chain(a => ab.getEither(a)), b => sa.reverseGet(ab.reverseGet(b)))
}

export const DateFromAny = compose(DateFromNumber, number)

DateFromAny.getEither('foo') // ok
DateFromNumber.getEither('foo') // static error
DateFromNumber.getEither(JSON.parse('foo')) // no error :(

A simple solution would be never ever pass a value typed any to a runtime type.

But how can we enforce that? Would a custom parse be enough in practice?

function parse(x: any): { __yolo__: never } {
  return JSON.parse(x)
}

DateFromNumber.getEither(parse('foo')) // static error :)
number.getEither(parse('foo')) // ok

from io-ts.

I have a problem with TypeScript though: Flow has a type named mixed which is perfect for managing untrusted data. TypeScript has just any.

Seems like this should work just like flow mixed, no?

type mixed = string | number | boolean | symbol | null | undefined | object

Wouldn't fix erroring out on any inputs though.

from io-ts.

Would a custom parse be enough in practice?

No it isn't. I'm playing with an implementation and there's another, more serious problem.

Let's say we define object and interface as

export declare const object: Prism<any, object>

export declare function type<P extends Props>(props: P): Prism<object, { [K in keyof P]: ExtractA<P[K]> }>

We are used to define Person as

const Person = type({
  name: string,
  age: number
})

This is its type

Prism<object, {
    name: string;
    age: number;
}>

which is unsafe, so we must add object into the mix

const Person = object.compose(type({
  name: string,
  age: number
})

where compose is defined as

export class Prism<S, A> {
  readonly _S: S
  readonly _A: A
  constructor(readonly getEither: (s: S) => Either<E, A>, readonly reverseGet: (a: A) => S) {}
  declare compose<B>(ab: Prism<A, B>, name?: string): Prism<S, B>
}

There's a lot of boilerplate involved and what if I forget to add object? What if Person owns a nested interface and I forget to add object there? I can't take a chance on that happening.

from io-ts.

I've almost managed to describe this type recursively in typescript, but not completely.

this should work

export type JSONObject = { [key: string]: JSONType }
export interface JSONArray extends Array<JSONType> {}
export type JSONType = null | string | number | boolean | JSONArray | JSONObject

export const JSONFromAny = t.recursion<JSONType>('JSONFromAny', Self =>
  t.union([t.null, t.string, t.number, t.boolean, t.array(Self), t.dictionary(Self)])
)

import { tryCatch } from 'fp-ts/lib/Either'

export const JSONFromString = new t.Type<string, JSONType>(
  'JSONFromString',
  JSONFromAny.is,
  (s, c) => tryCatch(() => JSON.parse(s)).fold(() => t.failure(s, c), a => t.success(a)),
  a => JSON.stringify(a)
)

but if someone starts to do class instantiation or similar it might get more complicated, possibly out of scope for this project

it's already possible (deserialization only with current version)

class Person {
  constructor(readonly name: string, readonly surname: string) {}
  getFullName(): string {
    return `${this.name} ${this.surname}`
  }
}

const PersonFromAny = t.mapWithName(
  ([name, surname]) => new Person(name, surname),
  t.tuple([t.string, t.string]),
  'PersonFromAny'
)

console.log(t.validate(['Giulio', 'Canti'], PersonFromAny).fold(() => 'error', p => p.getFullName())) // "Giulio Canti"

or with branch 53 (deserialization and serialization)

const PersonFromTuple = new t.Type<[string, string], Person>(
  'PersonFromTuple',
  (v): v is Person => v instanceof Person,
  ([name, surname]) => t.success(new Person(name, surname)),
  a => [a.name, a.surname]
)

const PersonFromAny = t.tuple([t.string, t.string]).pipe(PersonFromTuple, 'PersonFromAny')

console.log(t.validate(['Giulio', 'Canti'], PersonFromAny).fold(() => 'error', p => p.getFullName())) // "Giulio Canti"
console.log(PersonFromAny.serialize(new Person('Giulio', 'Canti'))) // [ 'Giulio', 'Canti' ]
console.log(PathReporter.report(t.validate(['giulio'], PersonFromAny))) // Invalid value undefined supplied to : PersonFromAny/1: string

from io-ts.

this should work

Sorry, I meant the specific type of running something through JSON.parse(JSON.stringify(x)), not the general type of the possible output of JSON.parse. Meaning mostly the same type, but all functions removed etc. Which is what we get when we pass an object through the network etc.

Here is how I got so far:

declare global {
	interface Number {
		'~type': "number";
	}
	interface String {
		'~type': "string";
	}
	interface Function {
		'~type': "function";
	}
	interface Object {
		'~type': "object";
	}
	interface Date {
		'~type': "date";
	}
}

export type ToJson<T extends any> = ToJson2<T>["result"];
export type ToJson1<T extends any> = { [TName in keyof T]: T[TName]["result"] };
export type ToJson2<T extends { '~type': any }> = {
	result: // result is required for TypeScript to accept recursion
	{
		number: number;
		string: string;
		function: undefined;
		object: ToJson1<{ [k in keyof T]: ToJson2<T[k]> }>;
		date: string;
	}[T["~type"]];
};

function toJson<T>(x: T): ToJson<T> {
	return JSON.parse(JSON.stringify(x));
}
let test = toJson({ a: "foo", b: "bar", c: { kept: 123, removed: () => 123, date: new Date() } });

// typeof test.a === string
// typeof test.c.kept === number
// typeof test.c.removed === undefined
// typeof test.c.date === string

It works ok but has some issues (e.g. toJson<{toJSON: () => 1}> should be "1")

never ever pass a value typed any to a runtime type. But how can we enforce that?

Not sure if there's a way to force the typescript compiler to reject any as an argument somewhere, I tried some things but nothing worked.

I'm playing with an implementation and there's another, more serious problem.

I don't think I understand what exactly is the problem here. a p: Prism<object, ...> should reject non-object arguments like p.getEither(123) (Argument of type '123' is not assignable to parameter of type 'object'.), right?
So you're forced to compose it with a Prism<mixed, object> when passing the result of something returning type mixed = string | number | boolean | symbol | null | undefined | object.

it's already possible (deserialization only with current version)

I know it works for the simple case, but for more complicated stuff there are probably unforseen issues (like maybe you don't want to call the constructor on deserialization, that's why python has two different kinds of constructor).

Though I suppose those things might not be a concern if this library only provides deserializers for primitives / plain objects and assumes all others are passed by the user.

from io-ts.

it's already possible (deserialization only with current version)

As an example: Someone has the Person class you mentioned:

class Person {
  constructor(readonly name: string, readonly surname: string) {}
  getFullName(): string {
    return `${this.name} ${this.surname}`
  }
}

const PersonFromPlain = new t.Type<{name: string, surname: string}, Person>(
  'PersonFromPlain',
  (v): v is Person => v instanceof Person,
  ({name, surname}) => t.success(new Person(name, surname)),
  a => ({a.name, a.surname})
)

But then they have 5 other classes that are serialized basically to the plain object with all the attributes from the class, just missing the functions. So they request a feature addition:

export function instanceType<P>(clazz: new (...args: any[]) => P) {
    return new t.Type<ToJson<P>, P>(
        `Instance<${clazz}>`,
        (v: any): v is P => v instanceof clazz,
        (o: ToJson<P>) => t.success(Object.assign(Object.create(clazz.prototype), o)),
        inst => JSON.parse(JSON.stringify(inst))
    );
}
const PersonFromPlain = instanceType(Person);

const PersonFromAny = t
.type({ name: t.string, surname: t.string })
.pipe(PersonFromPlain, "PersonFromAny"); // this could probably also be made general

const plainPerson = { name: "Giulio", surname: "Canti" };

console.log(
    t.validate(plainPerson, PersonFromAny).fold((l) => "error", p => p.getFullName())
); // "Giulio Canti"
console.log(PersonFromAny.serialize(new Person("Giulio", "Canti"))); // { name: 'Giulio', surname: 'Canti' } */

Which works just the same, except it works for

class Point {
   x: number;
   y: number;
  distance(p2: Point): number;
}

too. Which looks neat, but gets hairy as soon as you have constructors that do stuff, and then you get a barrage of issues from people where it doesn't work for their special case.

from io-ts.

I meant the specific type of running something through JSON.parse(JSON.stringify(x))

Ah I see. First of all thanks for explaining, I'm enjoying these conversations :)

I don't think I understand what exactly is the problem here

Let's say you start with a value typed any (likely from JSON.parse).

If t.type returns a Type<object, something> the first problem is that you can pass that value to Foo

declare function type<P extends t.Props>(props: P): t.Type<object, { [K in keyof P]: t.TypeOf<P[K]> }>

const Foo = type({
  a: t.string,
  b: type({
    c: t.number
  })
})

Foo.validate(1, []) // static error
Foo.validate(JSON.parse('1'), []) // no static error, runtime error

So let's say you define a custom parse, the second problem is that Foo has a nested interface

type mixed = string | number | boolean | symbol | null | undefined | object

declare function parse(x: any): mixed

declare const object: t.Type<mixed, object>

Foo.validate(parse('1'), []) // static error
object.pipe(Foo).validate(parse('1'), []) // ok, will be rejected
object.pipe(Foo).validate(parse('{}'), []) // no static error, runtime error

This is because you should really define Foo as

const Foo = object.pipe(type({
  a: t.string,
  b: object.pipe(type({ // <= !
    c: t.number
  }))
}))

which is verbose and error prone.

So t.type (and all the other combinators) should return Type<any, something> to avoid possible runtime errors.

But then you don't need a custom parse right?

Though I suppose those things might not be a concern if this library only provides deserializers for primitives / plain objects and assumes all others are passed by the user.

👍

from io-ts.

remove domain type from t.dictionary (doesn't play nicely with strictFunctionType)

Just reverted this breaking change, it's ok to pass a domain as usual.

from io-ts.

FYI the version contained in 53 is released in the io-ts@next channel if somebody wants to try it out and give some feedback (in particular about unexpected breaking changes)

from io-ts.

I'm still not sure what you mean. Here's my interpretation:

the following:

const Foo = object.pipe(type({
  a: t.string,
  b: object.pipe(type({ // <= !
    c: t.number
  }))
}));

could easily be mistyped as

const Foo = object.pipe(type({
  a: t.string,
  b: type({
    c: t.number
  })
}))

forgetting the inner object.pipe. But how can that happen, if type() is defined so that the Props input type is not

export type Props = { [key: string]: Prism<any, any> }

but

export type Props = { [key: string]: Prism<mixed, any> }

To force that the input properties are prisms that accept mixed as the input type?

from io-ts.

I'll try to recap what I found so far.

My intention was to keep the S in Prism<S, A> meaningful (i.e. not a catch-all type like any).

However that would lead to unsafety and verbosity.

Hence all basic prisms and those returned by the combinators should be of type Prism<W, A>, where W = any.

If I understand correctly you are proposing W = mixed instead, is it right?

p.s.
btw isn't mixed = {} | null | undefined?

from io-ts.

btw isn't mixed = {} | null | undefined?

No, {} | undefined | null doesn't play well with a dictionary type guard:

type unknown1 = object | number | string | boolean | symbol | undefined | null

type unknown2 = {} | undefined | null

const is1 = (x: unknown1): x is { [key: string]: unknown1 } => x !== null && typeof x === 'object'

const is2 = (x: unknown2): x is { [key: string]: unknown2 } => x !== null && typeof x === 'object'

declare const x1: unknown1

declare const x2: unknown2

if (is1(x1)) {
  x1 // here x1 correctly has type { [key: string]: unknown1 }
}

if (is2(x2)) {
  x2 // here x2 has type {}
}

from io-ts.

I've been looking for a way to deserialize primitive types. After combing through discussions, it doesn't look like this is natively supported (anymore)?

Given that serialization logic is indirectly present in most decoders from io-ts-types, would it make sense to consider providing this as a part of the library (at least for primitive types that define a deserialization logic)? If not, would it be possible to get some pointers on a recommended approach towards this?

Other libs offer it through an explicit conversion flag - convert in this case: https://github.com/hapijs/joi/blob/v14.4.1/API.md#validatevalue-schema-options-callback

Edit: I see that the argument has been discussed in prior comments as well - I'm just not sure where io-ts stands with regards to this at the moment. I'd be glad to take up the issue if we do decide to adopt a direction on this.

from io-ts.

@karthikiyengar this issue is pretty old, keep in mind that the APIs have changed in the meantime.

I'm just not sure where io-ts stands with regards to this at the moment

If you mean something like

convert - when true, attempts to cast values to the required types (e.g. a string to a number). Defaults to true.

the policy is that deserialization / serializaton must be encoded explicitly, see for example NumberFromString in io-ts-types

from io-ts.

@gcanti - I was just going to drop a comment about how bad my API exploration skills were - Just managed to figure it out.

Thanks for the super quick reply and the very enjoyable library!

from io-ts.