salarcode / bois Goto Github PK

private object ReadMember(BinaryReader reader, BoisMemberInfo memInfo, Type memType)
{
if (memInfo.IsNullable &&
!memInfo.IsSupportedPrimitive &&
(!memInfo.IsContainerObject || memInfo.IsStruct))
{
bool isNull = reader.ReadByte() != 0;

            if (isNull)
            {
                return null;
            }
        }

For the pseudo-code below, the classContained in ClassA is null. However, when we deserialize we get an object for ClassContained. Is this by design?

This causes issues, when we have nested contained classes, becuase of the above code.

classA
{
classContained obj;
}

classContained
{
}

main()
{
classA obj = new ClassA();
bois.Serialize(obj);

    deserialobj = bois.Deserialize() ;

}

Hi. I'm using Bois on a new project and am seeing the error System.IO.EndOfStreamException HResult=0x80070026 Message=Unable to read beyond the end of the stream. Source=System.Private.CoreLib on the last line (deserialize). My classes are all simple with a parameterless constructor. What am I doing wrong?

        public static void Main()
        {
            Agent john = Agent.Create(0, "John");
            Agent sarah = Agent.Create(1, "Sarah");
            var r1 = Relationship.Create(1, RelationshipKind.SiblingOf, 5);
            var r2 = Relationship.Create(0, RelationshipKind.SiblingOf, 3);
            john.Relations.Add(r1);
            sarah.Relations.Add(r2);

            BoisSerializer.Initialize<Agent>();
            var boisSerializer = new BoisSerializer();

            var mem = new MemoryStream();
            boisSerializer.Serialize(john, mem);
        
            Agent john2 = boisSerializer.Deserialize<Agent>(mem);

        }

This code throws an exception. (Tested with mono 3.12 on linux, bois 1.8.1 from nuget package)

[Serializable]
public class PrimitiveDictionary1K
{
    public Dictionary<int,PrimitiveType> Dict  { get; set; }
}

[Serializable]
public class PrimitiveType
{
    public bool BoolVar { get; set; }

    public byte ByteVar { get; set; }

    public short ShortVart { get; set; }

    public int IntVar { get; set; }

    public long LongVar { get; set; }

    public char CharVar { get; set; }

    public float FloatVar { get; set; }

    public double DoubleVar { get; set; }
}


class MainClass
{
    public static void Main (string[] args)
    {
        var ser = new BoisSerializer ();
        byte[] data;

        var dict = Create ();

        using (MemoryStream ms = new MemoryStream ()) {
            ser.Serialize (dict, ms);
            data = ms.ToArray ();
        }

        PrimitiveDictionary1K des1;

        using (MemoryStream ms = new MemoryStream (data)) {
            des1=ser.Deserialize<PrimitiveDictionary1K>(ms);
        }

        Console.WriteLine ("length1={0}, length2={1}", dict.Dict.Count, dict.Dict.Count);
    }

    public static PrimitiveDictionary1K Create()
    {
        Random rnd = new Random (100);
        PrimitiveDictionary1K dict = new PrimitiveDictionary1K(){Dict=new Dictionary<int, PrimitiveType> ()};

        for (int i = 0; i < 1024; i++) {
            int key;

            do {
                key = rnd.Next ();
            } while (dict.Dict.ContainsKey (key));

            dict.Dict.Add (key, new PrimitiveType () {
                BoolVar = rnd.Next () % 2 == 0,
                ByteVar = (byte)(rnd.Next () % 256),
                CharVar = (char)(rnd.Next () % 65536),
                DoubleVar = rnd.NextDouble (),
                FloatVar = (float)rnd.NextDouble (),
                IntVar = rnd.Next (),
                LongVar = (long)rnd.Next () * (long)rnd.Next (),
                ShortVart = (short)(rnd.Next () % 65536)
            });
        }
        return dict;
    }

}

System.ArgumentException: Destination array was not long enough. Check destIndex and length, and the array's lower bounds
at System.Array.Copy (System.Array sourceArray, Int32 sourceIndex, System.Array destinationArray, Int32 destinationIndex, Int32 length) [0x001c5] in /tmp/buildd/mono-3.12.0/mcs/class/corlib/System/Array.cs:970
at System.Array.Copy (System.Array sourceArray, System.Array destinationArray, Int32 length) [0x00022] in /tmp/buildd/mono-3.12.0/mcs/class/corlib/System/Array.cs:906
at Salar.Bois.PrimitivesConvertion.ReadInt32 (System.IO.BinaryReader reader, Int32 length) [0x00000] in :0
at Salar.Bois.PrimitivesConvertion.ReadVarInt32Nullable (System.IO.BinaryReader reader) [0x00000] in :0
at Salar.Bois.BoisSerializer.ReadObject (System.IO.BinaryReader reader, System.Type type) [0x00000] in :0
at Salar.Bois.BoisSerializer.ReadMember (System.IO.BinaryReader reader, Salar.Bois.BoisMemberInfo memInfo, System.Type memType) [0x00000] in :0
at Salar.Bois.BoisSerializer.ReadMember (System.IO.BinaryReader reader, System.Type memType) [0x00000] in :0
at Salar.Bois.BoisSerializer.ReadDictionary (System.IO.BinaryReader reader, System.Type memType) [0x00000] in :0
at Salar.Bois.BoisSerializer.ReadMember (System.IO.BinaryReader reader, Salar.Bois.BoisMemberInfo memInfo, System.Type memType) [0x00000] in :0
at Salar.Bois.BoisSerializer.ReadMembers (System.IO.BinaryReader reader, System.Object obj, Salar.Bois.BoisMemberInfo[] memberList, Int32 binaryMemberCount) [0x00000] in :0
at Salar.Bois.BoisSerializer.ReadObject (System.IO.BinaryReader reader, System.Type type) [0x00000] in :0
at Salar.Bois.BoisSerializer.ReadMember (System.IO.BinaryReader reader, Salar.Bois.BoisMemberInfo memInfo, System.Type memType) [0x00000] in :0
at Salar.Bois.BoisSerializer.ReadMember (System.IO.BinaryReader reader, System.Type memType) [0x00000] in :0
at Salar.Bois.BoisSerializer.Deserialize[PrimitiveDictionary1K](System.IO.Stream objectData) [0x00000] in :0
at ConsoleTest.MainClass.Main (System.String[] args) [0x0003b] in /home/sergey/Projects/SerializersBenchmarks/ConsoleTest/Program.cs:96

Hi,
I'm trying to serialize an object that have IDictionary and IList types. But it throws at ReadDictionary method, which it doesn't check if the type is generic or not:

using System.Collections.Concurrent;
using System.Runtime.Serialization;
using Salar.Bois;

[DataContract]
public class Asd
{
    [DataMember]
    public string Member { get; set; }
    public string NotMember { get; set; }
    [DataMember]
    public ExchangeRates Rates { get; set; }
    [DataMember]
    public List<ExchangeRate> RateList { get; set; }
    [DataMember]
    public ConcurrentDictionary<string, ExchangeRate> RateDic { get; set; }
    [DataMember]
    public ExchangeRateCustomList RateCustomList { get; set; }
}

[DataContract]
public class ExchangeRate
{
    [DataMember]
    public double Price { get; set; }
}

[CollectionDataContract(
        Name = "ExchangeRates",
        ItemName = "ExchangeRate",
        KeyName = "Currency",
        ValueName = "Rate")]
public class ExchangeRates : ConcurrentDictionary<string, ExchangeRate>
{
    
}

[CollectionDataContract(
        Name = "ExchangeRateList",
        ItemName = "ExchangeRate",
        ValueName = "Rate")]
public class ExchangeRateCustomList: List<ExchangeRate>
{
    
}

var asd = new Asd
{
    Member = "Member",
    NotMember = "NotMember",
    Rates = new ExchangeRates
    {
        ["TRY"] = new ExchangeRate
        {
            Price = 1.4d
        }
    },
    RateList = new List<ExchangeRate>
    {
        new ExchangeRate{
            Price = 1.4444213d
        }
    },
    RateDic = new ConcurrentDictionary<string, ExchangeRate>
    {
        ["TRY"] = new ExchangeRate
        {
            Price = 1.4d
        }
    },
    RateCustomList = new ExchangeRateCustomList
    {
        new ExchangeRate{
            Price = 1.4444213d
        }
    }
};

var boisSerializer = new BoisSerializer();
using (var ms = new MemoryStream())
{
    boisSerializer.Serialize(asd, ms);
    ms.Position = 0;
    boisSerializer.Deserialize<Asd>(ms).Dump();
} 

Hi,

I have a class with Private Setter, when I tried to serialize it, it threw me "Value cannot be null" exceptions. Is there anyway to serialize and deserialize private setter. I tried BinaryFormatter and it works but the performance is less than ideal. Thanks.

The sample class is as below.

public class PrivateSetPropertyEmployee
{
public PrivateSetPropertyEmployee()
{

    }

    public PrivateSetPropertyEmployee(string name, int age)
    {
        this.Name = name;
        this.Age = age;
    }

    public string Name { private set; get; }

    public int Age { get; private set; }
}

Hi there,

I've encountered an issue where attempting to deserialize a nullable enum throws an error. Here is the minimum reproduction code:

using Salar.Bois;

BoisSerializer.Initialize<TestSerializableClass>();

TestSerializableClass options = new TestSerializableClass()
{
    EnumValue = TestEnum.Value1,
};

BoisSerializer serializer = new BoisSerializer();
byte[] bytes;
using (var memoryStream = new MemoryStream())
{
    serializer.Serialize(options, memoryStream);
    bytes = memoryStream.ToArray();
}

using (var memoryStream = new MemoryStream(bytes))
{
    Console.WriteLine(serializer.Deserialize<TestSerializableClass>(memoryStream).EnumValue);
}

public class TestSerializableClass 
{
    public TestEnum? EnumValue { get; set; }
}

public enum TestEnum 
{
    Value1 = 1,
    Value2,
    Value3,
    Value4,
    Value5,
    Value6,
}

It appears to be an issue with reading from the stream twice (for example, inside the function ReadVarInt32Nullable). It reads once to check if the value is null or embedded, and then again if the value is not embedded or null. I suspect that this will be an issue with any non-embedded nullable integer types.

Seeking backwards by one byte in the stream (e.g. reader.BaseStream.Seek(-1, SeekOrigin.Current);) before entering the ReadInt32Zigzag function (line 156 of NumericSerializers.cs) appears to fix this issue. It would need to be done in every Nullable integer read function.

some time it throws exception as "Parameter Count error" while Dserializing collection. Objects are correct of Business Entity class. any help will be appriciated.
error message as per following
non-negative number required.\r\nparameter name: count

I'm getting a "Constructor on type 'SchemaSet' not found." error on deserialization.

BinaryReader binaryReader = new BinaryReader(inStream);
Salar.Bois.BoisSerializer s = new Salar.Bois.BoisSerializer();
return s.Deserialize<SchemaSet>(binaryReader.BaseStream);

SchemaSet is an interface with two concrete implementation.

Hi Salar, this is not an issue, I just want to know how Salar Bois compares with protobuf. I am looking for a fast serializer to ensure fast initial data loading of a Xamarin App. All other "faster" serializers fail as while they are faster, they take a hell of a time to initialize.

PROTOBUF seems to be an option for me due to its fast initialization time. Though i wonder how Salar Bois compares to it. Thanks!

This code is failed to deserialize

[Serializable]
public class ByteArray64K
{
    public byte[] Arr { get; set; }
}


class MainClass
{
    public static void Main (string[] args)
    {
        var ser = new BoisSerializer ();

        var arr = new ByteArray64K(){Arr = new byte[65536]};
        byte[] data;

        using (MemoryStream ms = new MemoryStream ()) {
            ser.Serialize (arr, ms);
            data = ms.ToArray ();
        }

        ByteArray64K des1;

        using (MemoryStream ms = new MemoryStream (data)) {
            des1=ser.Deserialize<ByteArray64K>(ms);
        }

        Console.WriteLine ("length1={0}, length2={1}", arr.Arr.Length, des1.Arr.Length);
    }
}

Result is: length1=65536, length2=0.

bois version 1.8
mono --version

Mono JIT compiler version 3.10.0 (tarball Wed Nov  5 13:32:50 UTC 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS:           __thread
SIGSEGV:       altstack
Notifications: epoll
Architecture:  x86
Disabled:      none
Misc:          softdebug 
LLVM:          supported, not enabled.
GC:            sgen

Calling into reflection emit will cause a PlatformNotSupportedException. (which is called into upon initialization)

PlatformNotSupportedException: Operation is not supported on this platform. at System.Reflection.Emit.DynamicMethod..ctor (System.String name, System.Type returnType, System.Type[] parameterTypes, System.Reflection.Module m, System.Boolean skipVisibility) [0x00000] in <00000000000000000000000000000000>:0 at Salar.Bois.Types.BoisTypeCompiler.ComputeWriter (System.Type type, Salar.Bois.Types.BoisComplexTypeInfo typeInfo, System.Action1[T] beforeMehodBody, System.Reflection.Module containerModule) [0x00000] in <00000000000000000000000000000000>:0 at Salar.Bois.Types.BoisTypeCache.GetRootTypeComputed (System.Type type, System.Boolean generateReader, System.Boolean generateWriter) [0x00000] in <00000000000000000000000000000000>:0

Possible solution to this is to perform Ahead-Of-Time compilation of the methods & types required.

Support form/to .Net new buffer type should be added.

Serialize from and to:

• ArraySegment<T>
• Span<T>
• ReadOnlySpan<T>
• Memory<T>
• ReadOnlyMemory<T>
• ReadOnlySequence<T>
• etc?

Also support these type as property to be serialized. These types will be treated as simple Arrays like ReadOnlySpan as String and ReadOnlySpan as byte[], but the boundaries (like position) will not be same in deserialization .

I'm getting an error like below,

Exception type: System.InvalidProgramException
Message : Common Language Runtime detected an invalid program.
Stacktrace:
at Computed_CloudTimeSeriesData_Reader_2114922277(BinaryReader reader, Encoding encoding)
at Computed_List`1_Reader_318810521(BinaryReader reader, Encoding encoding)
at Salar.Bois.Types.BoisComputedTypeInfo.InvokeReader[T](BinaryReader reader, Encoding encoding)
at Salar.Bois.BoisSerializer.Deserialize[T](Stream objectData)

I'm using async programming. It throws this error when multiple Deserialize works together at the same time.

Hi,
Serialization working but it shows "Non-negative number required. Parameter name: count" error while de-serialization. in different project same function works great. kindly suggest.

Thanks in Advance
Sunil

Hi,

You are free to close my previous issue #16. I thought everything was working flawlessly until this new curiosity popped up. My understanding is that we can use constructors with parameters in our code as long as we also provide a constructor without parameters, such as these two for my MapCellModel. However, when I save and reload a 5x5 map of cells, the X and Y data have default values while the terrain data retain the values assigned. Any idea why this might occur?

Thank you,
-david

        public MapCellModel(int x, int y, int elevation, TerrainKindEnum terrain)
        {
            this.X = x;
            this.Y = y;
            this.Elevation = elevation;
            this.TerrainKind = terrain;
            this.Things = new ObservableCollection<string>();
        }

        // this version is used by the serializer, Bois, for initialization
        public MapCellModel()
        {
            this.X = -1;
            this.Y = -1;
            this.Elevation = -1;
            this.TerrainKind = TerrainKindEnum.Field;
            this.Things = new ObservableCollection<string>();
        }

Before Saving Map 1_A

MapCell 0_0: x=0 y=0 terrain=Forest
MapCell 0_1: x=0 y=1 terrain=Forest
MapCell 0_2: x=0 y=2 terrain=Forest
MapCell 0_3: x=0 y=3 terrain=Field
MapCell 0_4: x=0 y=4 terrain=Water
MapCell 1_0: x=1 y=0 terrain=Water
MapCell 1_1: x=1 y=1 terrain=Forest
MapCell 1_2: x=1 y=2 terrain=Forest
MapCell 1_3: x=1 y=3 terrain=Forest
MapCell 1_4: x=1 y=4 terrain=Forest
MapCell 2_0: x=2 y=0 terrain=Water
MapCell 2_1: x=2 y=1 terrain=Field
MapCell 2_2: x=2 y=2 terrain=Field
MapCell 2_3: x=2 y=3 terrain=Field
MapCell 2_4: x=2 y=4 terrain=Forest
MapCell 3_0: x=3 y=0 terrain=Forest
MapCell 3_1: x=3 y=1 terrain=Water
MapCell 3_2: x=3 y=2 terrain=Water
MapCell 3_3: x=3 y=3 terrain=Water
MapCell 3_4: x=3 y=4 terrain=Forest
MapCell 4_0: x=4 y=0 terrain=Field
MapCell 4_1: x=4 y=1 terrain=Forest
MapCell 4_2: x=4 y=2 terrain=Field
MapCell 4_3: x=4 y=3 terrain=Water
MapCell 4_4: x=4 y=4 terrain=Forest

After loading saved map 1_A

MapCell 0_0: x=-1 y=-1 terrain=Forest
MapCell 0_1: x=-1 y=-1 terrain=Forest
MapCell 0_2: x=-1 y=-1 terrain=Forest
MapCell 0_3: x=-1 y=-1 terrain=Field
MapCell 0_4: x=-1 y=-1 terrain=Water
MapCell 1_0: x=-1 y=-1 terrain=Water
MapCell 1_1: x=-1 y=-1 terrain=Forest
MapCell 1_2: x=-1 y=-1 terrain=Forest
MapCell 1_3: x=-1 y=-1 terrain=Forest
MapCell 1_4: x=-1 y=-1 terrain=Forest
MapCell 2_0: x=-1 y=-1 terrain=Water
MapCell 2_1: x=-1 y=-1 terrain=Field
MapCell 2_2: x=-1 y=-1 terrain=Field
MapCell 2_3: x=-1 y=-1 terrain=Field
MapCell 2_4: x=-1 y=-1 terrain=Forest
MapCell 3_0: x=-1 y=-1 terrain=Forest
MapCell 3_1: x=-1 y=-1 terrain=Water
MapCell 3_2: x=-1 y=-1 terrain=Water
MapCell 3_3: x=-1 y=-1 terrain=Water
MapCell 3_4: x=-1 y=-1 terrain=Forest
MapCell 4_0: x=-1 y=-1 terrain=Field
MapCell 4_1: x=-1 y=-1 terrain=Forest
MapCell 4_2: x=-1 y=-1 terrain=Field
MapCell 4_3: x=-1 y=-1 terrain=Water
MapCell 4_4: x=-1 y=-1 terrain=Forest

Hi,

I encounter some problem during deserialization when Child object is normal class type and it is null. For e.g, when try to deserialize Employee object below will have error if the Child Object Kid is null.

public class Employee
{
public string Name {get;set}
public int Age {get;set}
public Kid Kid { get; set; }
}

public class Kid
{
public string Name {get;set}
public int Age {get;set}
}

Thanks for taking time to looking into this issue.

Hi, can't find any info on if inheritance is supported? The readme says that interfaces are supported though.

class A
{
public A() { }
}
class B : A
{
public B() { }
int PropertyB {get;set;}
}

var A = new B();

Serializing A seems to fail to store the value of PropertyB.

Is this the intended behavior or do I need to adjust the code in any way?

I am doing some test serializing an object from a client application build in .Net Core 6 framework and calling a server application running in .Net Framework 4.8 and Ihave observed that the same object is not serialized in the same manner under the two frameworks.
Is that a normal behavior ? Do we need to always serialize and deserialize with the same .Net framework or core version ?
Anoter point, is that my client application is a 64bit process while the server application is a 32 bit process : may be the issue is more related to this point...

I love your serialization library - it provides the storage I need, at speed, without the fluff and guff of other libraries.
Got it working with .NET 6.0 but noticed a behaviour - the following line is essential prior to using the Serialize() method

BoisSerializer.Initialize<MyCustomClass>();

As far as I could trace the source code, if this statement is not used then it cannot find the type in BoisTypeCache, and then var computedType = BoisTypeCache.GetRootTypeComputed(type, false, true); returns EnBasicKnownType.Unknown and from there it doesn't know what writer to invoke and the resultant stream is zero bytes. I had consistent behaviour with even primitives and basic objects like List.

I couldn't figure out why and I don't know enough about how DOTNETCORE internals are evolving .. to work out how to fix it but I was relieved to have it working by using the Initialize method. It's not an issue so much as just creating a trail for anyone else who might be searching for the same issue and read that Initialize is documented as optional.

I'm relatively new to c#. Your serializer example was clear to me and this works in my code:

                    var boisSerializer = new BoisSerializer();
                    using (var file = new FileStream(path, FileMode.Create))
                    {
                        boisSerializer.Serialize(myobject, file);
                    }

I don't understand the datastream in your deserializer example though. Do I need to open the file and then call Bois? Is there an approach that mirrors the code above?

I was hoping to use Bois as a generic serializer in the same way I'm able to use the BinaryFormatter. Hand any class to a generic serialization/deserialization routine that casts it as an object. But doing the following in ToBois appears to always return a null byte[]

	private static readonly BoisSerializer Serializer = new BoisSerializer();
	public static byte[] ToBois(this object obj) {
		using (var ms = new MemoryStream()) {
			Serializer.Serialize(obj, ms);
			return ms.ToArray();
		}
	}
	public static object FromBois(this byte[] bytes) {
		return Serializer.Deserialize<object>(new MemoryStream(bytes));
	}

Is this not a valid use-case? I was hoping to use it in a communication class, which doesn't care what the object being handed to it is. It just serializes it for transfer. On the other end, it should deserialize to object, then infer the sent class and handle it accordingly...

In the process of porting from .NET framework to .NET standard and the decision have been made to use netstandrad2.0.

In .NET framework Bois worked fine, but since the nuget only have netstandard2.1, we can't use it anymore.

Is there a specific reason for going with 2.1 or would it be possible to get a 2.0 version?

Seeing that 2.0 works for more targets:
https://docs.microsoft.com/en-us/dotnet/standard/net-standard

Is serializing objects for storing in a SQLite blob column a good use case for Bois? Would that be better/faster than using some kind of ORM?

LZ4Stream is available for use of streams

Bois for c# and specially with LZ4 is absolutely stunning. I will be very interested to use it in order to marshall structure of data between c# process and native process (kind of ipc). But for this I hardly need support to deserialization with C/C++ language.
A library just like mpack for messagePack format.
Is it a possible evolution for your library ? Adding an extention for C/C++ language ?

I'm getting a Out of memory error when serializing a Dictionary with 16 million items.

I'm using .Net Core 6.0 on Windows with 128GB of RAM. The total size before I serialize the dictionary is 18GB.
Any idea why the PICKLE() method is running out of memory? gcLargeObjects is allowed by default with .Net core

For NET6_0_OR_GREATER the types

• DateOnly
• DateOnly?
• TimeOnly
• TimeOnly?

Is currently not supported, would be great with added support for these types. Doesn't look too complex to add them in the existing codebase since they are both representable through an int.