Froto.Roslyn is a .NET 4.5 project. Using the latest protobuf-net, version 2.0.0.668, for nuget, it adds the dll from net40. Unfortunately, it looks to be compiled for .NET 3.5.

Roslyn emit then complains about mscorlib 3.5 missing from my system.

If I switch to use the dll from the net20, emit is successful, but I get a bunch of warnings.

(85,20): warning CS1701: Assuming assembly reference 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' used by 'protobuf-net' matches identity 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' of 'mscorlib', you may need to supply runtime policy

How do I get rid of these warnings?

Hi I saw your package the other day going around nuget. First of all I think is a Great Idea.

Maybe I am a bit confused on what you are trying to achive, let me know if I am far away from your concept. Are you trying to do the compilation or add support to F# of protocol buffers?.

I am interested in adding support for compilation for C# projects. On Proto files compilation I have done this work using node.js+protoc scripts but on my project I still have to add the things manually to the project. I think going through roslyn sounds great idea.

I have just started doing also some extensions for roslyn so I would be interested on helping you on this specially for C# projects. If you want a hand lets think how can I help, hopefully we can find a way to make it work for CLI bassed languages (F# and C#).

    type IMessage =
        abstract member SerializedLength: unit -> uint32
        abstract member Serialize: ZeroCopyBuffer -> unit
        abstract member ReadFrom: ZeroCopyBuffer -> unit

Currently when you are generating code you have to make a dependency on Froto.serialization which if fine, its the serialization lib after all, but then you also have to include Froto.codec too as you have to have a concrete interface or rely completly on SRTP which is not always easy and gets rapidly more complex if you have to add reflection into that mix. Having the interface as a hard reference in Froto.Serialization means you can more or less eliminate Froto.Codec from the type provider implementation and any custom code gen, which is what I have been doing. Ive been whittling away Froto.Codec but now Ive come to the point where the basic interface implementation of IMessage needs to have a home.

I pulled the lastest code from master branch, and opened the solution in VS 2015. All the projects appear unloaded, and when I try to reload them it does nothing. When I try to edit the csproj files via Solution Explorer, it complains "file is already opened by another project".

Any ideas?

I am trying to use blizzards sc2 proto files provided here: https://github.com/Blizzard/s2client-proto
The first problem is the use of the not supported "import" statement. I concatenated the content of the files to fix this.

Then i got the following error in debug.proto: "Specified method is not supported"
in this section:

// Issue various useful commands to the game engine.
message DebugCommand {
  oneof command {
    DebugDraw draw = 1;
    //gamestatemessage is broken with the Froto Typeprovider
    //DebugGameState game_state = 2; 
    DebugCreateUnit create_unit = 3;
    DebugKillUnit kill_unit = 4;
    DebugTestProcess test_process = 5;
    DebugSetScore score = 6;                // Useful only for single-player "curriculum" maps.
    DebugEndGame end_game = 7;
    DebugSetUnitValue unit_value = 8;
    DebugChat chat = 9;                     // TODO.
  }
}

enum DebugGameState {
  show_map = 1;
  control_enemy = 2;
  food = 3;
  free = 4;
  all_resources = 5;
  god = 6;
  minerals = 7;
  gas = 8;
  cooldown = 9;
  tech_tree = 10;
  upgrade = 11;
  fast_build = 12;
}

Any ideas why and how to fix this?

Issue is already logged elsewhere:
dotnet/sourcelink#179
dotnet/sourcelink#155

  Microsoft.Build.Tasks.Git.targets(20, 5): [MSB4018] The "Microsoft.Build.Tasks.Git.LocateRepository" task failed unexpectedly.
System.BadImageFormatException: Method has no body
File name: 'LibGit2Sharp'
  at LibGit2Sharp.Core.Proxy.git_buf_free (LibGit2Sharp.Core.Handles.GitBuf buf) [0x00000] in <06ab184b4ccd4259a7a7af2756f77645>:0 
  at LibGit2Sharp.Core.Handles.GitBuf.Dispose () [0x00000] in <06ab184b4ccd4259a7a7af2756f77645>:0 
  at LibGit2Sharp.Core.Proxy.ConvertPath (System.Func`2[T,TResult] pathRetriever) [0x0002e] in <06ab184b4ccd4259a7a7af2756f77645>:0 
  at LibGit2Sharp.Core.Proxy.git_repository_discover (LibGit2Sharp.Core.FilePath start_path) [0x00005] in <06ab184b4ccd4259a7a7af2756f77645>:0 
  at LibGit2Sharp.Repository.Discover (System.String startingPath) [0x00006] in <06ab184b4ccd4259a7a7af2756f77645>:0 
  at Microsoft.Build.Tasks.Git.GitOperations.LocateRepository (System.String directory) [0x00000] in <859aea960c5c4805ad7e678061631919>:0 
  at Microsoft.Build.Tasks.Git.RepositoryTasks.LocateRepository (Microsoft.Build.Tasks.Git.LocateRepository task) [0x00007] in <859aea960c5c4805ad7e678061631919>:0 
  at Microsoft.Build.Tasks.Git.LocateRepository.Execute () [0x00038] in <77223f90773a4c0e88a42a7f38595356>:0 
  at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute () [0x00023] in <9d534585aab94a68ab40b57567b8fbbe>:0 
  at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask (Microsoft.Build.BackEnd.ITaskExecutionHost taskExecutionHost, Microsoft.Build.BackEnd.Logging.TaskLoggingContext taskLoggingContext, Microsoft.Build.BackEnd.TaskHost taskHost, Microsoft.Build.BackEnd.ItemBucket bucket, Microsoft.Build.BackEnd.TaskExecutionMode howToExecuteTask) [0x001f6] in <9d534585aab94a68ab40b57567b8fbbe>:0 

Both FAKE and msbuild directly fail to build the froto project from a fresh git clone due to a missing library.

Resolution:

Add nuget System.Collections.Immutable to paket.dependencies.
Add System.Collection.Immutable to project Froto.Roslyn and Froto.Roslyn.Test.

travis ci failing to build TypeProvider.Tests

#74

Hello everyone,
Many thanks for this library, I just start trying to use it for my personal project. In the Serializer module, I observe that there are dehydrating functions for only signed arithmetic types. So how can I encode unsigned types?

Functions in the Encode module assume that "0" or "default" values should be elided (not serialized). However, in proto2, required fields must appear in the serialization stream, even if they have the default value.

Please add Encode module functions to support required fields. Encode.fromNondefaultedVarint exists, but none of the other data types are supported.

Froto appears to only parse a subset of the proto2 language, and allows a looser syntax. Requesting the parser be tightened up to match the official protoc compiler: handle the complete syntax and produce errors in the same cases, in order to provide better compatibility.

• Support enum as a top-level construct (outside any message)
• Support "group" keyword in a message
• Support "one of" keyword in a message
• Support "map" keyword in a message
• Support "extensions" keyword in a message
• Support "reserved" keyword in a message
• Support options on enum, enumItem
• Support alternate option syntax; e.g. option (myOpt) = { fieldA : 1, fieldB: "test" };
• Tighten up intLit to support proto2 octal (code uses pint32, which parses "0o77", but not "077")
• Same thing for strLit; can't recall the specifics, but don't think Froto handles all proto2 escape sequences
• Parse out strong literal types for constants (intLit, floatLit, boolLit, and strLit), rather than parsing all of those as string values.
• Tighten up identifier parser; can't recall the specifics, but IIRC Froto allows invalid identifiers

I've been hacking my own F#/FParsec parser and am almost done: have support for all the above w/tests, and only a few productions remaining (iirc, "service" and "extends"). Using this as a learning experience. Once I've implemented proto3 syntax (using an FParsec state and rejection of invalid items when in proto3 mode), I should probably circle back and fork Froto to help implement the above, if you're open to contributions :-).

type ProtoField (rule:ProtoFieldRule, tp:string, name:string, position:int32, options:ProtoOption list option) =

position here is defined as in in32, elsewhere the position is declared as a FieldNum which is an uint32:

and FieldNum = uint32

Proto actually declares the numbering as a uint16 beginning at 1.

The idea is to create a froto.exe that is similar to protoc.exe in that it reads in a .proto file and generates source code. There may be a few different source code generators, including:

• Roslyn based C# generator for proto2 that generates code for protobuf-net.
• F# code generator via FsAst that support proto3 and builds on top of the Google.Protobuf library. It will use F# Records and Options.
• Roslyn based C# generator for proto3 that generates code for Google.Protobuf library, but differs from the protoc.exe generated code in a few ways.

Just for information, the protoc.exe in Google.Protobuf 3.0.0-beta2 has this help:

PS C:\fs\froto> .\packages\Google.Protobuf\tools\protoc.exe --version
libprotoc 3.0.0
PS C:\fs\froto> .\packages\Google.Protobuf\tools\protoc.exe --help
Usage: C:\fs\froto\packages\Google.Protobuf\tools\protoc.exe [OPTION] PROTO_FILES
Parse PROTO_FILES and generate output based on the options given:
  -IPATH, --proto_path=PATH   Specify the directory in which to search for
                              imports.  May be specified multiple times;
                              directories will be searched in order.  If not
                              given, the current working directory is used.
  --version                   Show version info and exit.
  -h, --help                  Show this text and exit.
  --encode=MESSAGE_TYPE       Read a text-format message of the given type
                              from standard input and write it in binary
                              to standard output.  The message type must
                              be defined in PROTO_FILES or their imports.
  --decode=MESSAGE_TYPE       Read a binary message of the given type from
                              standard input and write it in text format
                              to standard output.  The message type must
                              be defined in PROTO_FILES or their imports.
  --decode_raw                Read an arbitrary protocol message from
                              standard input and write the raw tag/value
                              pairs in text format to standard output.  No
                              PROTO_FILES should be given when using this
                              flag.
  -oFILE,                     Writes a FileDescriptorSet (a protocol buffer,
    --descriptor_set_out=FILE defined in descriptor.proto) containing all of
                              the input files to FILE.
  --include_imports           When using --descriptor_set_out, also include
                              all dependencies of the input files in the
                              set, so that the set is self-contained.
  --include_source_info       When using --descriptor_set_out, do not strip
                              SourceCodeInfo from the FileDescriptorProto.
                              This results in vastly larger descriptors that
                              include information about the original
                              location of each decl in the source file as
                              well as surrounding comments.
  --dependency_out=FILE       Write a dependency output file in the format
                              expected by make. This writes the transitive
                              set of input file paths to FILE
  --error_format=FORMAT       Set the format in which to print errors.
                              FORMAT may be 'gcc' (the default) or 'msvs'
                              (Microsoft Visual Studio format).
  --print_free_field_numbers  Print the free field numbers of the messages
                              defined in the given proto files. Groups share
                              the same field number space with the parent
                              message. Extension ranges are counted as
                              occupied fields numbers.

  --plugin=EXECUTABLE         Specifies a plugin executable to use.
                              Normally, protoc searches the PATH for
                              plugins, but you may specify additional
                              executables not in the path using this flag.
                              Additionally, EXECUTABLE may be of the form
                              NAME=PATH, in which case the given plugin name
                              is mapped to the given executable even if
                              the executable's own name differs.
  --cpp_out=OUT_DIR           Generate C++ header and source.
  --csharp_out=OUT_DIR        Generate C# source file.
  --java_out=OUT_DIR          Generate Java source file.
  --javanano_out=OUT_DIR      Generate Java Nano source file.
  --js_out=OUT_DIR            Generate JavaScript source.
  --objc_out=OUT_DIR          Generate Objective C header and source.
  --python_out=OUT_DIR        Generate Python source file.
  --ruby_out=OUT_DIR          Generate Ruby source file.

The following definition results in a parse failure:

syntax = "proto3";

message SampleMessage {
    oneof test_oneof {
      string name = 4;
      int32 age = 9;
    }
}

Error in Ln: 4 Col: 22
    oneof test_oneof {
                     ^
Expecting: newline, tab, ' ', '/*', '//' or '='

This sample from the proto3 user guide,

syntax = "proto3";

message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 result_per_page = 3;
  enum Corpus {
    UNIVERSAL = 0;
    WEB = 1;
    IMAGES = 2;
    LOCAL = 3;
    NEWS = 4;
    PRODUCTS = 5;
    VIDEO = 6;
  }
  Corpus corpus = 4;
}

Fails with error

System.FormatException: Error in Ln: 7 Col: 15
  enum Corpus {
              ^
Expecting: newline, tab, ' ', '/*', '//' or '='

   at Froto.Parser.Parse.resultOrFail[a,b](ParserResult`2 parserResult)
   at Froto.Parser.Parse.fromFileWithParser[a](FSharpFunc`2 parser, String fileName)
   at Froto.Parser.Parse.fromFile(String fileName)
   at <StartupCode$FSI_0006>.$FSI_0006.main@()
Stopped due to error```

Hi,

Not really an issue... but I was wondering what library call was used to autogenerate https://github.com/ctaggart/froto/blob/master/Serialization.Test/ExampleProtoRecord.fs ?

I have a problem in using the function hydrateRepeated to deserialize repeated fields where fields are message themselves. Concretely, I have a Chunk message with a repeated field of Instruction message (the protobuf file can be referenced here, to deserialize the messages, I have used this code. I observe that each Instruction is correctly deserialized, but the output list of instructions contains only null objects.
(that means each message is correctly deserialized, but the deserialized result is not copied into the output list!?).

Looking into the source code of hydrateRepeated, I observe that element is an immutable value (though the argument passed to hydrater is mutable)

let hydrateRepeated<'a> (hydrater:'a ref -> RawField -> unit) propRef rawField =
    let element = Unchecked.defaultof<'a>
    hydrater (ref element) rawField
    propRef := element :: !propRef

so that might be the reason why the deserialized value is not copied into the list. A trivial example is the following code:

let increase (i:int ref) : int =
  i := !i + 1
  !i

let i = 3
increase (ref i)
i

The value of i is always 3. For the function hydrateRepeated, IMHO, the following fix might work

let hydrateRepeated<'a> (hydrater:'a ref -> RawField -> unit) propRef rawField =
    let element = ref Unchecked.defaultof<'a>
    hydrater element rawField
    propRef := !element :: !propRef

Now that froto.exe has been created #11. We need to get the existing protobuf-net proto2 code generation working.

The following .proto (created with protobuf-net)

import "bcl.proto"; // schema for protobuf-net's handling of core .NET types

message ClassA {
   optional string ID;
   optional string Name = 1 [default = aa];
   optional bcl.Guid mGuid = 2 [default = 00000000-0000-0000-0000-000000000000];
}

crashes with this message:

Error in Ln: 6 Col: 51
  optional bcl.Guid mGuid = 3 [default = 00000000-0000-0000-0000-000000000000];
                                                 ^
Expecting: newline, tab, ' ', ',', '/*', '//' or ']'

I try to parse it like this: FSharpList<Ast.PStatement> s = Parse.fromString(proto);

Be able to parse grpc documents like https://github.com/googleapis/googleapis/blob/master/google/pubsub/v1/pubsub.proto

service Publisher {
  rpc CreateTopic(Topic) returns (Topic) {
    option (google.api.http) = { put: "/v1/{name=projects/*/topics/*}" body: "*" };
  }

  rpc Publish(PublishRequest) returns (PublishResponse) {
    option (google.api.http) = { post: "/v1/{topic=projects/*/topics/*}:publish" body: "*" };
  }

  rpc GetTopic(GetTopicRequest) returns (Topic) {
    option (google.api.http) = { get: "/v1/{topic=projects/*/topics/*}" };
  }

  rpc ListTopics(ListTopicsRequest) returns (ListTopicsResponse) {
    option (google.api.http) = { get: "/v1/{project=projects/*}/topics" };
  }

  // Lists the name of the subscriptions for this topic.
  rpc ListTopicSubscriptions(ListTopicSubscriptionsRequest) returns (ListTopicSubscriptionsResponse) {
    option (google.api.http) = { get: "/v1/{topic=projects/*/topics/*}/subscriptions" };
  }

  rpc DeleteTopic(DeleteTopicRequest) returns (google.protobuf.Empty) {
    option (google.api.http) = { delete: "/v1/{topic=projects/*/topics/*}" };
  }
}

It appears System.Reflection.Metadata is missing from the project references for Froto.Compiler, this package is available in nuget.

Resolution

Update paket.dependencies to include nuget System.Reflection.Metadata
Update paket.references in Froto.Compiler to include System.Reflection.Metadata

In the Readme it's mentioned that Froto.Core package already exists, but I can't find it on NuGet. Am I missing something or it has been removed?

I'm interested in using this package since I've started working on a type provider for Protocol Buffers. I use Froto.Parser already, but it would be nice to be able to reuse serialization code as well.

Let me know if I can help somehow.

I'm going to move the project files over to .NET Core projects. I'll keep the targets the same. We can then discuss what should be .NET Standard.

@ctaggart has requested the ability to serialize/deserialize F# Record types, rather than .NET Class Types.

Since there is no tutorial of using Froto :(, I try to read some examples and source codes. Unfortunately, I still cannot figure out how to dehydrating/hydrating a union type. For example, given a message:

message int_union
{
  oneof {
    int32 value_32 = 1;
    int64 value_64 = 2;
  }
}

I observe that there is some functions like dehydrateOptionalMessage and hydrateOptionalMessage, so I guess that they are used to serialize/deserialize an optional message. But here int32 and int64 are primitive types, there is no need to wrap them into a message.

Im quickly trying to find usages but can't find any references in the core code?

The Roslyn bits (and hence Exe) have a dependency on Microsoft.CodeAnalysis.CSharp.Workspaces, which in turn were built to target .NET 4.5.2. However, mono / monodevelop don't (yet) appear to support that version so the build fails, even if those projects are set to target 4.5(.0).

Tried several ways to fix this up, but hit the limits of my Linux knowledge.

According to the protobuf serialization specification, repeated fields need to be kept in-order.

As a quick fix for issue #35, the code was modified to append repeated fields to the end of the list. However, that introduced an O(n^2) performance problem in hydrateOneRepeatedInstance.

Building the list in reverse order, then doing a final List.rev, would involve a more invasive set of changes: to be spec-compliant, the code must handle repeated fields that appear loose in the serialized stream and which may have other fields interspersed between.

Two implementation choices are:

1. Add pre- and post-processing abstract functions to MessageBase, which are run at the beginning and end of a call to merge a serialized stream into the object. These would do a List.rev on every defined repeated field (but not packed-repeated, which already work properly).
2. Redefine the API contract and use ResizeArray (or similar) for all repeated fields, and just append to the end in constant time.

Frankly, option 2 makes the most sense: the intention behind MessageBase is to make generated protobuf objects friendly to other .NET languages (namely C#). So, moving to a non-F# container here is actually more consistent with that design tenet.

With the following .proto:

message a { repeated string test = 1; }

Froto.Roslyn.ProtoGen.generate returns the following:

using ProtoBuf;
using System.IO;

namespace ab
{
    [ProtoContract]
    public sealed class a
    {
        public string test { get; set; }
    }
}

namespace ab
{
    using ab;

    public static class ba
    {
        public static a Createa()
        {
            return new a();
        }

        public static void Serializea(Stream stream, a instance)
        {
            Serializer.Serialize(stream, instance);
        }

        public static a Deserializea(Stream stream)
        {
            return Serializer.Deserialize<a>(stream);
        }
    }
}

It would be expected that the type of test is that of some sort of enumerable (perhaps array).

Hi, I didn't find any sample on that one, but would be nice to have serialization of Discriminated Unions

ASP.NET Core 2.2 Roadmap
aspnet/Announcements#307
HTTP/2 in Kestrel & HttpClient
The title says it all in this case. HTTP/2 support for all. We started this work some time ago, and have had the protocol mostly implemented for a while but not compiled into the product since it wasn't ready. This version we will finish that work and implement the missing features.

My understanding is that gRPC is a pretty thin layer on top of HTTP/2. With HTTP/2 finally coming with ASP.NET Core 2.2, would it be beneficial to build a gRPC layer on top?

Fix issues with Froto.Core.IO

• Restructure to handle deserialization
• Improve performance
• Reduce (or eliminate) making multiple copies of strings and byte strings
• Report errors on invalid encoding
• Handle big-endian architectures
• Unit test everything

The current Froto.Core.IO code, which does serialization/deserialization to/from the Protobuf Wire format, requires knowing the final datatype of a field before deserializing a field.

As background, the Wire Format only supports four data types: varint, fixed32, fixed64, and length-delimited. (There is also an "group" format, but that is deprecated, not supported outside google, and not officially documented). These get mapped into more specific data types, such as int32, sint32, byte string, and text string by a message definition, based on the field number.

The problem is that the final datatype isn't known until after parsing the field number and fields can arrive out-of-order. Therefore, one approach could be to parse all fields in a message into the above four data types, then flow these into the final field representation by looking up the datatype (and field name) using the field number, then doing the appropriate type converstion. This could also be done lazy via a sequence.

Another issue with the Core.IO code is that every varint requires creating a byte array and copying the underlying bytes, reducing performance and increasing the amount of GC pressure.

Requesting simultaneous support for both proto2 and proto3 language syntax, selected via the new syntax = "[proto2|proto3]"; statement.

https://developers.google.com/protocol-buffers/docs/reference/proto3-spec

As possible incentive, the official C# generator for Proto3 does not (and will not) support generating code for the proto2 language, and thus it is not (and will-not-be) possible to write protobuf tooling and self-describing services in .NET.

Perhaps Froto can be used to fill that need.

Create an F# Type Provider which reads .proto Files and produces classes for serializing and deserializing to protobuf wireformat.

This is being iterated on in branch type-provider.

Non-goals

• Generation of service wrapper classes.
• Importing from binary (compiled .proto) FileDescriptorSet.

It looks to me that there's some inconsistency in Encode module:

1. Encode.fromFixed32 should take uint32 value as a parameter rather than int, then it would match Decode.toFixed32 which returns uint32 already
2. Encode.fromFixed64 should take uint64 value as a parameter rather than int, then it would match Decode.toFixed64 which returns uint64 already
3. There's no function Encode.toSFixed32 taking int32, but dual function Decode.toSFixed32 already exists
4. There's no function Encode.toSFixed64 taking int64, but dual function Decode.toSFixed64 already exists

I consider this page as the source of mappings from protobuf scalar types to .NET ones.

If that's indeed something that needs to be fixed, let me know and I'll have a look.

As titled. This

syntax = "proto3";

message SearchRequest {
  string _query = 1;
}

Should be valid.

The primary parser functions - which parse strings, streams, and files - are very hard to discover in the current namespaces:

Current names:

• Froto.Parser.Proto.parseFile
• Froto.Parser.Proto.parseStream
• Froto.Parser.Proto.parseString

Unfortunately, all the internal parse functions also live in Froto.Parser.Proto, making these primary functions very hard to discover.

Recommending that the primary parsing functions above get moved into a Parse module, and that the Proto parsers get moved into a Parse.Parsers module. Further, the parser parameter should be removed from those functions, so the user does not need to supply the pProto parser to these functions.

This simplifies client code down to the following:

open Froto.Parser

let statements = Parse.fromFile fileName

https://twitter.com/funcOfJoe/status/943671450677927936

Span lands in .NET! We began this journey in Midori 10 years ago, a magical elixir of perf and safety. A heartfelt congratulations go out to the team for all the hard work shipping it, many of which stuck it out all these years. A labor of love! https://msdn.microsoft.com/en-us/magazine/mt814808?f=255&MSPPError=-2147217396

https://twitter.com/khellang/status/943778553111433216

One thing I was pleasantly surprised by is the how complete the span support is across the BCL now. Almost very single API that takes a byte array now supports spans ❤

I used the dotnet-core MyGet feed. Target netcoreapp2.1 and just pick a recent preview1 version for RuntimeFrameworkVersion 😊

cc @jhugard

The parser currently correctly turns proto2 and proto3 into an AST, but does no further processing or semantic validation.

Proposed additions:

• Process import statements #109
• Normalize type names
• Flatten type names
• Validate type usage
• Resolve constants
• Capture comments

Process import statements
Parser methods need to be enhanced with, e.g., a function parameter that can satisfy includes. For Parse.fromFile, this could be an include path or paths. How to provide this for strings and streams is an open question.

Normalize Type Names
Convert various naming conventions into a single normalized form. Should support outputting in caller-selectable camelCase, PascalCase, snake_case, etc.

Flatten Type Names
Messages can contain inner message and enum definitions. These are publically visible and can be referenced from other types. It will probably be convenient to flatten these symbols and move them from an inner scope to the top level scope, while renaming the type using the fully-qualified scope name; e.g., "Outer.Inner".

In addition, Record types cannot contain inner types. Since .NET can handle "+" in a symbol name, the names could be normalized from "Outer.Inner" to "Outer+Inner" either in this step, or when doing code-gen for Records.

Validate Type Usage
Ensure every message used as a field type and enum used as a constant are defined somewhere. Must handle use before definition (so cannot be single pass). Must also handle qualification with the package name or unqualified using the current .proto file's package.

Resolve Constants
An enum can be used as a constant in an Option definition. Consider either resolving these, or exposing the symbol table used during type usage validation so that client code can do the substitution.

Capture Comments
Google's protobuf compiler can capture comments related to a symbol. This can be quite useful for generating documentation, such as in a type provider. It is also needed to create a full FileDescriptorSet (a binary protobuf representation of a set of .proto files).

• Load and parse files from import statements.
  See https://developers.google.com/protocol-buffers/docs/proto3#importing-definitions
• Verify message identifiers used as field types are actually defined
  and enum values used in option value assignment are actually
  defined. Note these must handle forward references.
• For option assignment, verify type of literal matches type of option's
  message field definition.
• Predefine standard options and enums, or load these from an
  external "descriptor.proto" file. For standard option definitions,
  see https://github.com/google/protobuf/blob/master/src/google/protobuf/descriptor.proto#L251
• Generate errors with context including file/line/column when
  any of the above validations fail (note: this might be better than
  what protoc reports).
• Record enough source information in the AST to generate SourceCodeInfo
  (see https://github.com/google/protobuf/blob/master/src/google/protobuf/descriptor.proto#L84)
• Record any other information in the AST needed to generate a complete
  FileDescriptorProto and FileDescriptorSet.

Now that we are using .NET Core project files #73, thoughts on targeting .NET Standard 2.0 instead of net45? cc @jhugard @takemyoxygen

.NET Standard Versions

• https://github.com/dotnet/standard/blob/master/docs/versions.md
• https://blogs.msdn.microsoft.com/dotnet/2017/11/15/announcing-f-support-for-net-core-and-net-standard-projects-in-visual-studio/
• https://blogs.msdn.microsoft.com/dotnet/2017/08/14/f-and-net-core-roadmap-update/
• Multi-targeting does not work in VS dotnet/fsharp#4084

Proto2 mandates that required fields always appear in the serialized output, even when they match the default value. Not doing so will cause an error when interoperating with other implementations.

None of the Froto libraries are Portable Class Libraries, so they cannot be included in other PCL projects.

It would be useful to, at minimum, produce Froto.Serialization as a PCL.

I'm trying to build Froto on OS X and Windows, but build fails on both platforms with different error messages:

OS X 10.11.5 (Mono 4.4.0):

Sometimes it's

  TestParser.Service.Parse rpc [FAIL]
      System.ExecutionEngineException : Failed to create shadow copy .ini file.
      Stack Trace:
          at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
          at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) <0x3e5bae8 + 0x00093> in <filename unknown>:0 

but sometimes unit tests target completes successfully and build fails on the next step with

Running build failed.
Error:
System.Exception: Error during NuGet package creation. /Users/takemyoxygen/dev/code/froto/packages/NuGet.Commandline/tools/nuget.exe pack -Version 0.4.0-a1606261534 -OutputDirectory "/Users/takemyoxygen/dev/code/froto/bin" "/Users/takemyoxygen/dev/code/froto/Parser/bin/Release/Froto.Parser.0.4.0-a1606261534.nuspec"    
楆敬渠瑯映畯摮›䘧潲潴倮牡敳⹲摰❢ਮ
  at Fake.NuGetHelper.NuGet (Microsoft.FSharp.Core.FSharpFunc`2 setParams, System.String nuspecOrProjectFile) <0x91f2600 + 0x00233> in <filename unknown>:0 
  at [email protected] (Microsoft.FSharp.Core.Unit _arg7) <0x91f2168 + 0x0002f> in <filename unknown>:0 
  at Fake.TargetHelper+targetFromTemplate@195[a].Invoke (Microsoft.FSharp.Core.Unit unitVar0) <0x91ea810 + 0x00020> in <filename unknown>:0 
  at Fake.TargetHelper.runSingleTarget (Fake.TargetTemplate`1 target) <0x91e45c8 + 0x000bd> in <filename unknown>:0

With the latter I guess I know what might be the reason: if I try to create NuGet package manually by running:

mono packages/NuGet.CommandLine/tools/NuGet.exe pack Parser/bin/Release/Froto.Parser.0.4.0-a1606261534.nuspec 

I get the following error:

Attempting to build package from 'Froto.Parser.0.4.0-a1606261534.nuspec'.
File not found: 'Froto.Parser.pdb'.

So, it's just because there's no .pdb files produced during build on Mono.

Windows 10

After I do

.\paket.bootstrapper.exe
.\paket restore
.\build.ps1

build fails with:

Running build failed.
Error:
System.Exception: Could not detect package version for FParsec ---> System.Exception: Package FParsec was not found.
   at [email protected](String message) in C:\code\FAKE\src\app\FakeLib\NuGet\NugetHelper.fs:line 113
   at Fake.NuGetHelper.GetPackageVersion(String deploymentsDir, String package) in C:\code\FAKE\src\app\FakeLib\NuGet\NugetHelper.fs:line 113
   --- End of inner exception stack trace ---
   at Fake.NuGetHelper.GetPackageVersion(String deploymentsDir, String package) in C:\code\FAKE\src\app\FakeLib\NuGet\NugetHelper.fs:line 127
   at [email protected](NuGetParams p) in Y:\dev\code\froto\build.fsx:line 86
   at Fake.NuGetHelper.NuGet(FSharpFunc`2 setParams, String nuspecOrProjectFile) in C:\code\FAKE\src\app\FakeLib\NuGet\NugetHelper.fs:line 387
   at [email protected](Unit _arg7) in Y:\dev\code\froto\build.fsx:line 85
   at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\FAKE\src\app\FakeLib\TargetHelper.fs:line 484
System.Exception: Could not detect package version for FParsec ---> System.Exception: Package FParsec was not found.
   at [email protected](String message) in C:\code\FAKE\src\app\FakeLib\NuGet\NugetHelper.fs:line 113
   at Fake.NuGetHelper.GetPackageVersion(String deploymentsDir, String package) in C:\code\FAKE\src\app\FakeLib\NuGet\NugetHelper.fs:line 113
   --- End of inner exception stack trace ---
   at Fake.NuGetHelper.GetPackageVersion(String deploymentsDir, String package) in C:\code\FAKE\src\app\FakeLib\NuGet\NugetHelper.fs:line 127
   at [email protected](NuGetParams p) in Y:\dev\code\froto\build.fsx:line 86
   at Fake.NuGetHelper.NuGet(FSharpFunc`2 setParams, String nuspecOrProjectFile) in C:\code\FAKE\src\app\FakeLib\NuGet\NugetHelper.fs:line 387
   at [email protected](Unit _arg7) in Y:\dev\code\froto\build.fsx:line 85
   at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\FAKE\src\app\FakeLib\TargetHelper.fs:line 484

Are these ones known issues or I'm just missing something?

By the way, is that really necessary to specify all dependencies when creating a package in build.fsx and list all files that should packaged in nuspec file?

I mentioned in #11 that I want to create an F# code generator that supports proto3. I want to make use of F# Records and Options. For basic serialization, either Google.Protbuf library can be used or I can see if Froto.Core functions I created earlier will be enough. Similar to using Roslyn to generate C# code, I want to use the F# AST to generate F# code. I've made quite good progress on FsAst, but it will probably need to be worked on to support this effort.

I'm going to remove my first attempt at building the type provider prototype. It was a wrapper around Froto.Roslyn that I called Froto.Gen. Removing it for now.