azure / bicep-types Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 6.0 1.75 MB

Shared libraries for reading and writing Bicep type definitions

License: MIT License

C# 50.52% JavaScript 1.28% TypeScript 48.20%

bicep-types's Introduction

bicep-types

This repo exposes a set of tools for generating Bicep types and type indices, for consumption by the Bicep compiler.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

bicep-types's People

Contributors

Stargazers

Watchers

Forkers

johnthcall asilverman maartenvansambeek azurementor ms-henglu sourcecodecheck

bicep-types's Issues

Add a type property flag to mark sensitive data

To implement Azure/bicep-types-az#437, a new ObjectTypePropertyFlags member should be added to indicate that a resource provider considers a given property to be sensitive in nature.

Identification of which type properties are sensitive will be specific to each resource provider; for AZ types, this would be used when a property has the x-ms-secret Swagger extension.

Bicep language server startup issue in visual studio

I am seeing issue in bicep language server startup in visual studio.

On debugging, I see below exception:
System.PlatformNotSupportedException
HResult=0x80131539
Message=Operation is not supported on this platform.
Source=System.Text.Encodings.Web
StackTrace:
at System.Text.Encodings.Web.OptimizedInboxTextEncoder.AllowedAsciiCodePoints.get_AsVector()

VS version:
17.5.0 Preview 5.0

Call stack:

	System.Text.Encodings.Web.dll!System.Text.Encodings.Web.OptimizedInboxTextEncoder.AllowedAsciiCodePoints.AsVector.get()	Unknown
 	System.Text.Encodings.Web.dll!System.Text.Encodings.Web.OptimizedInboxTextEncoder.GetIndexOfFirstByteToEncodeSsse3(byte* pData, nuint lengthInBytes)	Unknown
 	System.Text.Encodings.Web.dll!System.Text.Encodings.Web.OptimizedInboxTextEncoder.GetIndexOfFirstByteToEncode(System.ReadOnlySpan<byte> data)	Unknown
 	System.Text.Encodings.Web.dll!System.Text.Encodings.Web.DefaultJavaScriptEncoder.FindFirstCharacterToEncodeUtf8(System.ReadOnlySpan<byte> utf8Text)	Unknown
 	System.Text.Json.dll!System.Text.Json.JsonWriterHelper.NeedsEscaping(System.ReadOnlySpan<byte> value, System.Text.Encodings.Web.JavaScriptEncoder encoder)	Unknown
 	System.Text.Json.dll!System.Text.Json.JsonEncodedText.EncodeHelper(System.ReadOnlySpan<byte> utf8Value, System.Text.Encodings.Web.JavaScriptEncoder encoder)	Unknown
 	System.Text.Json.dll!System.Text.Json.JsonEncodedText.TranscodeAndEncode(System.ReadOnlySpan<char> value, System.Text.Encodings.Web.JavaScriptEncoder encoder)	Unknown
 	System.Text.Json.dll!System.Text.Json.JsonEncodedText.Encode(System.ReadOnlySpan<char> value, System.Text.Encodings.Web.JavaScriptEncoder encoder)	Unknown
 	System.Text.Json.dll!System.Text.Json.JsonEncodedText.Encode(string value, System.Text.Encodings.Web.JavaScriptEncoder encoder)	Unknown
>	Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeJsonContext.TypeJsonContext() Line 14	C#
 	[Native to Managed Transition]	
 	[Managed to Native Transition]	
 	Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeJsonContext.s_defaultOptions.get() Line 16	C#
 	Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeJsonContext.TypeJsonContext() Line 35	C#
 	System.Private.CoreLib.dll!System.Activator.CreateInstance<Azure.Bicep.Types.Serialization.TypeJsonContext>() Line 140	C#
 	System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.AddContext<Azure.Bicep.Types.Serialization.TypeJsonContext>()	Unknown
 	Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeBaseConverter.TypeBaseConverter(Azure.Bicep.Types.Concrete.TypeFactory factory) Line 18	C#
 	Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeJsonContext.GetSerializerOptions(Azure.Bicep.Types.Concrete.TypeFactory factory) Line 26	C#
 	Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeSerializer.DeserializeIndex(System.IO.Stream contentStream) Line 36	C#
 	Azure.Bicep.Types.dll!Azure.Bicep.Types.TypeLoader.LoadTypeIndex() Line 42	C#
 	Azure.Bicep.Core.dll!Bicep.Core.TypeSystem.Az.AzResourceTypeLoader.AzResourceTypeLoader() Line 25	C#
 	[Native to Managed Transition]	
 	[Managed to Native Transition]	
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpret(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.Expression expr, DryIoc.FastExpressionCompiler.LightExpression.IParameterProvider paramExprs, object paramValues, DryIoc.Interpreter.ParentLambdaArgs parentArgs, out object result)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpretAndUnwrapContainerException(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.Expression expr, out object result)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.ApplyReuse(DryIoc.FastExpressionCompiler.LightExpression.Expression serviceExpr, DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Rules.ConcreteTypeDynamicRegistrations.AnonymousMethod__2(DryIoc.Request req)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.GetRuleSelectedServiceFactoryOrDefault(DryIoc.Rules rules, DryIoc.ImTools.ImHashMap<System.Type, object> serviceFactories, DryIoc.Request request, DryIoc.ServiceDetails details, System.Type serviceType)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IContainer.GetServiceFactoryOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IContainer.ResolveFactory(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.ResolveAndCache(int serviceTypeHash, System.Type serviceType, DryIoc.IfUnresolved ifUnresolved)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IResolver.Resolve(System.Type serviceType, DryIoc.IfUnresolved ifUnresolved)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!OmniSharp.Extensions.JsonRpc.JsonRpcHandlerCollectionExtensions.Populate(OmniSharp.Extensions.JsonRpc.IJsonRpcHandlerCollection collection, DryIoc.IResolverContext resolverContext, OmniSharp.Extensions.JsonRpc.IHandlersManager handlersManager)	Unknown
 	OmniSharp.Extensions.LanguageServer.Shared.dll!OmniSharp.Extensions.LanguageServer.Shared.LanguageProtocolServiceCollectionExtensions.AddLanguageProtocolInternals.AnonymousMethod__0_2(OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection manager, DryIoc.IResolverContext context)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Registrator.Initializer<OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection, OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection>(OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection service, DryIoc.IResolverContext resolver, System.Action<OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection, DryIoc.IResolverContext> initialize)	Unknown
 	[Native to Managed Transition]	
 	[Managed to Native Transition]	
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpretMethodCall(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.MethodCallExpression callExpr, DryIoc.FastExpressionCompiler.LightExpression.IParameterProvider paramExprs, object paramValues, DryIoc.Interpreter.ParentLambdaArgs parentArgs, ref object result)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpret(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.Expression expr, DryIoc.FastExpressionCompiler.LightExpression.IParameterProvider paramExprs, object paramValues, DryIoc.Interpreter.ParentLambdaArgs parentArgs, out object result)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpretAndUnwrapContainerException(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.Expression expr, out object result)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.ApplyReuse(DryIoc.FastExpressionCompiler.LightExpression.Expression serviceExpr, DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IContainer.GetDecoratorExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.ResolveAndCache(int serviceTypeHash, System.Type serviceType, DryIoc.IfUnresolved ifUnresolved)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IResolver.Resolve(System.Type serviceType, DryIoc.IfUnresolved ifUnresolved)	Unknown
 	OmniSharp.Extensions.JsonRpc.dll!DryIoc.Resolver.Resolve<OmniSharp.Extensions.LanguageServer.Server.LanguageServer>(DryIoc.IResolver resolver, DryIoc.IfUnresolved ifUnresolved)	Unknown
 	OmniSharp.Extensions.LanguageServer.dll!OmniSharp.Extensions.LanguageServer.Server.LanguageServer.Create(OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions options, System.IServiceProvider outerServiceProvider)	Unknown
 	OmniSharp.Extensions.LanguageServer.dll!OmniSharp.Extensions.LanguageServer.Server.LanguageServer.Create(System.Action<OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions> optionsAction, System.IServiceProvider outerServiceProvider)	Unknown
 	OmniSharp.Extensions.LanguageServer.dll!OmniSharp.Extensions.LanguageServer.Server.LanguageServer.Create(System.Action<OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions> optionsAction)	Unknown
 	OmniSharp.Extensions.LanguageServer.dll!OmniSharp.Extensions.LanguageServer.Server.LanguageServer.PreInit(System.Action<OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions> optionsAction)	Unknown
 	Bicep.LangServer.dll!Bicep.LanguageServer.Server.Server(System.Action<OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions> onOptionsFunc) Line 51	C#
 	Bicep.LangServer.dll!Bicep.LanguageServer.Program.RunServer(Bicep.LanguageServer.Program.CommandLineOptions options, System.Threading.CancellationToken cancellationToken) Line 99	C#
 	Bicep.LangServer.dll!Bicep.LanguageServer.Program.Main.AnonymousMethod__3(Bicep.LanguageServer.Program.CommandLineOptions options) Line 46	C#
 	CommandLine.dll!CommandLine.ParserResultExtensions.WithParsedAsync<Bicep.LanguageServer.Program.CommandLineOptions>(CommandLine.ParserResult<Bicep.LanguageServer.Program.CommandLineOptions> result, System.Func<Bicep.LanguageServer.Program.CommandLineOptions, System.Threading.Tasks.Task> action)	Unknown
 	Bicep.LangServer.dll!Bicep.LanguageServer.Program.Main.AnonymousMethod__0(System.Threading.CancellationToken cancellationToken) Line 44	C#
 	Bicep.LangServer.dll!Bicep.LanguageServer.Program.RunWithCancellationAsync(System.Func<System.Threading.CancellationToken, System.Threading.Tasks.Task> runFunc) Line 127	C#
 	Bicep.LangServer.dll!Bicep.LanguageServer.Program.Main(string[] args) Line 34	C#
 	Bicep.LangServer.dll!Bicep.LanguageServer.Program.<Main>(string[] args)	Unknown

Add `FunctionType` to markdown writer

FunctionType was added as a new type but it is not included in the markdown writer. Support for FunctionType should be added so it gets generated with the markdown file.

Issues with types.json and Bicep extensibility

This is a catalog of design issues/concerns found using the types.json format and tooling to teach Bicep about Kubernetes.

Individual items mentioned here represent design issues or challenges - basically all of the ways that the Bicep compiler e2e is coupled to ARM's types. Not all of these have to be solved, I'm erring on the side of including design issues even if there is a solution. In general the coupling between Bicep and ARM is spread across many layers currently - from the types.json code to the compiler's semantic diagnostics and code generation.

Structure and generation of types.json

types.json is the FFI format for ingesting types into the Bicep compiler. It's designed as a non-human-readable/writable representation of the Bicep type system. In constrast the APIs Bicep code wants to interface with are generally described using JSON-Schema or the the Swagger/OpenAPI variant of JSON-Schema.

There's therefore an impedence mismatch - JSON-schema can express a wide variety of complex schemas that the Bicep compiler cannot understand. The process of generating a types.json for an API means resolving this impedence mismatch and mapping the set of JSON-schema types for an API into the set of Bicep types. This process will likely be different for every API we want to map into Bicep.

The converter/generator of types.json in this repo is built for ARM's REST API conventions, and has deep understanding of the patterns that appear in those OpenAPI documents. When I tried to use the autorest-based converter, I found that I had to re-implement almost all of it just to get to hello world. Dialects of OpenAPI used by different systems are drastically different in practice.

Such a conversion tool is responsible many tasks:

Parsing OpenAPI/Swagger into an object model (generic)
Determining which types should be exposed as resources (domain-specific)
Mapping endpoints and types to Bicep resource type semantics: resource type name, api-version, resource scope, parent types (domain-specific)
Structural conversion from JSON-Schema to Bicep types (domain specific)
Generation of the types.json document and index structure from Bicep types (generic)

In practice for a non-ARM system all of the steps named as domain specific will need to be written from scratch or highly customized. Systems extend OpenAPI in proprietary ways, or use features sets that don't overlap with ARM.

Some examples:

Determining which types should be exposed as resources: look for x-kubernetes-kind
Mapping endpoints and types to Bicep resource type semantics: look for other x-kubernetes-... annotations, scrape parameter lists to determine scope
Structural conversion from JSON-Schema to Bicep types: Kubernetes objects tend to follow different conventions from ARM, will be described in detail later

I'd strongly suggest turning types.json into a fully-documented and human-readable/writable format, and let the extensibility author write their own generatior/mapper. Another clickstop would be to allow Bicep to read JSON-schema with annotations that are specific to Bicep's type system. Then the scope of a conversion tool is limited to trimming down the original document into just the relevant types and adding annotations to describe the Bicep behavior.

Coupling to ARM's conventions

types.json does not fully describe the behavior of a type once it gets inside the compiler. There are important semantic details and behaviors that are hardcoded in the compiler rather than data-driven by the format.

For reference there's a set of well-known properties defined:

These property are given special status by the Bicep compiler, and compiler features rely on these properties appearing at the top level of an object for correctness. For a motivating example the name property is required to be loop-variant, and required to be unique for a type in the same template. Many places in code will discard a resource that does not define name. However in Kubernetes the actual property with those semantics is .metadata.name.

For each of the special behaviors that can be granted to a property it should be possible to indicate in types.json with property flags to which properties they apply.

A short list of things that bake in ARM's conventions.

The name equivalent.
The scope equivalent.
The parent equivalent.
The equivalent of type and apiVersion. (kind in Kubernetes)
Parent/Child relationships are based on the format of an ARM fully-qualified type rather than metadata.
Ability to define a deploy-time-constant property

To detail this further - compare the property flags supported by types.json and the property flags supported by Bicep

Features that are not extensible

Functions & Decorators have no mechanism for extensibility via types.json. A function can only be implemented in side the C# part of a type provider.
Fallback type (the type that is used when the type cannot be resolved) is not extensible via types.json. A fallback type can only be implemented in side the C# part of a type provider.
Scopes are not extensible by any mechanism, and the compiler has deep coupling to ARM's scope concepts. Kubernetes has both namespace and cluster as meaningful scopes.

Missing features in object structure

Kubernetes objects largely don't use discriminated unions in the same way ARM resources do - they use a different pattern (oneOf) to express a choice. You can read more about their conventions here.

Providing a good tooling experience for other systems means supporting some consensus set of features they rely-on for validation/structure.

An example of this pattern using different sources to set environment variables. Only one of either value or valueFrom can be used.

apiVersion: v1
kind: Pod
metadata:
  name: envar-demo
spec:
  containers:
  - name: envar-demo-container
    image: gcr.io/google-samples/node-hello:1.0
    env:
    - name: DEMO_GREETING
      value: "Hello from the environment" # using .value to set an environment variable
    - name: DEMO_FAREWELL
      valueFrom:                          # using .valueFrom to set an environment variable
        secretKeyRef:
          name: mysecret
          key: goodbye

In ARM this would be expressed similar to:

resource foo '...' = {
    ...
    properties: {
        ....
        env: [
            {
                name: DEMO_GREETING
                value: {
                    kind: 'static'
                    value: 'Hello from the environment'
                }
            }
            {
                name: DEMO_FAREWELL
                value: {
                    kind: 'secret'
                    name: mysecret
                    key: goodbye
                }
            }
        ]
    }
}

TypeSerializer throws in .NET 7.

Description

When referencing the library from a .NET 7 project, using the TypeSerializer throws a System.NotSupportedException.

Expected behavior

The TypeSerializer does not throw.

Actual behavior

Calling TypeSerializer.Deserialize throws a System.NotSupportedException with the following message:

"Metadata for type 'Azure.Bicep.Types.Concrete.TypeBase[]' was not provided by TypeInfoResolver of type 'Azure.Bicep.Types.Serialization.TypeJsonContext'. If using source generation, ensure that all root types passed to the serializer have been indicated with 'JsonSerializableAttribute', along with any types that might be serialized polymorphically."

Steps to reproduce

Create a new dotnet console application using the .NET 7 sdk.
```
dotnet new console
```
Add the Azure.Bicep.Types package.
```
dotnet add package Azure.Bicep.Types
```

Edit Program.cs to contain the following:

Azure.Bicep.Types.Serialization.TypeSerializer.Deserialize(new System.IO.MemoryStream(System.Text.Encoding.UTF8.GetBytes("[]")));

Console.WriteLine("It works!");

Run the program.
```
dotnet run
```

See the exception.

Unhandled exception. System.NotSupportedException: Metadata for type 'Azure.Bicep.Types.Concrete.TypeBase[]' was not provided by TypeInfoResolver of type 'Azure.Bicep.Types.Serialization.TypeJsonContext'. If using source generation, ensure that all root types passed to the serializer have been indicated with 'JsonSerializableAttribute', along with any types that might be serialized polymorphically.
at System.Text.Json.ThrowHelper.ThrowNotSupportedException_NoMetadataForType(Type type, IJsonTypeInfoResolver resolver)
at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(Type type, Boolean ensureConfigured, Boolean resolveIfMutable)
at System.Text.Json.JsonSerializer.GetTypeInfo(JsonSerializerOptions options, Type inputType)
at System.Text.Json.JsonSerializer.Deserialize[TValue](Stream utf8Json, JsonSerializerOptions options)
at Azure.Bicep.Types.Serialization.TypeSerializer.Deserialize(Stream contentStream)
at Program.<Main>$(String[] args) in Program.cs:line 1

Possible solution

Add the [JsonSerializable(typeof(TypeBase[]))] attribute to TypeJsonContext.

Specification

Version: dotnet SDK 7.0.100
Platform: any (tested on windows and wsl)