azure / bicep-types Goto Github PK
View Code? Open in Web Editor NEWShared libraries for reading and writing Bicep type definitions
License: MIT License
Shared libraries for reading and writing Bicep type definitions
License: MIT License
This is a catalog of design issues/concerns found using the types.json
format and tooling to teach Bicep about Kubernetes.
Individual items mentioned here represent design issues or challenges - basically all of the ways that the Bicep compiler e2e is coupled to ARM's types. Not all of these have to be solved, I'm erring on the side of including design issues even if there is a solution. In general the coupling between Bicep and ARM is spread across many layers currently - from the types.json
code to the compiler's semantic diagnostics and code generation.
types.json
is the FFI format for ingesting types into the Bicep compiler. It's designed as a non-human-readable/writable representation of the Bicep type system. In constrast the APIs Bicep code wants to interface with are generally described using JSON-Schema or the the Swagger/OpenAPI variant of JSON-Schema.
There's therefore an impedence mismatch - JSON-schema can express a wide variety of complex schemas that the Bicep compiler cannot understand. The process of generating a types.json
for an API means resolving this impedence mismatch and mapping the set of JSON-schema types for an API into the set of Bicep types. This process will likely be different for every API we want to map into Bicep.
The converter/generator of types.json
in this repo is built for ARM's REST API conventions, and has deep understanding of the patterns that appear in those OpenAPI documents. When I tried to use the autorest-based converter, I found that I had to re-implement almost all of it just to get to hello world. Dialects of OpenAPI used by different systems are drastically different in practice.
Such a conversion tool is responsible many tasks:
types.json
document and index structure from Bicep types (generic)In practice for a non-ARM system all of the steps named as domain specific will need to be written from scratch or highly customized. Systems extend OpenAPI in proprietary ways, or use features sets that don't overlap with ARM.
Some examples:
x-kubernetes-kind
x-kubernetes-...
annotations, scrape parameter lists to determine scopeI'd strongly suggest turning types.json
into a fully-documented and human-readable/writable format, and let the extensibility author write their own generatior/mapper. Another clickstop would be to allow Bicep to read JSON-schema with annotations that are specific to Bicep's type system. Then the scope of a conversion tool is limited to trimming down the original document into just the relevant types and adding annotations to describe the Bicep behavior.
types.json
does not fully describe the behavior of a type once it gets inside the compiler. There are important semantic details and behaviors that are hardcoded in the compiler rather than data-driven by the format.
For reference there's a set of well-known properties defined:
These property are given special status by the Bicep compiler, and compiler features rely on these properties appearing at the top level of an object for correctness. For a motivating example the name
property is required to be loop-variant, and required to be unique for a type in the same template. Many places in code will discard a resource that does not define name. However in Kubernetes the actual property with those semantics is .metadata.name
.
For each of the special behaviors that can be granted to a property it should be possible to indicate in types.json
with property flags to which properties they apply.
A short list of things that bake in ARM's conventions.
kind
in Kubernetes)To detail this further - compare the property flags supported by types.json
and the property flags supported by Bicep
types.json
. A function can only be implemented in side the C# part of a type provider.types.json
. A fallback type can only be implemented in side the C# part of a type provider.namespace
and cluster
as meaningful scopes.Kubernetes objects largely don't use discriminated unions in the same way ARM resources do - they use a different pattern (oneOf
) to express a choice. You can read more about their conventions here.
Providing a good tooling experience for other systems means supporting some consensus set of features they rely-on for validation/structure.
An example of this pattern using different sources to set environment variables. Only one of either value
or valueFrom
can be used.
apiVersion: v1
kind: Pod
metadata:
name: envar-demo
spec:
containers:
- name: envar-demo-container
image: gcr.io/google-samples/node-hello:1.0
env:
- name: DEMO_GREETING
value: "Hello from the environment" # using .value to set an environment variable
- name: DEMO_FAREWELL
valueFrom: # using .valueFrom to set an environment variable
secretKeyRef:
name: mysecret
key: goodbye
In ARM this would be expressed similar to:
resource foo '...' = {
...
properties: {
....
env: [
{
name: DEMO_GREETING
value: {
kind: 'static'
value: 'Hello from the environment'
}
}
{
name: DEMO_FAREWELL
value: {
kind: 'secret'
name: mysecret
key: goodbye
}
}
]
}
}
When referencing the library from a .NET 7 project, using the TypeSerializer
throws a System.NotSupportedException
.
The TypeSerializer
does not throw.
Calling TypeSerializer.Deserialize
throws a System.NotSupportedException
with the following message:
"Metadata for type 'Azure.Bicep.Types.Concrete.TypeBase[]' was not provided by TypeInfoResolver of type 'Azure.Bicep.Types.Serialization.TypeJsonContext'. If using source generation, ensure that all root types passed to the serializer have been indicated with 'JsonSerializableAttribute', along with any types that might be serialized polymorphically."
Create a new dotnet console application using the .NET 7 sdk.
dotnet new console
Add the Azure.Bicep.Types package.
dotnet add package Azure.Bicep.Types
Edit Program.cs to contain the following:
Azure.Bicep.Types.Serialization.TypeSerializer.Deserialize(new System.IO.MemoryStream(System.Text.Encoding.UTF8.GetBytes("[]")));
Console.WriteLine("It works!");
Run the program.
dotnet run
See the exception.
Unhandled exception. System.NotSupportedException: Metadata for type 'Azure.Bicep.Types.Concrete.TypeBase[]' was not provided by TypeInfoResolver of type 'Azure.Bicep.Types.Serialization.TypeJsonContext'. If using source generation, ensure that all root types passed to the serializer have been indicated with 'JsonSerializableAttribute', along with any types that might be serialized polymorphically.
at System.Text.Json.ThrowHelper.ThrowNotSupportedException_NoMetadataForType(Type type, IJsonTypeInfoResolver resolver)
at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(Type type, Boolean ensureConfigured, Boolean resolveIfMutable)
at System.Text.Json.JsonSerializer.GetTypeInfo(JsonSerializerOptions options, Type inputType)
at System.Text.Json.JsonSerializer.Deserialize[TValue](Stream utf8Json, JsonSerializerOptions options)
at Azure.Bicep.Types.Serialization.TypeSerializer.Deserialize(Stream contentStream)
at Program.<Main>$(String[] args) in Program.cs:line 1
Add the [JsonSerializable(typeof(TypeBase[]))]
attribute to TypeJsonContext
.
To implement Azure/bicep-types-az#437, a new ObjectTypePropertyFlags
member should be added to indicate that a resource provider considers a given property to be sensitive in nature.
Identification of which type properties are sensitive will be specific to each resource provider; for AZ types, this would be used when a property has the x-ms-secret
Swagger extension.
I am seeing issue in bicep language server startup in visual studio.
On debugging, I see below exception:
System.PlatformNotSupportedException
HResult=0x80131539
Message=Operation is not supported on this platform.
Source=System.Text.Encodings.Web
StackTrace:
at System.Text.Encodings.Web.OptimizedInboxTextEncoder.AllowedAsciiCodePoints.get_AsVector()
VS version:
17.5.0 Preview 5.0
Call stack:
System.Text.Encodings.Web.dll!System.Text.Encodings.Web.OptimizedInboxTextEncoder.AllowedAsciiCodePoints.AsVector.get() Unknown
System.Text.Encodings.Web.dll!System.Text.Encodings.Web.OptimizedInboxTextEncoder.GetIndexOfFirstByteToEncodeSsse3(byte* pData, nuint lengthInBytes) Unknown
System.Text.Encodings.Web.dll!System.Text.Encodings.Web.OptimizedInboxTextEncoder.GetIndexOfFirstByteToEncode(System.ReadOnlySpan<byte> data) Unknown
System.Text.Encodings.Web.dll!System.Text.Encodings.Web.DefaultJavaScriptEncoder.FindFirstCharacterToEncodeUtf8(System.ReadOnlySpan<byte> utf8Text) Unknown
System.Text.Json.dll!System.Text.Json.JsonWriterHelper.NeedsEscaping(System.ReadOnlySpan<byte> value, System.Text.Encodings.Web.JavaScriptEncoder encoder) Unknown
System.Text.Json.dll!System.Text.Json.JsonEncodedText.EncodeHelper(System.ReadOnlySpan<byte> utf8Value, System.Text.Encodings.Web.JavaScriptEncoder encoder) Unknown
System.Text.Json.dll!System.Text.Json.JsonEncodedText.TranscodeAndEncode(System.ReadOnlySpan<char> value, System.Text.Encodings.Web.JavaScriptEncoder encoder) Unknown
System.Text.Json.dll!System.Text.Json.JsonEncodedText.Encode(System.ReadOnlySpan<char> value, System.Text.Encodings.Web.JavaScriptEncoder encoder) Unknown
System.Text.Json.dll!System.Text.Json.JsonEncodedText.Encode(string value, System.Text.Encodings.Web.JavaScriptEncoder encoder) Unknown
> Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeJsonContext.TypeJsonContext() Line 14 C#
[Native to Managed Transition]
[Managed to Native Transition]
Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeJsonContext.s_defaultOptions.get() Line 16 C#
Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeJsonContext.TypeJsonContext() Line 35 C#
System.Private.CoreLib.dll!System.Activator.CreateInstance<Azure.Bicep.Types.Serialization.TypeJsonContext>() Line 140 C#
System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.AddContext<Azure.Bicep.Types.Serialization.TypeJsonContext>() Unknown
Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeBaseConverter.TypeBaseConverter(Azure.Bicep.Types.Concrete.TypeFactory factory) Line 18 C#
Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeJsonContext.GetSerializerOptions(Azure.Bicep.Types.Concrete.TypeFactory factory) Line 26 C#
Azure.Bicep.Types.dll!Azure.Bicep.Types.Serialization.TypeSerializer.DeserializeIndex(System.IO.Stream contentStream) Line 36 C#
Azure.Bicep.Types.dll!Azure.Bicep.Types.TypeLoader.LoadTypeIndex() Line 42 C#
Azure.Bicep.Core.dll!Bicep.Core.TypeSystem.Az.AzResourceTypeLoader.AzResourceTypeLoader() Line 25 C#
[Native to Managed Transition]
[Managed to Native Transition]
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpret(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.Expression expr, DryIoc.FastExpressionCompiler.LightExpression.IParameterProvider paramExprs, object paramValues, DryIoc.Interpreter.ParentLambdaArgs parentArgs, out object result) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpretAndUnwrapContainerException(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.Expression expr, out object result) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.ApplyReuse(DryIoc.FastExpressionCompiler.LightExpression.Expression serviceExpr, DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Rules.ConcreteTypeDynamicRegistrations.AnonymousMethod__2(DryIoc.Request req) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.GetRuleSelectedServiceFactoryOrDefault(DryIoc.Rules rules, DryIoc.ImTools.ImHashMap<System.Type, object> serviceFactories, DryIoc.Request request, DryIoc.ServiceDetails details, System.Type serviceType) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IContainer.GetServiceFactoryOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IContainer.ResolveFactory(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.ResolveAndCache(int serviceTypeHash, System.Type serviceType, DryIoc.IfUnresolved ifUnresolved) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IResolver.Resolve(System.Type serviceType, DryIoc.IfUnresolved ifUnresolved) Unknown
OmniSharp.Extensions.JsonRpc.dll!OmniSharp.Extensions.JsonRpc.JsonRpcHandlerCollectionExtensions.Populate(OmniSharp.Extensions.JsonRpc.IJsonRpcHandlerCollection collection, DryIoc.IResolverContext resolverContext, OmniSharp.Extensions.JsonRpc.IHandlersManager handlersManager) Unknown
OmniSharp.Extensions.LanguageServer.Shared.dll!OmniSharp.Extensions.LanguageServer.Shared.LanguageProtocolServiceCollectionExtensions.AddLanguageProtocolInternals.AnonymousMethod__0_2(OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection manager, DryIoc.IResolverContext context) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Registrator.Initializer<OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection, OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection>(OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection service, DryIoc.IResolverContext resolver, System.Action<OmniSharp.Extensions.LanguageServer.Shared.SharedHandlerCollection, DryIoc.IResolverContext> initialize) Unknown
[Native to Managed Transition]
[Managed to Native Transition]
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpretMethodCall(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.MethodCallExpression callExpr, DryIoc.FastExpressionCompiler.LightExpression.IParameterProvider paramExprs, object paramValues, DryIoc.Interpreter.ParentLambdaArgs parentArgs, ref object result) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpret(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.Expression expr, DryIoc.FastExpressionCompiler.LightExpression.IParameterProvider paramExprs, object paramValues, DryIoc.Interpreter.ParentLambdaArgs parentArgs, out object result) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Interpreter.TryInterpretAndUnwrapContainerException(DryIoc.IResolverContext r, DryIoc.FastExpressionCompiler.LightExpression.Expression expr, out object result) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.ApplyReuse(DryIoc.FastExpressionCompiler.LightExpression.Expression serviceExpr, DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IContainer.GetDecoratorExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.ReflectionFactory.CreateExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Factory.GetExpressionOrDefault(DryIoc.Request request) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.ResolveAndCache(int serviceTypeHash, System.Type serviceType, DryIoc.IfUnresolved ifUnresolved) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Container.DryIoc.IResolver.Resolve(System.Type serviceType, DryIoc.IfUnresolved ifUnresolved) Unknown
OmniSharp.Extensions.JsonRpc.dll!DryIoc.Resolver.Resolve<OmniSharp.Extensions.LanguageServer.Server.LanguageServer>(DryIoc.IResolver resolver, DryIoc.IfUnresolved ifUnresolved) Unknown
OmniSharp.Extensions.LanguageServer.dll!OmniSharp.Extensions.LanguageServer.Server.LanguageServer.Create(OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions options, System.IServiceProvider outerServiceProvider) Unknown
OmniSharp.Extensions.LanguageServer.dll!OmniSharp.Extensions.LanguageServer.Server.LanguageServer.Create(System.Action<OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions> optionsAction, System.IServiceProvider outerServiceProvider) Unknown
OmniSharp.Extensions.LanguageServer.dll!OmniSharp.Extensions.LanguageServer.Server.LanguageServer.Create(System.Action<OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions> optionsAction) Unknown
OmniSharp.Extensions.LanguageServer.dll!OmniSharp.Extensions.LanguageServer.Server.LanguageServer.PreInit(System.Action<OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions> optionsAction) Unknown
Bicep.LangServer.dll!Bicep.LanguageServer.Server.Server(System.Action<OmniSharp.Extensions.LanguageServer.Server.LanguageServerOptions> onOptionsFunc) Line 51 C#
Bicep.LangServer.dll!Bicep.LanguageServer.Program.RunServer(Bicep.LanguageServer.Program.CommandLineOptions options, System.Threading.CancellationToken cancellationToken) Line 99 C#
Bicep.LangServer.dll!Bicep.LanguageServer.Program.Main.AnonymousMethod__3(Bicep.LanguageServer.Program.CommandLineOptions options) Line 46 C#
CommandLine.dll!CommandLine.ParserResultExtensions.WithParsedAsync<Bicep.LanguageServer.Program.CommandLineOptions>(CommandLine.ParserResult<Bicep.LanguageServer.Program.CommandLineOptions> result, System.Func<Bicep.LanguageServer.Program.CommandLineOptions, System.Threading.Tasks.Task> action) Unknown
Bicep.LangServer.dll!Bicep.LanguageServer.Program.Main.AnonymousMethod__0(System.Threading.CancellationToken cancellationToken) Line 44 C#
Bicep.LangServer.dll!Bicep.LanguageServer.Program.RunWithCancellationAsync(System.Func<System.Threading.CancellationToken, System.Threading.Tasks.Task> runFunc) Line 127 C#
Bicep.LangServer.dll!Bicep.LanguageServer.Program.Main(string[] args) Line 34 C#
Bicep.LangServer.dll!Bicep.LanguageServer.Program.<Main>(string[] args) Unknown
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.