bitwalker / conform Goto Github PK
View Code? Open in Web Editor NEWEasy, powerful, and extendable configuration tooling for releases.
License: MIT License
Easy, powerful, and extendable configuration tooling for releases.
License: MIT License
It would be very nice to have validation types.
Example:
datatype: Module
Where Module
implements behaviour, implements function validate
, which returns {:ok, value} | {:error, error}
It will allow to build own validation libraries on top of conform.
.
desired sys.config
[
{my_app, [{'Elixir.Bar', value}]}
]
Conform splits the to
keypath on .
which makes the above sys.config impossible.
atom
sdesired sys.config
[{other_app, [{<<"some.string">>, value}]}]
Conform expects each key component to be converted to an atom
For some configuration, the current state of conform doesn't offer enough flexibility. If someone wants a list or map of values, for example:
[[name: "buzz", type: :person, age: 25],
[name: "fido", type: :dog, age: 5]]
Currently there is no way to define a mapping from a value in the .conf to that format. The proposal for supporting this scenario looks something like the following:
# my_app.conf
my_app.complex_list.buzz.type = person
my_app.complex_list.buzz.age = 25
my_app.complex_list.fido.type = dog
my_app.complex_list.fido.age = 5
# my_app.schema.exs
[mappings: [
"my_app.complex_list.*": [
to: "my_app.complex_list",
datatype: [complex: {"my_app.complex_list.*.type", "my_app.complex_list.*.age"}],
default: []
]
"my_app.complex_list.*.type" [
to: "my_app.complex_list",
datatype: :atom,
default: :undefined
],
"my_app.complex_list.*.age" [
to: "my_app.complex_list",
datatype: :integer,
default: 0
]]
translations: [
"my_app.complex_list.*": fn
_, {name, {type, age}}, nil -> [[name: name, type: type, age: age]]
_, {name, {type, age}}, acc -> [[name: name, type: type, age: age] | acc]
]]
Stated more plainly, the tasks to implement this are as follows:
[]
complex
data type, which takes a tuple of keys to mappings which will populate that position in the tuple passed to the translation (or simply returned if no translation is provided)I'm open to suggestions, improvements, etc. Please provide examples of all three files (.conf, .schema.exs, and sys.config) so that it's clear what the inputs and outputs are.
As indicated in the pull request, failures occur when there is only a single sublist defined for a nested list config.
The API has changed for transforms a while ago, this is rather confusing.
The source code has embedded code samples which are closer to reality. The new API format is rather impractical, here is a code snippet for a transform that takes a value of "format abcde"
and turns it into {:format, "abcde"}
, useful for myapp.some_crypto_key = base64 YTM0NZomIzI2OTsmIzM0NTueYQ==
transforms: [
"myapp.some_crypto_key": fn confpid ->
# discard the wrappers, get at the value
[{_,val}] = Conform.Conf.get(confpid, "myapp.some_crypto_key")
#IO.puts "encry #{inspect val}"
[k, v] = String.split(to_string(val))
k = String.to_atom(k)
{k, v}
end
],
It would be nice to be able to define a module of functions that can be reused for translations. Either as an importable script, or defined inline. It seems conform.effective handles this already, so potentially the only thing that needs to happen is to fix exrm's conform plugin, and work on some kind of support for importing a script containing the functions to use.
In my config.exs
, have this config:
use Mix.Config
config :conform_test, test: [
{:hd_prime,<<
0xC7,0x1C,0xAE,0xB9,0xC6,0xB1,0xC9,0x04,0x8E,0x6C,0x52,0x2F,0x70,0xF1,0x3F,0x73,
0x98,0x0D,0x40,0x23,0x8E,0x3E,0x21,0xC1,0x49,0x34,0xD0,0x37,0x56,0x3D,0x93,0x0F,
0xFA,0x33,0x6F,0x6E,0x0A,0xC9,0x25,0x13,0x95,0x43,0xAE,0xD4,0x4C,0xCE,0x7C,0x37,
0xDC,0x97,0x46,0x51,0x29,0x69,0x32,0x84,0x54,0xF1,0x8F,0xAF,0x8C,0x59,0x5F,0x64,
0x24,0x77,0xFE,0x96,0xBB,0x2A,0x94,0x1D,0x5B,0xCD,0x1D,0x4A,0xC8,0xCC,0x49,0x88,
0x0D,0x81,0x15,0xF6,0x35,0xB1,0x05,0xEE,0x2E,0x4E,0x15,0xD0,0x4B,0x24,0x54,0xBF,
0x48,0x19,0x8A,0x0A,0xA7,0xC1,0x40,0x58,0x22,0x94,0x93,0xD2,0x25,0x30,0xF4,0xDB,
0x07,0x08,0xFA,0x9B,0x37,0x8E,0x3C,0x4F,0x3A,0x90,0x60,0xBE,0xE6,0x7C,0xF9,0xA4,
0x20,0xFD,0x51,0xF6,0x94,0x58,0x70,0x5A,0xC6,0x8C,0xD4,0xFE,0x6B,0x6B,0x13,0xAB,
0x0D,0xBA,0x74,0xD8,0xA8,0x4B,0x2A,0x14,0xB3,0x14,0x4E,0x0E,0xF1,0x28,0x47,0x54
>>}
]
When i execute the mix conform.new
the following errors raised:
➜ conform_test mix conform.new
You already have a schema at config/conform_test.schema.exs.
Do you want to overwrite this schema with a new one?
[Yn]
** (FunctionClauseError) no function clause matching in Conform.Utils.Code.to_heredoc/3
lib/conform/utils/code.ex:213: Conform.Utils.Code.to_heredoc(<<199, 28, 174, 185, 198, 177, 201, 4, 142, 108, 82, 47, 112, 241, 63, 115, 152, 13, 64, 35, 142, 62, 33, 193, 73, 52, 208, 55, 86, 61, 147, 15, 250, 51, 111, 110, 10, 201, 37, 19, 149, 67, 174, 212, 76, 206, 124, 55, 220, 151, ...>>, :open, "\"\"\"\n ")
lib/conform/utils/code.ex:84: Conform.Utils.Code.format_list_item/2
lib/conform/utils/code.ex:63: Conform.Utils.Code.format_list/2
lib/conform/utils/code.ex:84: Conform.Utils.Code.format_list_item/2
lib/conform/utils/code.ex:62: Conform.Utils.Code.format_list/2
lib/conform/utils/code.ex:84: Conform.Utils.Code.format_list_item/2
lib/conform/utils/code.ex:63: Conform.Utils.Code.format_list/2
lib/conform/schema.ex:289: Conform.Schema.stringify/2
You can try this test project at https://github.com/developerworks/conform_test
Hi all (I hope people are listening!),
I've been working on some significant improvements to how conform works today. One of the biggest problems I've been working on is a more intuitive way to do mappings/translations for complex types. For instance, let's say you want to get the following config output:
[ lager: [ handlers: [
lager_console_backend: [ level: :info ],
lager_file_backend: [ level: :error, path: "/var/log/error.log" ],
lager_file_backend: [ level: :info, path: "/var/log/info.log" ] ]
So you write a schema that looks like this:
[ mappings: [
"lager.handlers.*": [
to: "lager.handlers",
datatype: [list: :complex],
default: [],
... ],
"lager.handlers.console.level": [
to: "lager.handlers.lager_console_backend.level",
datatype: [enum: [:info, :error]],
default: :info,
...],
"lager.handlers.file.*": [
to: "lager.handlers.lager_file_backend",
datatype: [list: :complex],
default: [],
...],
"lager.handlers.file.error": [
to: "lager.handlers.lager_file_backend.error",
datatype: :binary,
default: "/var/log/error.log",
...],
"lager.handlers.file.info": [
to: "lager.handlers.lager_file_backend.info",
datatype: :binary,
default: "/var/log/info.log",
...] ],
translations: [
"lager.handlers.lager_file_backend.error": fn _mapping, {_, path} ->
[level: :error, path: path]
end,
"lager.handlers.lager_file_backend.info": fn _mapping, {_, path} ->
[level: :info, path: path]
end
] ]
For one, the above does not work. In fact I've found that there isn't actually a way to do this properly that works to cover a variety of edge cases. It is also unintuitive, contains a lot of what is effectively boilerplate/garbage mappings, and already contains bits which are difficult to understand in context (for instance, what args are passed to the translations, what are their values?).
I'm working on, and would like feedback on my solution to these problems. I'm planning on conforming more closely to the way cuttlefish handles this, so our schema would look more like the following:
[ mappings: [
"lager.handlers.$backend.level": [
to: "lager.handlers.lager_$backend_backend.level",
datatype: [enum: [:info, :error]],
default: :info, ... ],
"lager.handlers.file.error": [
to: "lager.handlers.lager_file_backend.error",
datatype: :binary,
default: "/var/log/error.log",
...],
"lager.handlers.file.info": [
to: "lager.handlers.lager_file_backend.info",
datatype: :binary,
default: "/var/log/info.log",
...] ],
translations: [
"lager.handlers": fn conf ->
file_handlers = Conform.Conf.get(conf, "lager.handlers.lager_file_backend.$level")
|> Enum.map(fn {[_, _, backend, level], path} -> {backend, [level: level, path: path]} end)
console_handlers = Conform.Conf.get(conf, "lager.handlers.lager_console_backend")
|> Enum.map(fn {[_, _, backend], conf} -> {backend, conf}
console_handlers ++ file_handlers
end
] ]
In a nutshell:
to
key path.Conform.Conf
will be exposed to schemas, which provides functions for querying the configuration state for data. In the above example, get
is a function which allows you to fetch values based on a key path, which can contain variables. The result of get
will always be a keyword list, where the key is the tokenized path (as atoms), and the value is the configuration at that path.conf
argument will also contain configuration read from config.exs
that isn't exposed via the .conf.I think the above significantly reduces the complexity of conform's implementation, while also providing more powerful tools for manipulating the configuration which is output.
As part of this sweep of changes, I'm also planning on introducing validators, which can be applied to mappings (prior to translations). And the following new mapping settings:
commented
- Boolean, if true will write the default value to the conf, but comment it out.required
- Boolean, if true will throw an error if the value is not present in the confhidden
- Boolean, will not write the setting to the conf when generated, in this way you can define advanced settings which can be configured, but which won't be shown to users of the .conf by defaultI'd like to get feedback on all of this, as they are significantly large breaking changes. I wish we could avoid such a large set of changes, but after a lot of work in private, I'm beginning to see how the current architecture is not very flexible moving forward, and is currently error prone for some cases. I really think this is worth the pain, but I want to make sure others feel that way as well.
For many of you, these changes may not actually affect you, but for those using translations, or wildcard mappings, it will.
When using an umbrella application, the config files for any of the apps (but especially the primary application) do not get packaged as part of the exrm release.
I was able to replicate this with https://github.com/bitwalker/exrm-umbrella-test.
$ MIX_ENV=prod mix conform.new
$ MIX_ENV=prod mix conform.configure
$ cd apps/master_app
$ MIX_ENV=prod mix release
$ find . -type f -name "*.conf"
I expected there to be a master_app.conf
in rel/master_app/releases/0.0.1/
There is, as far as I can tell, no way to encode the following config option:
config :my_app, prop_with_quotes: ~s(foo "bar" baz)
I'm trying to run the test suite, from master, and I get this as a failing test:
mix test
...............
1) test can generate config as Elixir terms from .conf and schema with imports (ConfTranslateTest)
test/conf_translate_test.exs:57
** (MatchError) no match of right hand side value: {:ok, []}
stacktrace:
test/conf_translate_test.exs:83: anonymous fn/5 in ConfTranslateTest.test can generate config as Elixir terms from .conf and schema with imports/1
(ex_unit) lib/ex_unit/capture_io.ex:146: ExUnit.CaptureIO.do_capture_io/2
(ex_unit) lib/ex_unit/capture_io.ex:119: ExUnit.CaptureIO.do_capture_io/3
test/conf_translate_test.exs:66
............
Finished in 8.3 seconds (0.1s on load, 8.2s on tests)
28 tests, 1 failure
Randomized with seed 187604
I've looked at the code for that test and the sys.config
file remains empty-- not sure what's exactly going on...
When running mix conform.effective
you have to have a myapp.conf
file already built. This wasn't immediately obvious to me, so I ended up running it expecting to see some output generated by my schema. This resulted in an undefined function error.
I think it would be pretty easy and helpful to add a check for whether or not a .conf
file exists before executing conform.effective
, and print a warning instead of users seeing a stacktrace.
Right now if you have app A which depends on app B, and both have conform schemas, you have to duplicate B's schema info in A if you want to use the .conf to configure both A and B. We can improve the usability and maintainability of configs by merging schemas from dependencies into the schema of the top-level app when generating it.
Potential issues:
I'm hesitant to consider this a bug outright, but I'll describe an example scenario.
If I create a release with 5 sublists defined for a particular key in prod.exs and need to deploy that release to a different environment where only 3 sublists are necessary (via a different .conf file), the merging process gives me the 3 sublists that I want, but also the last 2 from the initial prod.exs. I can see how a strict merge and a replacement of the list both make sense in various scenarios.
I seem to be fighting an uphill battle with sublists, so maybe I'm approaching the problem incorrectly, but figured I'd open it to discussion.
Currently exrm leverages conform to merge all dependency schemas and generate a single schema
# Get top-level schema...
schema = app |> String.to_atom |> Conform.Schema.read
# Get schemas from all dependencies
dep_schemas = Conform.Schema.coalesce
# Merge together
merged = Conform.Schema.merge(dep_schemas, schema)
this would be useful as a conform mix task which one could leverage to output the schema to a file without needing to build a release.
something like mix conform.schema [outfilepath]
I'm just going to say ahead of time this is my first Elixir/Erlang deploy (with or without exrm/conform) so be kind. I'm also assuming this is my own error and not a bug. I'm just not sure of a better way to get help/feedback. I appreciate your time.
I am using exrm 1.0.0-rc7
and conform 1.0.0-rc8
.
When I build on the same machine I want to deploy on, the conf file that has the correct values for our production env are not being applied correctly. Specifically, I have a list of kafka brokers defined as a [list: :ip]
data type in the schema. The conf file has this list of broker ips defined, but app start fails.
From the crash dump:
$ head erl_crash.dump
=erl_crash_dump:0.3
Wed Jan 13 22:41:57 2016
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,kafka_ex,{{function_clause,[{prim_inet,setopts,[nil,[binary,{packet,4},{active,false}]],[]},{'Elixir.KafkaEx.NetworkClient'
System version: Erlang/OTP 18 [erts-7.2] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false]
When I enter the console I get a KafkaNetwork exception as it's trying to connect to a local Kafka instance (the default value in the schema).
$ gretel console
Using /app/gretel/current/releases/0.0.1/gretel.sh
Exec: /app/gretel/current/erts-7.2/bin/erlexec -boot /app/gretel/current/releases/0.0.1/gretel -boot_var ERTS_LIB_DIR /app/gretel/current/erts-7.2/../lib -env ERL_LIBS /app/gretel/current/lib -config /app/gretel/current/running-config/sys.config -args_file /app/gretel/current/running-config/vm.args -user Elixir.IEx.CLI -extra --no-halt +iex -- console
Root: /app/gretel/current
/app/gretel/current
Erlang/OTP 18 [erts-7.2] [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false]
22:50:26.996 [info] Child Logger.ErrorHandler of Supervisor Logger.Supervisor started
Pid: #PID<0.68.0>
Start Call: Logger.Watcher.watcher(:error_logger, Logger.ErrorHandler, {true, true, 500}, :link)
Restart: :permanent
Shutdown: 5000
Type: :worker
22:50:27.012 [info] Application logger started at :"[email protected]"
22:50:27.014 [info] Application poison started at :"[email protected]"
22:50:27.014 [error] Could not connect to broker "127.0.0.1" on port 9092
{"Kernel pid terminated",application_controller,"{application_start_failure,kafka_ex,{{function_clause,[{prim_inet,setopts,[nil,[binary,{packet,4},{active,false}]],[]},{'Elixir.KafkaEx.NetworkClient',send_sync_request,3,[{file,\"lib/kafka_ex/network_client.ex\"},{line,30}]},{'Elixir.Enum',do_find_value,3,[{file,\"lib/enum.ex\"},{line,2490}]},{'Elixir.KafkaEx.Server',metadata,5,[{file,\"lib/kafka_ex/server.ex\"},{line,249}]},{'Elixir.KafkaEx.Server',init,1,[{file,\"lib/kafka_ex/server.ex\"},{line,26}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]},{'Elixir.KafkaEx',start,[normal,[]]}}}"}
Crash dump is being written to: erl_crash.dump...done
Kernel pid terminated (application_controller) ({application_start_failure,kafka_ex,{{function_clause,[{prim_inet,setopts,[nil,[binary,{packet,4},{active,false}]],[]},{'Elixir.KafkaEx.NetworkClient'
# schema file
# ...
"kafka_ex.brokers": [
doc: "Provide documentation for kafka_ex.brokers here.",
to: "kafka_ex.brokers",
datatype: [ list: :ip ],
default: [{"127.0.0.1", 9092}]
],
# ...
# gretel.conf
kafka_ex.brokers = some.ip.range:9092, some.ip.range:9092, some.ip.range:9092
I have placed my gretel.conf
in the release version directory /app/gretel/current/releases/0.0.1/gretel.conf
. With previous versions of the hex package, the gretel.conf was copied to /app/gretel/current/running-config/gretel.conf
, but this current version of conform doesn't do that.
At this point I'm unsure where to go. As indicated I've tried rolling back to the last stable releases of both exrm and conform, but that ran me into a separate set of issues. Specifically, running escript with the built conform file was failing with a message saying that conform_escript
function was undefined or something.
I'm sorry if that's too much info, but trying to do my homework here and not submit a bogus issue. It appears that I'm missing a simple step, but I'll be damned if I can find it. Any help would be greatly appreciated. Thanks!
The Readme says
I currently am planning on supporting user-defined types, but that work has not yet been completed.
but then it also shows an example of creating a custom type.
I tried using one, but it doesn't seem to be invoked at all, parsing the corresponding value as a binary instead.
I have updated my test project to also use the latest exrm, https://github.com/alco/conformist/commit/1a7ee58744b520a23b7defde1c5d2d3993c158fa.
When I run mix release
for the first time (or after removing the _build
directory), it builds the release. When I try to run it for the second time, I get the following error:
$ mix release
Building release with MIX_ENV=prod.
==> conform
Unchecked dependencies for environment prod:
* exrm (Hex package)
the dependency is not locked (run "mix deps.get" to generate "mix.lock" file)
==> Failed to execute before_release hook for Elixir.ReleaseManager.Plugin.Conform!
==> conformist
** (Mix) Can't continue due to errors on dependencies
jfyi, this is part of Phoenix app default config:
config :myapp, Myapp.Endpoint,
live_reload: [
patterns: [
~r{priv/static/.*(js|css|png|jpeg|jpg|gif)$},
~r{web/views/.*(ex)$},
~r{web/templates/.*(eex)$}
]
]
and this is actually translated into the following:
"myapp.Elixir.Myapp.Endpoint.live_reload.patterns": [
doc: "Provide documentation for myapp.Elixir.Myapp.Endpoint.live_reload.patterns here.",
to: "myapp.Elixir.Myapp.Endpoint.live_reload.patterns",
datatype: [
list: :binary
],
default: [
%{__struct__: Regex, opts: "", re_pattern: {:re_pattern, 1, 0, 0, <<69, 82, 67, 80, 157, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 112, 0, 47, 0, 0, 0, 1, 0, 0, 0, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, source: "priv/static/.*(js|css|png|jpeg|jpg|gif)$"},
%{__struct__: Regex, opts: "", re_pattern: {:re_pattern, 1, 0, 0, <<69, 82, 67, 80, 106, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 119, 0, 120, 0, 0, 0, 1, 0, 0, 0, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, source: "web/views/.*(ex)$"},
%{__struct__: Regex, opts: "", re_pattern: {:re_pattern, 1, 0, 0, <<69, 82, 67, 80, 116, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 119, 0, 120, 0, 0, 0, 1, 0, 0, 0, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...>>}, source: "web/templates/.*(eex)$"}
]
],
I have a config.exs value that is a list. When I go to make a release, I get a failure in conform.effective. I'm not sure if I'm intended to be able to use lists in a config, or how that would end up looking in my ops conf file when generating an exrm release.
If this is intended behaviour, maybe a more useful warning? If not, I'm happy to help put together a PR to fix this.
See my test project again, https://github.com/alco/conformist/commit/11df8b2f204578353694ae055d3f1d52fea26e28.
Every time I call mix release
, it keeps recompiling conform
. The reason could be that conform builds an escript during the release process. I haven't had time to confirm that though.
$ mix release
Building release with MIX_ENV=prod.
==> conform
Compiled src/conf_parse.erl
lib/conform/schema.ex:1: warning: redefining module Conform.Schema
Compiled lib/conform/schema/transform.ex
Compiled lib/conform/sysconfig.ex
Compiled lib/conform/schema/validator.ex
Compiled lib/conform/schema/mapping.ex
Compiled lib/conform.ex
Compiled lib/conform/type.ex
lib/mix/tasks/conform.archive.ex:1: warning: redefining module Mix.Tasks.Conform.Archive
Compiled lib/conform/types/enum.ex
Compiled lib/conform/validators/range_validator.ex
lib/mix/tasks/conform.configure.ex:1: warning: redefining module Mix.Tasks.Conform.Configure
Compiled lib/conform/parse.ex
lib/mix/tasks/conform.effective.ex:1: warning: redefining module Mix.Tasks.Conform.Effective
Compiled lib/mix/tasks/conform.archive.ex
lib/mix/tasks/conform.new.ex:1: warning: redefining module Mix.Tasks.Conform.New
lib/mix/tasks/conform.release.ex:1: warning: redefining module Mix.Tasks.Conform.Release
Compiled lib/mix/tasks/conform.release.ex
Compiled lib/conform/utils/utils.ex
Compiled lib/conform/conf.ex
Compiled lib/mix/tasks/conform.configure.ex
Compiled lib/mix/tasks/conform.new.ex
Compiled lib/conform/utils/code.ex
Compiled lib/mix/tasks/conform.effective.ex
Compiled lib/conform/schema.ex
Compiled lib/conform/translate.ex
Generated conform app
Generated escript conform with MIX_ENV=prod
==> The release for conformist-0.0.1 is ready!
==> You can boot a console running your release with `$ rel/conformist/bin/conformist console`
$ mix release
==> conform
Compiled src/conf_parse.erl
Compiled lib/conform/schema/transform.ex
Compiled lib/conform/sysconfig.ex
Compiled lib/conform/schema/mapping.ex
Compiled lib/conform/schema/validator.ex
Compiled lib/conform.ex
Compiled lib/conform/type.ex
Compiled lib/conform/conf.ex
Compiled lib/conform/validators/range_validator.ex
Compiled lib/conform/types/enum.ex
Compiled lib/conform/parse.ex
Compiled lib/mix/tasks/conform.archive.ex
Compiled lib/mix/tasks/conform.release.ex
Compiled lib/conform/utils/utils.ex
Compiled lib/mix/tasks/conform.configure.ex
Compiled lib/mix/tasks/conform.new.ex
Compiled lib/conform/utils/code.ex
Compiled lib/mix/tasks/conform.effective.ex
Compiled lib/conform/schema.ex
Compiled lib/conform/translate.ex
Generated conform app
Building release with MIX_ENV=prod.
==> conform
Generated escript conform with MIX_ENV=prod
==> The release for conformist-0.0.1 is ready!
==> You can boot a console running your release with `$ rel/conformist/bin/conformist console`
Hello,
I know that if the config.exs imports a config from a submodule, conform imports the settings from the imported config and adds them to the schema when using mix conform.new
. Never the less one would have to recreate the schema every time a dependency changes its settings or translations.
As a feature request I would like to ask for a function to import configurations of dependencies to add them into the configuration of the main applications.
As an example of how it should be used:
deps/dependency/config/dependency.schema.exs
[
mappings: [
"dependency.setting": [
doc: "Setting of dependency",
datatype: :binary,
default: "foo"
],
translations: []
]
config/myapp.schema.exs
[
import_settings: [
:dependency
],
mappings: [
"myapp.setting": [
doc: "Setting of myapp",
datatype: :binary,
default: "bar"
],
translations: []
]
using mix conform.configure
results in
# Setting of dependency
dependency.setting = "foo"
# Setting of myapp
myapp.setting = "bar"
The schemas must also be imported somehow of course to use the translations.
Sincerely,
Tobias
Seems, like conform.new
can't accept all config types on generation.
proxy = [{:default_route, {{127, 0, 0, 1}, 1813, "secret"}},
{:options, [{:type, :realm}, {:strip, true}, {:separator, '@'}]},
{:routes, [{'test', {{127, 0, 0, 1}, 1815, "secret"}}]}]
config :eradius, :servers,
proxy: {'127.0.0.1', [1812, 1813]}
config :eradius,
session_nodes: [node],
proxy: [{ {:eradius_proxy, 'proxy', proxy }, [{'127.0.0.1', "secret"}] }],
logfile: './radius.log',
client_ip: {0, 0, 0, 0}
** (Protocol.UndefinedError) protocol Enumerable not implemented for "secret"
(elixir) lib/enum.ex:1: Enumerable.impl_for!/1
(elixir) lib/enum.ex:112: Enumerable.reduce/3
(elixir) lib/enum.ex:1398: Enum.reduce/3
(elixir) lib/enum.ex:1106: Enum.map_reduce/3
(elixir) lib/macro.ex:226: Macro.do_postwalk/3
(elixir) lib/macro.ex:234: Macro.do_postwalk/3
(elixir) lib/enum.ex:1102: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
(elixir) lib/macro.ex:226: Macro.do_postwalk/3
07:57:38.606 [error] Error in process <0.46.0> with exit value: {#{'__exception__'=>true,'__struct__'=>'Elixir.Protocol.UndefinedError',description=>nil,protocol=>'Elixir.Enumerable',value=><<6 bytes>>},[{'Elixir.Enumerable','impl_for!',1,[{file,"lib/enum.ex"},{line,1}]},{'Elixir.Enumerable',reduce,3,[{file...
Issue: UTF8 characters in config files are not supported / handled properly.
Code: The code at https://github.com/bitwalker/conform/blob/master/src/conform_parse.erl#L155 converts utf8 encoded content explicitly to latin1.
Effect: So if we have e.g. a value like "ú..." (U+00FA) in the config file the output will be <<250,...>> which is no valid utf8 content. (the correct output would be <<195, 186,...>>)
Question 1: Why? Is there a problem with leaving content encoded as it is?
Question 2: Can the code please be updated / fixed to support utf8 encoded config files?
Invalid input configuration (even before schema checking) will cause errors displayed but exit status is 0 (success).
% ./conform --conf /etc/passwd --schema /dev/null --config /dev/null
Invalid conf file at line 1, column 1:
root:x:0:0:root:/root:/bin/bash
echo $?
0
Sorry, it was my fault, it was 2 o'clock night.
My project has the config file like following:
config.exs:
use Mix.Config
config :sts,
key: :value,
limit: 42
import_config "#{Mix.env}.exs"
config/dev.exs:
use Mix.Config
config :sts,
env: :dev
config/test.exs:
use Mix.Config
config :sts,
env: :test
I generate schema.exs as: mix conform.new
,
and I find the sts.env
is test
not dev
, shouldn't it dev
as default?
Furthermore, when I input as: MIX_ENV=dev mix conform.new
, it doesn't change the sts.env
Having problem with version
{:exrm, "> 1.0.0-rc5"},> 1.0.0-rc4"}
{:conform, "
MIX_ENV=prod mix release
Building release with MIX_ENV=prod.
==> conform
Unchecked dependencies for environment prod:
It works ok without conform
Hi there.
I have the following chunk of awesomeness in my config.exs
config :rocket, Rocket.Endpoint,
url: [host: "localhost"],
root: Path.dirname(__DIR__),
secret_key_base: "Hoopdie doopdie doo",
debug_errors: false,
http: [
dispatch: [
{:_, [
{"/ws", Pixie.Adapter.Cowboy.HttpHandler, {Pixie.Adapter.Plug, []}},
{"/messagerocket.js", :cowboy_static, {:file, Path.join(Path.dirname(__DIR__),"priv/static/messagerocket.js")}},
{:_, Plug.Adapters.Cowboy.Handler, {Rocket.Endpoint, []}}
]
}
]
]
And when I run mix conform.new
I get the following amazing backtrace:
** (FunctionClauseError) no function clause matching in Macro.do_traverse_args/4
(elixir) lib/macro.ex:290: Macro.do_traverse_args({Pixie.Adapter.Plug, []}, [], #Function<16.54478476/2 in Macro.postwalk/3>, #Function<11.54688206/2 in Conform.Utils.Code.stringify/1>)
(elixir) lib/macro.ex:269: Macro.do_traverse/4
(elixir) lib/enum.ex:1247: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
(elixir) lib/macro.ex:282: Macro.do_traverse/4
(elixir) lib/macro.ex:277: Macro.do_traverse/4
(elixir) lib/enum.ex:1247: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
(elixir) lib/enum.ex:1247: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
(elixir) lib/macro.ex:282: Macro.do_traverse/4
I am so excited to raise this bug for you to fix. You must be very happy as the maintainer of a popular open source project to serve my every whim.
All kidding aside, this is with conform 2.1.1 on Elixir 1.3.2 with Erlang 19. If there's any way I can help you diagnose this then feel free to ask.
I used mix conform.new
to generate a .schema from my config.exs in which I had this:
config :logger, :console,
level: :debug,
format: "$time $metadata[$level] $levelpad$message\n"
the generated .schema (and .config) didn't properly include the \n leading to no newlines in my logger.
I guess the recommended workaround is to write my own translation for logger.console.format that special cases \n characters?
On the readme, the schema is listed as having mappings, and translations (as in cuttlefish), but in the generated schema, there's mappings, transforms, and validators. This leads me to think that the readme is quite a bit out of date - is this correct, or am I reading it wrong? Is there a canonical place for up to date documentation?
For example here is the schema generated for a conf entry that takes a module name:
"backend": [
to: "my_app.Elixir.MyApp.backend",
datatype: :atom,
default: MyApp.SomeBackend
],
In the conf file, you have to prefix the value with Elixir.
:
backend = Elixir.MyApp.SomeBackend
Example:
[
mappings: [
"db.user": [
doc: """
username:password
""",
to: "app",
datatype: :binary,
default: "root:"
]
],
translations: [
"app": fn
_, user_pass, acc ->
case String.split(user_pass, ":") do
[user, pass] ->
repo_conf = acc[App.Repo] || []
put_in((acc || []), [App.Repo], [{:username, user}, {:password, pass} | repo_conf])
_ ->
IO.puts("Username and password should be separated with `:`, got: #{user_pass}")
exit(1)
end
end]
]
Conform.Schema.read!("config") |> Conform.Schema.stringify |> IO.puts
transforms to:
[
mappings: [
"db.user": [
doc: """
username:password
""",
to: "app",
datatype: :binary,
default: "root:"
]
],
translations: [
app: fn _, user_pass, acc ->
case String.split(user_pass, ":") do
userpass ->
repo_conf = acc[App.Repo] || []
put_in(acc || [], [App.Repo], [{:username, user}, {:password, pass} | repo_conf])
_ ->
IO.puts("Username and password should be separated with `:`, got: #{user_pass}")
exit(1)
end
end
]
]
Changes the line to: [user, pass] ->
to userpass
CC: @umatomba
On top of #89 I am debugging in issue around https://github.com/bitwalker/conform/blob/master/lib/mix/tasks/conform.archive.ex#L33 . I can only describe it as "the accumulator drops items". My example schema "extends" two other schemas. Those are found, but the accumulator somehow drops them.
Pulling my hair out on this one.
Hello @bitwalker, I have some issues:
I need to merge schemes from dependency of my application and some of its dependencies in release. I see that I can use extends
attribute for this purpose, but in the same way I see that conform uses Mix.Dep.children/0
to collect dependency schemes.
This leads to following problem: What if I want to merge schema from not direct dependency, but for example schema from dependency of a dependency? In this way we will get:
** (CaseClauseError) no case clause matching: []
lib/mix/tasks/conform.archive.ex:34: anonymous fn/4 in Mix.Tasks.Conform.Archive.run/1
(elixir) lib/enum.ex:1473: Enum."-reduce/3-lists^foldl/2-0-"/3
lib/mix/tasks/conform.archive.ex:23: Mix.Tasks.Conform.Archive.run/1
lib/conform_exrm.ex:27: ReleaseManager.Plugin.Conform.before_release/1
lib/mix/tasks/release.ex:248: anonymous fn/2 in Mix.Tasks.Release.execute_before_hooks/1
(elixir) lib/enum.ex:1473: Enum."-reduce/3-lists^foldl/2-0-"/3
lib/mix/tasks/release.ex:77: Mix.Tasks.Release.do_run/1
(mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2
Can I do it with the current conform? If not, what do you think If we will replace Mix.Dep.children
from the conform.archive.ex
with project_path/deps/extended_dep/config/schema
?
I will send PR soon for following discuss.
And also I have second question. I see that some transformers are arirty 1 and some transformers are arity 3 in conform's README.md. For all of my transformers I'm getting:
** (Conform.Schema.SchemaError) Invalid transform for my_mapping, it must be a function of arity 1.
error. How can I use transformers with arity 3?
Thank you.
Compiling conform in Elixir 1.3 shows three warnings that should be addressed:
warning: the variable "key" is unsafe as it has been set inside a case/cond/receive/if/&&/||. Please explicitly return the variable value instead. For example:
case int do
1 -> atom = :one
2 -> atom = :two
end
should be written as
atom =
case int do
1 -> :one
2 -> :two
end
Unsafe variable found at:
lib/conform/translate.ex:189
warning: function Mix.Dep.children/0 is undefined or private
lib/conform/schema.ex:395
warning: function Mix.Dep.children/0 is undefined or private
lib/mix/tasks/conform.archive.ex:24
Tested with conform 2.0
. If the application's sys.config
contains a tuple:
[{cqerl,[{maps,true},
{cassandra_nodes,[{<<"127.0.0.1">>,<<"9042">>}]},...
and the schema contains:
"cqerl.cassandra_nodes": [
datatype: [ list: :ip ],
default: [ {"127.0.0.1", "9099"} ], # note canary in port value
to: "cqerl.cassandra_nodes"
],
conform will generate an erroneous output sys.config
with a list of lists (instead of a list of tuples) holding the data from the input sys.config
list of tuples:
{cqerl,[{cassandra_nodes,[[<<"127.0.0.1">>,<<"9042">>]]},
I can not reopen #98 so I open a new issue.
on mix conform.effective
utf8 encoded content in .conf
file is supported,
but in a release it is still transcoded to latin-1.
https://github.com/basho/cuttlefish supports generating the vm.args
from the app.conf
, is something similar planned for conform? Just being able to set the node name and cookie would already solve all my problems.
Hi,
Currently some tasks, like conform.effective validates the environment used to execute it. Is this really necessary? For instance I use more than just three valid environments, and the task requires it to pass this test:
unless env in [:test, :dev, :prod] do
error "The value provided for --env is not a valid environment"
exit(:normal)
end
I've already modified the code locally to do what I want, and I've noticed no issues, if this isn't necessary I could create a pull request.
While
config :logger, :backends, [
:console,
{ExSyslog, :exsyslog_error},
{ExSyslog, :exsyslog_debug}
]
works fine with 1.0.0-pre's mix conform.new, and appears to generate a correct .schema.exs file:
"logger.backends": [
commented: false,
datatype: [
list: :atom
],
default: [
:console,
"ExSyslog": :exsyslog_error,
"ExSyslog": :exsyslog_debug
],
doc: "Provide documentation for logger.backends here.",
hidden: false,
to: "logger.backends"
],
, there is a failure when generating a new conf using mix conform.configure:
╰─$ mix conform.configure
** (ArgumentError) argument error
:erlang.atom_to_binary({:ExSyslog, :exsyslog_error}, :utf8)
lib/conform/translate.ex:275: Conform.Translate.write_datatype/3
(elixir) lib/enum.ex:1043: anonymous fn/3 in Enum.map/2
(elixir) lib/enum.ex:1385: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:1043: Enum.map/2
lib/conform/translate.ex:290: Conform.Translate.write_datatype/3
lib/conform/translate.ex:55: anonymous fn/2 in Conform.Translate.to_conf/1
(elixir) lib/enum.ex:1385: Enum."-reduce/3-lists^foldl/2-0-"/3
Not sure if this is a dupe of #20, but figured you might want to know since this is probably a very common configuration option for Elixir in production. Thanks for conform and exrm!!! ❤️
Why do you use custom stringify here instead of Macro.to_string
?
Link to original issue: bitwalker/distillery#92
We need to run conform's pre_start hook pre_upgrade as well now that the hooks for it are in distillery master.
warning: redefining @doc attribute previously set at line 10
lib/conform/schema.ex:17: Conform.Schema (module)
warning: redefining @doc attribute previously set at line 9
lib/conform/translate.ex:16: Conform.Translate (module)
warning: function Enum.member/2 is undefined or private
lib/conform/schema.ex:255
warning: function TranslateErorr.exception/1 is undefined (module TranslateErorr is not available)
lib/conform/translate.ex:398
I must add I'm new to the elixir world, so these may be my own fault..
For configuring guardian's ttl, the configuration expects a tuple in the form of {30, :days}. No matter what I do, conform seems to always turn this into [30, :days], which causes guardian's internal pattern matching to fail. Is there anyway to configure this?
Here's a link to guardian:
https://github.com/ueberauth/guardian#installation
config.exs
contains
config :engine,
host_id: "some uuid",
the generated engine.schema.exs
:
"engine.host_id": [
commented: false,
datatype: :binary,
default: "nice uuid",
doc: "Provide documentation for engine.host_id here.",
hidden: false,
to: "engine.host_id"
],
when I generate engine.conf
, it contains what I understand to be spurious double quotes:
# Provide documentation for engine.host_id here.
engine.host_id = "nice uuid"
And then this seems to be rather wrong:
$ mix conform.effective | grep host_id
engine: [host_id: "\"nice uuid\"", middlewares: [],
Not a showstopper, I am cleaning up the spurious double-quotes. But buggy.
If you clone this project https://github.com/alco/exrmtest and run mix conform.release
in it, it'll work the first time. But every subsequent attempt produces the following output:
$ mix conform.release
==> conform
Unchecked dependencies for environment dev:
* neotoma (Hex package)
the dependency is not locked (run "mix deps.get" to generate "mix.lock" file)
* ex_doc (Hex package)
the dependency is not available, run "mix deps.get"
* earmark (Hex package)
the dependency is not available, run "mix deps.get"
==> exrmtest
** (Mix) Can't continue due to errors on dependencies
When running it with MIX_ENV=prod
, it only complains about neotoma
:
$ MIX_ENV=prod mix conform.release
==> conform
Unchecked dependencies for environment prod:
* neotoma (Hex package)
the dependency is not locked (run "mix deps.get" to generate "mix.lock" file)
==> exrmtest
** (Mix) Can't continue due to errors on dependencies
The errors appear to come from the invocation of escript.build
but I haven't been able to investigate it further.
Elixir v1.2.3.
hey guys,
I have an umbrella app with 5 apps within umbrella. I use distillery to build the release, and edeliver for deployment.
Distillery is configured to build single release for whole umbrella application which as well results in generating a single sys.config
file.
Configuration is rather complex and I was hoping to use conform
, but I'm struggling a bit with making it work. For example, when I run mix conform.new
it goes into each app and generates a schema for each app under umbrella. Next task mix conform.configure
generates .conf
files for each app in respective config directory as well.
Now that's what I'm struggling to understand, please correct me if I'm wrong:
.conf
into sys.config
files according to rules in schema filesys.config
(with configs of 5 apps lumped together)How do I generate single schema and single .conf
file with this setup? and how does conform
fits into this?
Thank you,
Alex
# mix release
Building release with MIX_ENV=prod.
You have dependencies (direct/transitive) which are not in :applications!
The following apps should to be added to :applications in mix.exs:
conform_exrm -> conform -> neotoma => neotoma is missing from conform
Continue anyway? Your release may not work as expected if these dependencies are required! [Yn]:
Versions used:
conform_exrm 0.3.0
exrm 1.0.1
conform 1.0.0-rc8
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.