verizon / knobs Goto Github PK
View Code? Open in Web Editor NEWA reasonable configuration library for Scala
Home Page: http://verizon.github.io/knobs/
License: Apache License 2.0
A reasonable configuration library for Scala
Home Page: http://verizon.github.io/knobs/
License: Apache License 2.0
I'm trying to import files using string substitution but it seems it is not working as in the documentation
If I for example do this
web-server {
import "$(user.home)/test.cfg"
}
I get an exception saying
java.io.FileNotFoundException: /home/software/seqexec_server_gs_test-0.2.0/conf/$(user.home)/test.cfg (No such file or directory)
The same happens if I use $(HOME)
Should this work?
Hi,
I gave knobs a spin and this are my concerns:
Config
using the Typesafe
extension I got an error since in the CfgValue
AST there's no case for a Map
-like object. Is there a good reason for that? I mean, apart from the TSC integration I think it would be useful being able to define a list of maps in the config file.Typesafe
object just expose a config
method type which, internally, loads a TSC object using the default behaviour, that is ConfigFactory.load
. Since you can use other methods to load a TSC wouldn't it make sense exposing a method which accepts a TSC object and returns a knobs Config
? Of course keeping also the config
method type which uses the default behaviour.Config.require[A:Configured]
throws a KeyError
exception if the key is not found and this is reasonable. The problem is it throws that exception even if the key is present but the CfgValue
could have not been converted into A
. I think it should be changed to give it a more reasonable behaviour.Configured[Long]
. Is there a good reason for that?Once I was there playing with the toy I tried to solve the four issues aforementioned and, apparently, I made it. I updated the docs too to include the lists of maps value type.
Let me know if you may be interested in a PR of all or some of these new features.
P.S.: This is a REPL session of the new features, where objlist.cfg
is a simple config file as in:
objList = [{a = 42, b = 24}, {a = 1, b = 2}]
long = 10000000000
hi = "hello"
REPL session:
scala> import java.io.File
import java.io.File
scala> import knobs._
import knobs._
scala> import knobs.{Required,FileResource,Config}
import knobs.{Required, FileResource, Config}
scala> import scalaz.concurrent.Task
import scalaz.concurrent.Task
scala> val cfg: Task[Config] = knobs.loadImmutable(Required(FileResource(new File("core/src/test/resources/objlist.cfg"))) :: Nil)
cfg: scalaz.concurrent.Task[knobs.Config] = scalaz.concurrent.Task@26b3524c
scala> val config = cfg.run
config: knobs.Config = Config(Map(objList -> CfgList(List(CfgMap(Map(a -> CfgNumber(42.0), b -> CfgNumber(24.0))), CfgMap(Map(a -> CfgNumber(1.0), b -> CfgNumber(2.0))))), long -> CfgNumber(1.0E10), hi -> CfgText(hello)))
scala> config.require[List[Map[String, Int]]]("objList")
res0: List[Map[String,Int]] = List(Map(a -> 42, b -> 24), Map(a -> 1, b -> 2))
scala> config.require[List[CfgValue]]("objList")
res1: List[knobs.CfgValue] = List(CfgMap(Map(a -> CfgNumber(42.0), b -> CfgNumber(24.0))), CfgMap(Map(a -> CfgNumber(1.0), b -> CfgNumber(2.0))))
scala> config.require[CfgValue]("objList")
res2: knobs.CfgValue = CfgList(List(CfgMap(Map(a -> CfgNumber(42.0), b -> CfgNumber(24.0))), CfgMap(Map(a -> CfgNumber(1.0), b -> CfgNumber(2.0)))))
scala> res2.pretty
res3: String = [{a = 42.0, b = 24.0}, {a = 1.0, b = 2.0}]
scala> config.require[Long]("long")
res4: Long = 10000000000
scala> config.require[Int]("missingKey")
knobs.KeyError: No such key: missingKey
at knobs.Config$$anonfun$1.apply(Config.scala:40)
at knobs.Config$$anonfun$1.apply(Config.scala:40)
...
scala> config.require[Int]("hi")
knobs.ConversionError: Could not convert "hello" to int
at knobs.Config$$anonfun$require$1.apply(Config.scala:41)
at knobs.Config$$anonfun$require$1.apply(Config.scala:41)
...
How would I go about reading something like this:
something = [{
key1 = "yay",
key2 = "yoi"
},
{
key1 = "food",
key2 = "insurmountable"
},
]
?
It looks like the latest version of this project now supports Cats. It would be great to have some documentation/examples of how to use this with Cats.
Custom watching implementation in knobs can be replaced with WatchService
wrapper from fs2.
It looks like ๐น the loadp
method in knobs/package.scala
doesn't give access to the thread pool used to load stuff.
Can we get access to this?
perhaps changing
โ
private [knobs] def loadp(paths: List[(Name, KnobsResource)]): Task[BaseConfig] = for {
// snip
_ <- Task(ticks.evalMap(_ => bc.reload).run.runAsync(_.fold(_.printStackTrace, _ => ())))
} yield bc
to something like this
private [knobs] def loadp(paths: List[(Name, KnobsResource)])(implicit pool: ExecutorService = Strategy.DefaultExecutorService): Task[BaseConfig] = for {
// snip
_ <- Task(ticks.evalMap(_ => bc.reload).run.runAsync(_.fold(_.printStackTrace, _ => ())))(pool)
} yield bc
and chain it up to the public load
/loadImmutable
methods?
๐น correct me if I'm wrong
I'm not sure if the docs are previewing something yet to come or talking about something that is no longer with us, but the docs reference this resource:
URLResource - loads any URI supported by the class java.net.URI.
Alas, though it sounds terrific, there is no URLResource that I can see.
As far as I can tell, there's no way to put an infinite duration as a value in a knobs config file. Therefore it seems to me like CfgDuration
should wrap a FiniteDuration
instead of a Duration
. This would make it so that consumers don't have to tell the compiler "trust me, this is going to be a finite value".
If this isn't changed, I think knobs should at least provide a Configured[FiniteDuration]
instance that returns None
if the duration is infinite.
Either way I'm happy to submit a PR -- I just wanted to get some feedback on which approach is preferred.
In Typesafe config, we can have non-string values read from environment variables defaulting to other values if no environment variable is set:
foo {
bar = 123
bar = ${?BAR}
}
When there is no BAR
environment variable set, Typesafe config will find 123
for the key foo.bar
but in Knobs, knobs.Typesafe.config.run.require[Int]("foo.bar")
fails with knobs.KeyError: No such key: foo.bar
Currently a FileResource
will always result in a Watch
ed resource. I think this should probably be optional, as sometimes you know that a file is not going to change, or you do not want your config to reload if it does, or you want to always explicitly reload.
Now that cats-effect 1.0 is out it'd be nice if this used it. I am receiving errors like this:
Exception in thread "main" java.lang.NoSuchMethodError: cats.effect.Async.shift(Lscala/concurrent/ExecutionContext;)Ljava/lang/Object;
at knobs.Resource$.watchStream(Resource.scala:155)
at knobs.Resource$.$anonfun$watchEvent$4(Resource.scala:222)
at knobs.Resource$.$anonfun$watchEvent$4$adapted(Resource.scala:221)
at cats.effect.IO$Map.apply(IO.scala:1407)
at cats.effect.IO$Map.apply(IO.scala:1403)
at cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:216)
at cats.effect.internals.IORunLoop$.step(IORunLoop.scala:216)
at cats.effect.IO.unsafeRunTimed(IO.scala:321)
at cats.effect.IO.unsafeRunSync(IO.scala:240)
I assume this is because the cats-effect version is a bit older?
I get this at runtime
java.lang.NoSuchMethodError: scalaz.Free$.point(Lscala/Function0;)Lscalaz/Free;
java.lang.NoSuchMethodError: scalaz.Free$.point(Lscala/Function0;)Lscalaz/Free;
at scalaparsers.Parsing$$anon$3$$anon$2.apply(ParsingUtil.scala:37)
at scalaparsers.Parser$$anon$7$$anonfun$apply$4.apply(Parser.scala:59)
at scalaparsers.Parser$$anon$7$$anonfun$apply$4.apply(Parser.scala:58)
at scalaz.Free$$anonfun$resume$2.apply(Free.scala:107)
at scalaz.Free$$anonfun$resume$2.apply(Free.scala:107)
at scalaz.std.FunctionInstances$$anon$1$$anonfun$map$1.apply(Function.scala:77)
at scalaz.Free$$anonfun$run$1.apply(Free.scala:239)
at scalaz.Free$$anonfun$run$1.apply(Free.scala:239)
at scalaz.Free.go2$1(Free.scala:150)
at scalaz.Free.go(Free.scala:153)
at scalaz.Free.run(Free.scala:239)
at scalaparsers.Parser.run(Parser.scala:21)
at knobs.ConfigParser$.runParser(Parser.scala:203)
at knobs.ConfigParser$ParserOps.parse(Parser.scala:33)
at knobs.Resource$$anonfun$loadFile$1$$anonfun$apply$2$$anonfun$apply$3.apply(Resource.scala:190)
at knobs.Resource$$anonfun$loadFile$1$$anonfun$apply$2$$anonfun$apply$3.apply(Resource.scala:190)
at scalaz.concurrent.Task$$anonfun$delay$1.apply(Task.scala:272)
at scalaz.concurrent.Task$$anonfun$delay$1.apply(Task.scala:272)
at scalaz.concurrent.Task$$anonfun$suspend$1$$anonfun$4.apply(Task.scala:280)
at scalaz.concurrent.Task$$anonfun$suspend$1$$anonfun$4.apply(Task.scala:280)
at scalaz.concurrent.Task$.Try(Task.scala:387)
at scalaz.concurrent.Task$$anonfun$suspend$1.apply(Task.scala:280)
at scalaz.concurrent.Task$$anonfun$suspend$1.apply(Task.scala:280)
at scalaz.concurrent.Future.step(Future.scala:110)
at scalaz.concurrent.Future.listen(Future.scala:75)
at scalaz.concurrent.Future.runAsync(Future.scala:142)
at scalaz.concurrent.Future.run(Future.scala:159)
at scalaz.concurrent.Task.run(Task.scala:97)
I am suspecting a binary compatibility issue. Not sure, though.
Here are some more dependencies:
lazy val Http4sVersion = "0.15.0-SNAPSHOT"
lazy val CirceVersion = "0.4.1"
libraryDependencies ++= Seq(
"oncue.knobs" %% "core" % "3.8.1",
"io.circe" %% "circe-core" % CirceVersion,
"io.circe" %% "circe-generic" % CirceVersion,
"io.circe" %% "circe-parser" % CirceVersion,
"org.http4s" %% "http4s-dsl" % Http4sVersion,
"org.http4s" %% "http4s-circe" % Http4sVersion,
"org.http4s" %% "http4s-blaze-client" % Http4sVersion
Is it possible to replace scalaz streams with fs2 streams, considering that many libraries are now moving toward the cats/fs2 ecosystem?
Knobs is working well for me and working as expected for my use cases, but I've been seeing a NullPointerException output from a different thread. Apparently there's an issue in watchEvent and an unhandled error. As this isn't on the main thread, it doesn't abort any processing and isn't trappable by the calling app (as far as I know).
Here's the exception stack trace:
java.lang.NullPointerException
at knobs.Resource$.watchEvent(Resource.scala:181)
at knobs.Resource$$anon$1$$anonfun$watch$1$$anonfun$apply$12$$anonfun$apply$13$$anonfun$apply$14.apply(Resource.scala:194)
at knobs.Resource$$anon$1$$anonfun$watch$1$$anonfun$apply$12$$anonfun$apply$13$$anonfun$apply$14.apply(Resource.scala:194)
at scalaz.stream.Process$$anonfun$map$1.apply(Process.scala:53)
at scalaz.stream.Process$$anonfun$map$1.apply(Process.scala:53)
at scalaz.stream.Process$$anonfun$flatMap$1.apply(Process.scala:46)
at scalaz.stream.Process$$anonfun$flatMap$1.apply(Process.scala:46)
at scalaz.stream.Util$.Try(Util.scala:38)
at scalaz.stream.Process$class.flatMap(Process.scala:46)
at scalaz.stream.Process$Emit.flatMap(Process.scala:577)
at scalaz.stream.Process$class.map(Process.scala:53)
at scalaz.stream.Process$Emit.map(Process.scala:577)
at knobs.Resource$$anon$1$$anonfun$watch$1$$anonfun$apply$12$$anonfun$apply$13.apply(Resource.scala:194)
at knobs.Resource$$anon$1$$anonfun$watch$1$$anonfun$apply$12$$anonfun$apply$13.apply(Resource.scala:194)
at scalaz.concurrent.Task$.Try(Task.scala:386)
at scalaz.concurrent.Task$$anonfun$apply$10.apply(Task.scala:286)
at scalaz.concurrent.Task$$anonfun$apply$10.apply(Task.scala:286)
at scalaz.concurrent.Future$$anonfun$apply$15$$anon$4.call(Future.scala:380)
at scalaz.concurrent.Future$$anonfun$apply$15$$anon$4.call(Future.scala:380)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
And some other details that may be useful:
Mac OS X 10.10.5
Scala 2.11.7
Knobs 3.3.0
Java 1.8
The BaseConfig
class keeps its internal mutable state as IORef
s, and we should change this to scalaz.stream.async.mutable.Signal
to make it easier to subscribe to changes. Users should be able to get a Process[Task,Config]
from (or even instead of) a MutableConfig
.
Greetings,
I just tried loading a ClassPathResource, following the example in the Usage documentation, but I'm getting a type mismatch error...I'm guessing I'm missing something extremely basic, but for the moment I'm stumped...here's the example (I've tried using knobs-core 3.2.1 and 3.2.2-SNAPSHOT):
import knobs._
import scalaz.concurrent.Task
val config: Task[Config] = knobs.loadImmutable(Required(ClassPathResource("knobs-example.cfg")))
<console>:20: error: type mismatch;
found : knobs.Required[knobs.ResourceBox]
required: List[knobs.KnobsResource]
(which expands to) List[knobs.Worth[knobs.ResourceBox]]
val config: Task[Config] = knobs.loadImmutable(Required(ClassPathResource("knobs-example.cfg")))
Am I missing or misunderstanding something simple here? Any help would be appreciated.
Thanks,
Tim
Knobs has a couple notable aspects to its versioning strategy:
a
suffix convention. I think this is based on which version of scalaz-stream it depends on, but I'm not sure that I totally understand it.This versioning strategy should be documented.
It looks like internally Typesafe config is storing durations in a stringy fashion. So when Typesafe config is converted to knobs config, it becomes a CfgText
as opposed to a CfgDuration
. And then when you try to look it up as a Duration
, you get a None
instead of the actual value.
Unfortunately, I don't know of any way that you could solve this within the typesafe module itself, unless you proactively attempt to parse every String value as a duration and fall back to string if that doesn't work which sounds really gross and not necessarily what's desired.
One other option is to have the Configured[Duration]
instance attempt to parse the CfgText
value if that's what it gets instead of CfgDuration
. I think that this could solve the problem without really being harmful. The only downsides that I can think of are that it is a change to core
that is really meant to support typesafe
and that it is just a little less simple than the current approach.
Example below.
scala> import com.typesafe.config.ConfigFactory, scala.concurrent.duration.Duration
import com.typesafe.config.ConfigFactory
import scala.concurrent.duration.Duration
scala> val ts = ConfigFactory.parseString(""" "dur" = 3 seconds """)
ts: com.typesafe.config.Config = Config(SimpleConfigObject({"dur":"3 seconds"}))
scala> ts.getDuration("dur", java.util.concurrent.TimeUnit.MILLISECONDS)
res0: Long = 3000
scala> val k = knobs.Typesafe.config(ts).run // YOLO
k: knobs.Config = Config(Map(dur -> CfgText(3 seconds)))
scala> k.lookup[Duration]("dur")
res1: Option[scala.concurrent.duration.Duration] = None
scala> k.lookup[String]("dur")
res2: Option[String] = Some(3 seconds)
Hi,
I found an error in the documentation. I was about to open a PR to update the doc, but the file is already up to date with the Maven Central repository information. Maybe the docs just need to be regenerated.
https://verizon.github.io/knobs/usage is the page where I found the invalid information.
Thanks.
We've done something like this to support decoding dates form our config files
implicit val configuredLocalDate: Configured[LocalDate] = new Configured[LocalDate] {
def apply(a: CfgValue) = Some(LocalDate.parse(a.pretty.replace("\"", ""), ofPattern("dd MMM yyyy")))
}
Is that something that could/should go into the CfgValue
trait? We'd do a pull request but aren't sure how you'd prefer it to slot in.
I'd guess you'd need a 2.12 version of https://github.com/Verizon/sbt-rig beforehand, I'm hitting
[warn] Note: Unresolved dependencies path:
[warn] org.scalatest:scalatest_2.12:2.2.6 ((verizon.build.common) RigPlugin.scala#L79)
[warn] +- io.verizon.knobs:knobs_2.12:3.11.0-SNAPSHOT
[warn] org.scalacheck:scalacheck_2.12:1.12.5 ((verizon.build.common) RigPlugin.scala#L79)
[warn] +- io.verizon.knobs:knobs_2.12:3.11.0-SNAPSHOT
[warn] org.scoverage:scalac-scoverage-runtime_2.12:1.1.1 ((scoverage.ScoverageSbtPlugin) ScoverageSbtPlugin.scala#L30)
[warn] +- io.verizon.knobs:knobs_2.12:3.11.0-SNAPSHOT
[warn] org.scoverage:scalac-scoverage-plugin_2.12:1.1.1 ((scoverage.ScoverageSbtPlugin) ScoverageSbtPlugin.scala#L30)
[warn] +- io.verizon.knobs:knobs_2.12:3.11.0-SNAPSHOT
Are there any plans to add defaults for environment variables?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.