Support Logstash 1.2.x conditionals about chef-logstash HOT 13 CLOSED

I just finished up a first pass at enabling basic conditionals. Check the 0.7.0 branch.

Currently it just pulls out the 'type' field in a filter/output and turns it into an if statement. I have some thoughts around enabling other conditionals ... but it's not straightforward :)

from chef-logstash.

@jhmartin - FYI (and for others who may stumble on this), 8655b9f adds initial support for a condition key with a condition string that applies to a set of plugins. That's the approach I was going to suggest here, so good call @paulczar.

from chef-logstash.

@pdf @paulczar @jhmartin Hey Guys - it seems there is a major problem with the current implementation of the conditional logic. If we have an output config that contains non conditionals, pertains to all conditions, and a condition - you end up with only the condition in the output { } section. We need to rethink how we want to enable the conditions, b/c it also seems like if you have more than one condition you will end up with a problem as well.

My thoughts on this would be to have separate fields for non conditional configs - like:

default['logstash']['server']['outputs']

all other conditionals will go into a field like -

default['logstash']['server']['outputs_conditionals']

We can do this for both the outputs and filters - since they are the only ones that really are supported. This will make the backend logic more straight forward.

The overall goal would be to allow multiple conditionals and nested conditionals. With this in mind we should have a Conditional type of "object" that contains a LOGIC portion and a CONFIG portion. It can also have a nested Conditional "object" - that way we could take full advantage of nested IF ; ELSE-IF ; and ELSE logic. We then need to concatenate the non conditional and conditional blocks into a single filter {} or output {} blocks.

Doing it this way would be the most useful and in my mind the most easy to explain to end users. I am working on developing changes to support this logic now and should be done with it soon. Any other ideas or thoughts are welcome!

from chef-logstash.

@NickPadilla see another option in my comment here: #151 (comment)

While any one of numerous (breaking) changes are achievable to make it possible for Chef attributes to be bludgeoned into producing a valid Logstash config, my thinking has since shifted to believe that doing this stuff in attributes probably doesn't make any real sense. In the 0.7.0 branch we can just use templates instead. This lets you simply write logstash syntax, and use ERB for the dynamic parts. Personally, I think I'd consider deprecating attribute-based config generation entirely, in favour of templates.

from chef-logstash.

The attribute part has worked out to be very useful though; being able to set a role attribute that specifies the LS config for that role, and having chef merge them all together for a combined LS config, is quite useful.

from chef-logstash.

@jhmartin that use-case is easily satisfied by simply having a list of config templates on the role, no?

from chef-logstash.

If they get mashed together at the end? Yes, that would work.

from chef-logstash.

@pdf @jhmartin Hey Guys - While I agree that using the template direction is quickest, i believe it isn't in the best interest of the community. The whole idea, I thought, for community chef cookbooks is to allow server configuration providing minimal server "attributes". Using custom templates everywhere would negate the idea of code reuse. I don't think we would have to break anything for current cookbooks, just provide a better way of doing things. However if breaking a cookbook has a logical reason and actually benefits the use, then I'd have to say breaking changes are in order.

I do understand that complex logic makes things harder in the backend, however, I also believe that some servers are inherently complex to build. Providing a way to build out complex servers while also having code reuse would best serve the community at large, IMHO. If I can override attributes versus having to copy and paste a template - i'd take the attribute override.

from chef-logstash.

@jhmartin you're right, there needs to be a mechanism for merging the list of templates.

@NickPadilla see my comments inline:

The whole idea, I thought, for community chef cookbooks is to allow server configuration providing minimal server "attributes".

Attribute-driven configuration is great, where it makes sense, as it does with most configuration files that are essentially just key-value stores. I doubt it was envisioned being used to encapsulate a DSL like this.

Using custom templates everywhere would negate the idea of code reuse.

Using templates is no different at all to using attributes as far as re-use goes, since no one is sharing their attributes. And there's an argument to say that it's easier to share a template file than it is to share a data structure.

Providing a way to build out complex servers while also having code reuse would best serve the community at large, IMHO. If I can override attributes versus having to copy and paste a template - i'd take the attribute override.

Templates are actually far easier to use to create a complex Logstash file, and you're not just overriding an attribute here - you're trying to wedge a whole DSL into attributes. It's a square-peg, round-hole problem, and as I said above - there's no sensible way to share your attribute-described configurations either, so the code-sharing detractor is a red herring.

I don't think there's a cogent argument here for maintaining an attribute kludge that's going to be painful to maintain, is already a source of confusion for users, and is a poor fit for the complex structure of the Logstash configuration DSL.

from chef-logstash.

Hey @pdf - While I disagree with the premise that using templates is far easier than having a straight forward configuration design, I am not the maintainer of this cookbook and will move in the direction of the cookbook. A case can be made for attribute bloat versus template bloat and an easy to read attribute configuration is much easier to read - IMHO.

I would say then that the "support" of conditionals should be removed in "favor" of templates, and would agree that merging a list of templates would be helpful.

from chef-logstash.

I agree the future direction of the cookbook should be focussed more on templating, I think the data driven approach especially with 1.2.x will just be too difficult for larger configurations. That being said I do like to have basic attribute driven configuration for when I want to quickly test something out.

As for the problem described by @NickPadilla ... I can spin up a config with both conditional and non-conditional blocks in both filters and outputs. example - https://gist.github.com/paulczar/729c8bc5c6c10782cb1e

from chef-logstash.

@paulczar Hey Paul - it seems that you have your configs wrapped in an additional set of {}. I am sure this is the problem on my end. However, I have already removed my code and am moving in the direction of using a custom template.

from chef-logstash.

Closing out due to inactivity.

from chef-logstash.