bbaugher / confluent Goto Github PK
View Code? Open in Web Editor NEWA Chef cookbook to install the Confluent Platform
License: MIT License
A Chef cookbook to install the Confluent Platform
License: MIT License
@bbaugher we use confluent cookbook in our repos as core and make necessary changes in local.
We provide "depends 'confluent'" in our metadata.rb but recent changes are not getting reflecting after chef build and run. Can you please suggest on how we can pull recent changes.
Starting and stopping the service does not work. The class attribute is wrong and should be changed from
default["confluent"]["schema-registry"]["class"] = "io.confluent.kafka.schemaregistry.rest.Main"
to
default["confluent"]["schema-registry"]["class"] = "io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain"
which seems to work.
Sorry due to company policy I cannot commit to open source projects hence the issue.
Currently we install the confluent package into /opt/confluent/confluent-VERSION. As more versions are installed this takes up more space but we likely don't need to maintain all our other versions
In the /etc/init.d/kafka script, the findPid() function uses java class to identify the kafka process:
CLASS=kafka.Kafka
findPid() {
FOUND_PID=`pgrep -u $USER -f $CLASS`
}
However, the java class listed in the command line for the kafka process is:
io.confluent.support.metrics.SupportedKafka
This prevents service commands for kafka from functioning properly. Modifying the class name in the init.d script resolves this issue.
Issue identified on confluent version 4.1.1. Have not tested on other versions.
In an early version of confluent the start script had a bug where it wouldn't start so we packaged the fix in our cookbook. It was only supposed to be applied to 1.0 but was actually applied to everything else
As we like and use this cookbook in production, I was curious if there was a way that you could add topic creations support within this cookbook. It would help support the managing the cluster from cookbook perspective.
Things that would be helpful are these things
It's a bad practices to derived attributes in default.rb because it's not recompute when you run the chef-client.
It will be a better place to put this code in recipes/_install.rb
version_numbers = node["confluent"]["version"].split(".")
archive_version = "#{version_numbers[0]}.#{version_numbers[1]}"
default["confluent"]["artifact_url"] = File.join("http://packages.confluent.io/archive", archive_version, "confluent-#{node["confluent"]["version"]}-#{node["confluent"]["scala_version"]}.zip")
See Avoid derived attributes in attribute files in
https://github.com/pulseenergy/chef-style-guide#attributes
As jars are deployed to connect via node['confluent']['kafka-connect']['jar_urls']
they are written to a confluent shared lib directory. These jars could change (update version or remove) and the JAR would still be in the shared directory. These JARs should be cleaned up so they are removed from connect's classpath
Almost all template resources have backup false
.
Could this be updated to be an attribute, defaulting to false, for those that would like to keep a backup of what changed?
Hi,
Both Kafka-rest and Schema-registry do not start after a reboot with Ubuntu 14.04 (first noticed after an image was created). They fail because Kafka at this time has not finished starting. Have tried forcing the priority with little effect.
It would also be useful to disable auto start of these services via attributes as I want to get them to use the auto broker ids which isn't possible due to the configuration.
Is the next release going to be out soon? I'd like to be able to use the recipes for kafka and schema registry. I'm cloning the repository at master but I'd like to be able to pin a version.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.