tylerholien / milena Goto Github PK
View Code? Open in Web Editor NEWthe (very humble) beginnings of a kafka client for haskell
License: BSD 3-Clause "New" or "Revised" License
the (very humble) beginnings of a kafka client for haskell
License: BSD 3-Clause "New" or "Revised" License
I'm working on this right now. I'll hopefully have something by the end of today.
It would be great if milena supported user authentication, introduced in kafka 0.9.
No email or twitter on your GH account, so just popping in to say that I'm glad Milena is still bumping along :) 馃憤 馃惢 馃嵂 馃槃
Not cereal
. Why you ask?
Because of stuff like this: http://hackage.haskell.org/package/cereal-0.4.1.0/docs/Data-Serialize.html#v:decode
Ick.
Possible alternative (still in development): https://github.com/NICTA/tickle
This isn't an immediate priority, we can roll with the homespun stuff for now just to focus on the more Kafka specific stuff. Hopefully around the time we're ready to tackle serialization, Tickle is ready.
Any thoughts on the delta between Kafka 0.8 and 0.9? I'm looking to use Kafka 0.9 + melena so if there isn't already progress we might send in a PR - any experience you already have on this path would be appreciated.
We're gonna start deploying a couple of milena applications to our staging servers this week. It'd be best for us to use a release rather than master.
I had been using code in the form of:
xs <- parseFile
produceMessages xs
In one case length xs == 70000
and the process took significant processing power. Changing the code to instead use mapM_ produceMessages (chunksOf 1024 xs)
is a huge improvement. Perhaps the exactly implications of produceMessages (xs ++ ys)
vs produceMessages xs >> produceMessages ys
should be documented?
If I try to use an invalid topic name, I get a KafkaDeserializationError
.
Error code 17 is documented as "InvalidTopicName" here: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ErrorCodes
Took me a while to figure out what I am doing wrong. So it would be great if this error could be handled in order to provide a better error message.
Example:
位> :set -XOverloadedStrings
位> let k = mkKafkaState "foo" ("192.168.99.100", 9092)
位> runKafka k $ produceMessages [TopicAndMessage "\"foo" (makeMessage "msg")]
Left (KafkaDeserializationError "Failed reading: invalid error code: 17\nEmpty call stack\n")
Well. I pinged the two Zookeeper client maintainers. They both said their respective clients worked.
https://hackage.haskell.org/package/hzk-1.0.0 - they aren't properly Cabalized
https://github.com/motus/haskell-zookeeper-client
I really have no idea for sure if they work. This a bookmark for further investigation and seeing what we want to do about integration.
I'm still not sold on clients controlling their offsets, but there might be other reasons to need/want to talk to the ZK cluster.
Most or at least many haskell packages went the way of using MonadThrow instead of typed Exceptions for IO code.
See Michael Snoyberg's essay for a rationale.
Would you accept a pull request with simple non-derived instances for show and read that render topics as any other text?
Hi, is Kafka 0.9.0 support planned?
Looks like it introduced some features that would make life easier for writing consumers, such as the ability to subscribe to a topic and built-in balancing consumer functionality.
Could we get a point release?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
馃枛 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 馃搳馃搱馃帀
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google 鉂わ笍 Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.