johnynek / tormenta Goto Github PK

Scala extensions for the Storm distributed computation system. Tormenta adds a type-safe wrapper over Storm's Kafka and Kestrel spouts. This type safety allows the user to push mapping and filtering transformations down to the level of the spout itself:

// produces strings:
val scheme: ScalaScheme[String] = new StringScheme

// produces integers w/ string length:
val mappedScheme: ScalaScheme[Int] = scheme map { _.length }

// filters out all tuples less than 5:
val filteredScheme: ScalaScheme[Int] = mappedScheme filter { _ > 5 }

// produces lengths for input strings > length of 5
val spout: KestrelSpout[Int] = new KestrelSpout(filteredScheme, hostSeq, "spout")

To use a ScalaSpout[T] in a Storm topology, call the getSpout method:

topologyBuilder.setSpout("spoutName", spout.getSpout, 10)

Now you're cooking with gas.

Current version is 0.1.0. groupid="com.twitter" artifact="tormenta_2.9.2".

• Oscar Boykin https://twitter.com/posco
• Sam Ritchie https://twitter.com/sritchie

Copyright 2012 Twitter, Inc.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0