consumer.commit() causes Exception about kafka-python HOT 8 CLOSED

@rosejn Consumer offset commits are not supported in Kafka 0.8. This feature will be added in 0.8.1, so for now just set auto_commit=False in the consumer.

I added this feature before it was decided that the consumer offset commits would not be in the 0.8 release, and I didn't really want to take it out. Sorry for the confusion.

from kafka-python.

@mumrah is there support for commit offset in kafka trunk already?

What would you say if we update the 0.8 branch to match the kafka 0.8 branch and keep the trunks in sync with kafka trunk? This way, we could have commit disabled by default in 0.8 and leave them enabled in trunk.

At this point, I believe it makes more sense to be working on current stable version of 0.8.

I would update kafka-src to 0.8-beta1 as well in the 0.8 branch after merging with master.

from kafka-python.

@mrtheb Yes offset commits are in trunk.

+1 to updating kafka-src to the 0.8 release candidate
-1 to using 0.8 branch. This branch was used for the initial 0.8 protocol development and is pretty out of date.

AFAIK, offset commits are the only API incompatibility between Kafka 0.8 and trunk. I like the idea of a single version of kafka-python (for now) with a big warning about this feature not being 0.8 compatible. 0.8.1 should follow closely behind 0.8, so this won't be an issue for long (hopefully).

Basically, I'd like to avoid two actively maintained branches.

from kafka-python.

Ahh, ok, thanks for the quick responses. So if we are using Kafka as a message broker, what would be the best way to do this without offset commits? For example, If we connect a consumer then by default it will start at offset zero. I guess we can always seek to the latest offset to only handle new messages, but is there a way to just seek past what has already been consumed?

from kafka-python.

@rosejn until 0.8.1, you'll need to handle your own offsets unfortunately. You can seek to the end of the queue when you start consuming, but you might miss messages that way.

You might use a simple file to keep track of offsets in between restarting your Python app.

AFAIK, 0.8.1 will follow on closely behind 0.8 - so probably a few weeks from now.

from kafka-python.

When I worked on the Zookeeper support for the kafka-python client, I thought about adding support for this. Specially for folks using older versions of Kafka. We can easily store commit offsets in kafka.

Is there a way we can determine the kafka version at run-time in the kafka-python client? It will make life a lot simpler.

from kafka-python.

Maybe setting auto_commit=False would be better for now though?

Otherwise, it might need some clarifications but the Consumer has control over commit and I think it's fine the way it is.

Use cases:

• Kafka 0.8.0 (commit not supported)
  • Consumer(auto_commit=False) works in all cases (should be default IMO)
  • Consumer(auto_commit=True) will fail (BIG WARNING if this happens);
• Kafka 0.8.1 (commit supported)
  • Consumer(auto_commit=False) still works
  • Consumer(auto_commit=True) also works

from kafka-python.

I agree with @mrtheb, we should avoid the complexity of managing offsets.

It is my assumption that people who will start using 0.8 will upgrade to 0.8.1 once it is released. People who are using Kafka at this point are early adopters anyways, so it's likely they will keep up with the latest releases.

I've created a 0.8.0-1 milestone, once Kafka 0.8.0 is finalized, we should prepare for a release of kafka-python as well. This probably means setting auto_commit=False as the default and adding a warning message if it is set to true.

from kafka-python.