Giter VIP home page Giter VIP logo

Comments (11)

mhausenblas avatar mhausenblas commented on June 11, 2024 1

Got it. It would be awesome to get an PR on this! IMO it theoretically should be as simple as adding fn(…) somewhere in this fragment. And: no worries, Go is simple to read and I suppose rather simple to write as well ;)

from burry.sh.

mhausenblas avatar mhausenblas commented on June 11, 2024

Thanks for the kind words @geekity!

Now, concerning your issue, I'm not entirely sure what the reason is (since it's sorta hard to reproduce your exact environment) but here are two things OTTOMH that you could check:

  • The problem could be in the write-path, that is, during backup. So, when you do the backup to your local drive and have a look at the directory test-kafka10-test-eu-west-1/brokers/topics/test is the data there? I'm not entirely sure but to me the two screen shots look like from the same cluster?
  • The problem could be read-path, that is, during restore. As you can see from this comment in zk.go only znodes that don't already exist will be created (this is a safety feature I built in to make sure not to unintentionally overwrite something valuable without a chance to undo it).

Can you pls first check if the backup process actually created the data on disk correctly and then we take it from there?

from burry.sh.

geekity avatar geekity commented on June 11, 2024

Thanks for the quick response! :)

With regards to your suggestion:

  1. I checked the dump of data from my Exhibitor cluster 1 and under test-kafka10-test-eu-west-1/brokers/topics/test there is only a partitions folder and no content file (that's where the znode data is saved, yea?). I can see how the 2 screen shots look very similar (since the second is a replica of the first) but they're definitely 2 different clusters.

  2. The second zookeeper cluster is empty when I try to copy into it. I just spun all the infrastructure up for testing. I don't think the overwrite safety feature can apply to this case so. Good thing to know about it though, I hadn't noticed it earlier.

With regards to getting the dump of data from Exhibitor cluster 1 and storing it to local, this was the output from the command:

➜ burry -e 172.31.128.160:2181 -t local
INFO[0000] Selected operation: BACKUP                    func=main
INFO[0000] My config: {InfraService:zk Endpoint:172.31.128.160:2181 StorageTarget:local Creds:{StorageTargetEndpoint: Params:[]}}  func=main
2017/05/17 17:16:06 Connected to 172.31.128.160:2181
2017/05/17 17:16:06 Authenticated: id=3917030430742675462, timeout=4000
2017/05/17 17:16:06 Re-submitting `0` credentials after reconnect
INFO[0001] Operation successfully completed. The snapshot ID is: 1495037766  func=main
➜ ls
1495037766.zip

If you'd like I can upload the full dump for you, it was a test cluster with nothing else in it.

I'd say for reproducing this problem you probably don't need to run all the extra replicas, just having a Kafka (I used Kafka 0.10.2.1) and Zookeeper and making a dummy test topic should be sufficient. The data missing between the origin and copy is the Kafka partition and replica assignment to different Kafka brokers for the given topic.

from burry.sh.

mhausenblas avatar mhausenblas commented on June 11, 2024

Thanks for the detailed report here. Hmmm, interesting … now, I'm still not entirely sure what the cause might be. Can you please try to run it with debug-level logging enabled, that is, DEBUG=true burry -e 172.31.128.160:2181 -t local so that we can have a closer look at what's going on?

For the sake of completeness, the hunch I have is that it's an ephemeral znode that doesn't get reaped properly (which I thought would work correctly, so if that's true then I need to check if it's an issue with the ZK package I'm using or if it's me using it incorrectly).

from burry.sh.

geekity avatar geekity commented on June 11, 2024

Here's the command output.

➜ DEBUG=true burry -e 172.31.170.190:2181 -t local
INFO[0000] Selected operation: BACKUP                    func=main
INFO[0000] My config: {InfraService:zk Endpoint:172.31.170.190:2181 StorageTarget:local Creds:{StorageTargetEndpoint: Params:[]}}  func=main
DEBU[0000] On node /                                     func=visitZK
2017/05/18 18:03:55 Connected to 172.31.170.190:2181
2017/05/18 18:03:55 Authenticated: id=4205267394443673606, timeout=4000
2017/05/18 18:03:55 Re-submitting `0` credentials after reconnect
DEBU[0000] / has 2 children                              func=visitZK
DEBU[0000] Next visiting child /zookeeper                func=visitZK
DEBU[0000] On node /zookeeper                            func=visitZK
DEBU[0000] /zookeeper has 1 children                     func=visitZK
DEBU[0000] Next visiting child /zookeeper/quota          func=visitZK
DEBU[0000] On node /zookeeper/quota                      func=visitZK
DEBU[0000] /zookeeper/quota has 0 children               func=visitZK
DEBU[0000] Stored /zookeeper/quota in /Users/nhanzlikova/training/burry/1495127035/zookeeper/quota with 0 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1          func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1 has 8 children   func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/consumers  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/consumers  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/consumers has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/consumers in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/consumers with 0 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/cluster  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/cluster  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/cluster has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/cluster/id  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/cluster/id  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/cluster/id has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/cluster/id in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/cluster/id with 45 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config   func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config has 3 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config/topics  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config/topics  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config/topics has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config/topics/test  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config/topics/test  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config/topics/test has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/config/topics/test in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/config/topics/test with 25 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config/clients  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config/clients  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config/clients has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/config/clients in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/config/clients with 0 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/config/changes  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/config/changes  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/config/changes has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/config/changes in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/config/changes with 0 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/controller  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/controller  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/controller has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/controller in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/controller with 54 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/isr_change_notification  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/isr_change_notification  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/isr_change_notification has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/isr_change_notification in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/isr_change_notification with 0 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/admin  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/admin    func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/admin has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/admin/delete_topics  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/admin/delete_topics  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/admin/delete_topics has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/admin/delete_topics in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/admin/delete_topics with 0 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers has 3 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/seqid  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/seqid  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/seqid has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/seqid in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/seqid with 0 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions has 6 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3 has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state with 76 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2 has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state with 76 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1 has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state with 76 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0 has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state with 76 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5 has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state with 76 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4 has 1 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state with 76 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids has 6 children  func=visitZK
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/3  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/3  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/3 has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/3 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/3 with 278 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/2  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/2  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/2 has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/2 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/2 with 280 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/1  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/1  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/1 has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/1 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/1 with 280 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/6  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/6  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/6 has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/6 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/6 with 280 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/5  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/5  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/5 has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/5 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/5 with 280 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/brokers/ids/4  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/brokers/ids/4  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/brokers/ids/4 has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/brokers/ids/4 in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/ids/4 with 280 bytes  func=store
DEBU[0000] Next visiting child /test-kafka10-test-eu-west-1/controller_epoch  func=visitZK
DEBU[0000] On node /test-kafka10-test-eu-west-1/controller_epoch  func=visitZK
DEBU[0000] /test-kafka10-test-eu-west-1/controller_epoch has 0 children  func=visitZK
DEBU[0000] Stored /test-kafka10-test-eu-west-1/controller_epoch in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/controller_epoch with 1 bytes  func=store
DEBU[0000] Added metadata to /Users/nhanzlikova/training/burry/1495127035  func=addmeta
DEBU[0000] 1495127035/.burrymeta                         func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/admin/delete_topics/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/1/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/2/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/3/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/4/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/5/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/ids/6/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/seqid/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/0/state/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/1/state/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/2/state/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/4/state/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/5/state/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/cluster/id/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/config/changes/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/config/clients/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/config/topics/test/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/consumers/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/controller/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/controller_epoch/content  func=arch
DEBU[0000] 1495127035/test-kafka10-test-eu-west-1/isr_change_notification/content  func=arch
DEBU[0000] 1495127035/zookeeper/quota/content            func=arch
DEBU[0000] Backup available in /Users/nhanzlikova/training/burry/1495127035.zip  func=arch
INFO[0000] Operation successfully completed. The snapshot ID is: 1495127035  func=main

This was actually run on a replicated set-up described above, since I had deleted my original clusters due to running costs. I had set up the configuration and topic the same as last time, although the exact data expected in the test znode is slightly different. To make sure everything is still the same from dump perspective, I checked the dump path test-kafka10-test-eu-west-1/brokers/topics/test, which still only contains the partitions folder and no data.

from burry.sh.

mhausenblas avatar mhausenblas commented on June 11, 2024

Thank you! This is really strange since the detailed output tells us that apparently there was data, so properly reaped, for example here:

Stored /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state in /Users/nhanzlikova/training/burry/1495127035/test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state

Just to make 100% sure and to confirm: when you look into the file 1495127035.zip then there is no sub-dir /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state, yes?

from burry.sh.

geekity avatar geekity commented on June 11, 2024

There is a sub-dir /test-kafka10-test-eu-west-1/brokers/topics/test/partitions/3/state, which has content. I think the problem is with /test-kafka10-test-eu-west-1/brokers/topics/test, which should have a content for the data in the znode but only has a sub-dir partitions. Could the problem be that the znode normally stores data as well as acts as a parent to more znodes?

from burry.sh.

mhausenblas avatar mhausenblas commented on June 11, 2024

OMG, you're absolutely correct! Only leaf nodes are reaped (== stored) , see here but that can certainly be changed. I'm not 100% sure why I had that policy in the first place but maybe it's time to review that decision?

from burry.sh.

geekity avatar geekity commented on June 11, 2024

Well, Kafka uses the top level topic node for storing the partition-to-broker mappings, and the leaf nodes for the current ISR data. Without the partition-to-broker mapping the data on the brokers is pretty inaccessible (unless you manually restore this mapping, which would probably be pretty tricky) afaik. In my case, Kafka coordination is my main use case for Zookeeper, so if you do decide to revisit the policy, I'd be all for it :D

from burry.sh.

mhausenblas avatar mhausenblas commented on June 11, 2024

Yeah, I can't think of any reason now that this should not be valid use case (or cause some data corruption) so let's do it. Now, I'm on travel from tomorrow on, heading to Sydney for a week and hence it would be helpful if you could give me a bit of an indication how urgent and how important this is for you. I'm more than happy to put a few cycles into it, essentially working on a branch that you can try out and when we're happy make this the master branch.

from burry.sh.

geekity avatar geekity commented on June 11, 2024

It's not particularly urgent, so please don't worry on my account. I was looking for a tool to use at work for backing up my Zookeeper and this seems like a really nice one, but it was more preventative rather than urgent. If you're busy and you'd like I can have a crack at making a PR with it, although it would be my first time with Go, but if you're not taking PRs I'm also perfectly happy leaving it in your hands.

from burry.sh.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.