meltwater / rabbitio Goto Github PK

RabbitIO is used to move data in and out of RabbitMQ

License: Apache License 2.0

Makefile 2.18% Go 97.82%

rabbitio's Introduction

RabbitIO

RabbitIO is a tool to backup and restore RabbitMQ messages.

To learn more about common problems with queue management and our reasoning and design principles when building RabbitIO, please read this blog post.

Typical use case

A typical use case is to consume all messages from a RabbitMQ deadletter queue to your local dev machine, do some analysis of the messages, and possibly push (some of) them back to an exchange for reprocessing.

Installing

Download binary

Pick your binary from Releases and download, in addition you'll need to set the binary to be executable.

Example with linux-amd64 and version v0.5.4:

wget https://github.com/meltwater/rabbitio/releases/download/v0.5.4/rabbitio-v0.5.4-linux-amd64 -O rabbitio
chmod 755 rabbitio

Using `go get`

go get -u github.com/meltwater/rabbitio

How to use RabbitIO

After installing rabbitio, you can quickly test out rabbitio by using docker-compose.

Included is a docker-compose file to set up local rabbitmq.

cd $GOPATH/src/github.com/meltwater
docker-compose up -d

Go to your now running local rabbit and create example exchange rabbitio-exchange and queue rabbitio-queue. Then bind the queue to the exchange.

Publish your first message

echo "My first message" > message # write a message into a file
tar cfz message.tgz message # create a tarball containing this message
rabbitio in -e rabbitio-exchange -q rabbitio-queue -f message.tgz

This will publish your first message into rabbitio-exchange and you'll see your message in the queue rabbitio-queue

Consume your first message

$ mkdir data
$ rabbitio out -e rabbitio-exchange -q rabbitio-queue -d data/
2018/03/15 15:37:35 RabbitMQ connected: amqp://guest:guest@localhost:5672/
2018/03/15 15:37:35 Bind to Exchange: "rabbitio-exchange" and Queue: "rabbitio-queue", Messaging waiting: 1
^C Interruption, saving last memory bits..
2018/03/15 15:37:45 Wrote 203 bytes to data/1_messages_1.tgz
2018/03/15 15:37:45 tarball writer closing

We interrupt when the queue is empty by directly using a combination of CTRL + C once. This will save the last bits and ack the message.

Detailed Usage

$ rabbitio
Rabbit IO will help backup and restore your messages in RabbitMQ

Usage:
  rabbitio [command]

Available Commands:
  help        Help about any command
  in          Publishes documents from tarballs into RabbitMQ exchange
  out         Consumes data out from RabbitMQ and stores to tarballs
  version     Prints the version of Rabbit IO

Flags:
  -e, --exchange string     Exchange to connect to
  -h, --help                help for rabbitio
  -p, --prefetch int        Prefetch for batches (default 100)
  -q, --queue string        Queue to connect to
  -r, --routingkey string   Routing Key, if specified will override tarball routing key configuration (default "#")
  -t, --tag string          AMQP Client Tag (default "Rabbit IO Connector ")
  -u, --uri string          AMQP URI, uri to for instance RabbitMQ (default "amqp://guest:guest@localhost:5672/")

Use "rabbitio [command] --help" for more information about a command.

AMQP Headers and Routing Key

When you read messages from a queue, the headers as well as the routing key will be saved as metadata in the tarballs, utilizing what in tar is called PAX Records. This is helpful if you one day want to replay the data back into the original queue, while keeping the attributes that belong to the message.

Header Format	AMQP Headers	Tar PAX Records
Format Translation	map[String] Bool, Integer, String, Float	map[String] String
Body Type	Bytes	Bytes

The tar metadata can be accessed using pax:

pax -r -zf 1_message_100.tgz

This will output the messages and in addition a PaxHeaders.0 directory containing identical filenames as the messages, enabling access of the metadata. Right now there is no way to selectively publish only certain messages containing a header value, this is a planned feature.

Contributing

If you plan to work on rabbitio you will need Golang. PRs are welcome as well as implementation discussions.

Clone rabbitio

mkdir -p $GOPATH/src/github.com/meltwater
cd $GOPATH/src/github.com/meltwater
git clone [email protected]:meltwater/rabbitio.git

Building

cd rabbitio
make && make build

Maintainers

For any bug reports or change requests, please create a GitHub issue or submit a PR.

Also feel free to drop a line to the maintainers:

Joel (@vorce)
Stian (@stiangrindvoll)

rabbitio's People

Contributors

Stargazers

Watchers

Forkers

tirasmuturi renso tafthorne heliocampos leonliu315 inside-track abc94e4f31 champ centja1 mendorf

rabbitio's Issues

Preserve order

Is it possible to make rabbitio preserve the order of the messages?

For some applications the order is important.

Does rabbitio preserve the message order when creating the tgz file?
(1. message from rabbitmq will be first message in tgz file)

Does rabbitio preserve the message order in the tgz file when uploading messages?
(1. message in tgz file will be first message in uploaded to rabbitmq)

If the order is preserved, the problem is "just" when extracting the messages (e.g. to filter them) as all messages seem to get a random GUID as filename, losing the original order of the files. Perhaps offer a different file naming scheme?

Filtering rules when publish/consuming messages

based on header data or content in the body.

rabbitio out panic: "send on closed channel" on SIGTERM

Panic: "send on closed channel" on SIGTERM

$ rabbitio out -b 222 -p 500 -u amqp://user:user@host:post/vhost -e exchange_name -r name -q name -d ./data                                                                                                                                  
2029/10/11 11:21:23 RabbitMQ connected: amqp://user:user@host:port/vhost
2029/10/11 11:21:23 Bind to Exchange: "exchange_name" and Queue: "name", Messaging waiting: 935577                  
2029/10/11 11:21:23 Wrote 32815 bytes to data/1_messages_222.tgz
2029/10/11 11:21:23 Wrote 32800 bytes to data/2_messages_222.tgz
....
2029/10/11 11:21:24 Wrote 32869 bytes to data/23_messages_222.tgz
2029/10/11 11:21:24 Wrote 32804 bytes to data/24_messages_222.tgz
^C Interruption, saving last memory bits..
2029/10/11 11:21:24 Wrote 32804 bytes to data/25_messages_222.tgz
panic: send on closed channel

goroutine 34 [running]:
github.com/meltwater/rabbitio/rmq.(*RabbitMQ).Consume 
        /home/user/go/src/github.com/meltwater/rabbitio/rmq/consume.go:141         
created by github.com/meltwater/rabbitio/cmd.glob..func2
        /home/user/go/src/github.com/meltwater/rabbitio/cmd/out.go:50

Additional storage support

e.g: to AWS S3.

Examples in help output

When running rio --help I do get the help instructions. To quickly get started I would suggest to add ...

a sample command to this repository's README
a sample command to the --help output

What do you think?

Add template publishing support

The general idea here is that if you want to manually publish data in a certain format, but depend on data to be inserted into a specific format, we can create a template and merge the data into the template using RabbitIO.

Template /tmp/template.trio

{
   "data": {{MYDATA}},
   "size": {{MYSIZE}}
}

example way of execution:
$ rio in --template /tmp/template.trio --template-variable [email protected] --template-variable MYDATA=5 --amqp-header myheader=myheadervalue --amqp-header myheader2=myheadervalue2

Web UI to simplify the process of consuming and publishing to different storages

Messages are being acked before they are safely saved.

Need to do multi-ack after tarballs have been saved. not as it has been fetched from rabbit

Control the speed when publishing messages

In some cases a consumer has specific needs or limits were the consumption rate would need to be constant or at least more controllable. By adding limits to the publishing speed, we will be able to support the specific needs of the consumer.

Using /dev/null as an output directory

I was just trying to use rabbitio as an easy way to bleed off messages from a queue. I used -d /dev/null as parameters for the out command, but received an error that it could not write to a file within /dev/null. Behavior I would expect is to just write to the file handle in blissful ignorance.

rabbitio out -e PXC.TRIGGER -p 250 -q PXC.TRIGGER.PXC -r \# -u 'amqp://usr:[email protected]:5672/rmq-vhl' -d /dev/null/
2022/02/02 20:00:01 RabbitMQ connected: amqp://user:[email protected]:5672/rmq-vhl
2022/02/02 20:00:01 Bind to Exchange: "PXC.TRIGGER" and Queue: "PXC.TRIGGER.PXC", Messaging waiting: 9843598
Error: open /dev/null/1_messages_1000.tgz: not a directory

Documentation Enhancement: Add a contributing md file to help other people contribute!

I noticed there is a Contributing section in the README, but it would be awesome to have a contributing md file that would help users know immediately what contribution should look like.

Thanks!

Password with special characters fail

Describe the bug

When using a URL with username:password and the password has characters as #&`,:! rabbitio fails to decode the URL. There doesn't seem to be a different way to supply the password.

Steps to reproduce

Steps to reproduce the behavior:

Try using the password foo#bar in the URL.
Run rabbitio.
Get this error: 2021/10/27 11:04:11 writer failed to connect to Rabbit: strconv.ParseInt: parsing "foo": invalid syntax

Expected behavior

Password should be accepted and no error shown.

Environment (anything that would help us investigate the bug)

Used v0.5.4 windows precompiled tool.

Could have used a dedicated password option.

Cant Build within docker 1.8

Hi,

I am trying to build RabbitIO inside of a docker container(as my RabbitMQ instance is not accessable outside of my internal docker network) with the following Dockerfile:

FROM golang:1.8

RUN go get -u github.com/meltwater/rabbitio

CMD ["bash"]

However I get the following error while doing a docker build :

PS D:\LocalDev\RabbitIO> docker build .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM golang:1.8
1.8: Pulling from library/golang
4176fe04cefe: Pull complete
851356ecf618: Pull complete
6115379c7b49: Pull complete
69914558965c: Pull complete
b108f9aa98db: Pull complete
df7abcd2981e: Pull complete
3e60cb3f592b: Pull complete
Digest: sha256:f0b5dab7581eddb49dabd1d1b9aa505ca3edcdf79a66395b5bfa4f3c036b49ef
Status: Downloaded newer image for golang:1.8
 ---> 0d283eb41a92
Step 2/3 : RUN go get -u github.com/meltwater/rabbitio
 ---> Running in c56f5bdf77b2
# github.com/pborman/uuid
src/github.com/pborman/uuid/time.go:15: syntax error: unexpected = in type declaration

Not sure if this is an error here, or an error i should be raising with pborman?

Thanks.

Documentation enhancement: what typical use case does this solve?

What is the typical use case for this? Is its purpose to help replay rabbimq conversations? It would be useful to see this kind of thing put forth in the documentation.

Be able to override/add headers to publishing messages

This change would also require to support more than just string headers
NOTE, currently tarballs headers only support storing map[string]string, while the amqp.Table uses mag[string]interface{}. Which is why we currently only support string headers,
We'll need to be specific in how to handle the transition.

Imported messages sent to error queue

RabbitMQ version: 3.7.9
RabbitIO version: 0.5.4

When using this tool to export messages, it is working perfectly fine. Messages are sent to tarballs as expected. However, when restoring those same messages, it would seem that the restore is successful since they get added back to their queue. As soon as a consumer is able to process them, they immediately get sent to our error queue.

Any ideas on how to troubleshoot this issue?

RabbitIO not load messages in rabbitmq 3.8.x

Im use this command:

rabbitio in -e news -u "amqp://adm:[email protected]:5672/" -q news -f news/1_messages_1.tgz -p 1

RabbitIO send my message:

2020/01/24 13:26:47 RabbitMQ connected: amqp://adm:[email protected]:5672/
2020/01/24 13:26:47 Extracted 1 Messages from tarball: news/1_messages_1.tgz
2020/01/24 13:26:47 Total 1 Messages from tarballs
2020/01/24 13:26:47 RabbitMQ Connection closed with success

rabbitmq queue "news" empty(

github like a pro - stale branches

Hi team,

You have some branches in your repository that look like they are stale (older than 21 days).
Mind taking a look if you can clean up to keep the repo tidy?

(potentially) stale branches:

fileTests - last commit was more than 73 days ago by @stiangrindvoll
interfaceSend - last commit was more than 70 days ago by @stiangrindvoll

If you have any questions about this, or would like to request a feature or report a bug on github-like-a-pro, please report them via https://github.com/meltwater/github-like-a-pro/issues

Cheers
github-like-a-pro