No events received when using `mysql-binlog-connector-java` with scala about mysql-binlog-connector-java HOT 22 CLOSED

Hello,

I have created a sample project which works for me (please grab it from here). Here is what I did:

1. executed "ssh -L 9797:localhost:3306 my-username@mysql-server-hostname"
2. "cd mysql-binlog-connector-java-using-scala" (sample project mentioned earlier)
3. "./mvnw clean test" (you can use mvn instead of ./mvnw if you already have Maven)

Please try it yourself. You should get something similar to:

INFO: Connected to 127.0.0.1:9797 at mysql-bin.000002/107
connected
New event
New event

Let me know if you have any issues.

P.S. I've used different username and password (src/test/scala/junit.scala) so you might want to change them back before running the test.

from mysql-binlog-connector-java.

Hello,

Thank you for your help!

I tried your project and it doesn't work with my server, so I guess the problem is not with mysql-binlog-connector-java.

I also don't get any of the lifecycle callbacks, the program just blocks after outputing this:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running BinaryLogClientTest

So I guess there is some problem with the sql server. Do you have any idea on how I could debug this?

thanks!

from mysql-binlog-connector-java.

Hi again,

Could you please try mysql -h127.0.0.1 --port=9797 --protocol=TCP -uuser -ppass? Is it working? Also, which OS do you use?

I think you don't get lifecycle callbacks called because BinaryLogClient gets stuck at the very beginning (while trying to connect to the socket). You may want to replace client.connect with client.connect(3000) (3s). At least this way it won't 'hang'.

from mysql-binlog-connector-java.

Hello,

Yes I can connect using mysql -h 127.0.0.1 --port 9797 --protocol=tCP --user user -p

What I can't do is connect directly using the mysqlbinlog binary:

/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;                 
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
ERROR: Got error reading packet from server: Slave can not handle replication events with the checksum that master is configured to log; the first event 'mysql-bin.000701' at 4, the last event read from '/data/mysql/mysql-bin.000701' at 120, the last byte read from '/data/mysql/mysql-bin.000701' at 120.
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

I think this is because my mysql-client version is more recent than the servers version, although I am not sure this is the problem.

But I thought mysql-binlog-connector-java used it's own Java library to connect to mysql, so this should not be a problem?

I am using ubuntu 13.10.

Thanks!

from mysql-binlog-connector-java.

Hi,

I should have asked this before, but what is the version of the MySQL server you trying to connect to?

from mysql-binlog-connector-java.

I am using 5.6.14.

from mysql-binlog-connector-java.

I need about 10 minutes to verify that tests pass against 5.6.14 release. In a meantime, do you think you could debug into BinaryLogClient.connect and check how far it goes (line number would be great)?

from mysql-binlog-connector-java.

Yep, tests passed. So this is not MySQL server version related.

from mysql-binlog-connector-java.

If you were willing to install vagrant you could:

git clone https://github.com/shyiko/mysql-binlog-connector-java.git
cd mysql-binlog-connector-java/
mvn clean verify \
    -Dvagrant.integration.box=supplement/vagrant/mysql-5.6.12-sandbox-prepackaged

This way one can verify that mysql-binlog-connector-java works against sandboxed environment.

from mysql-binlog-connector-java.

Hi,

Thank you for your help!

I took a thread dump from ./mvnw clean test and it looks like it's stuck getting the filename and position:

at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:170)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:45)
at com.github.shyiko.mysql.binlog.network.protocol.PacketChannel.read(PacketChannel.java:55)
at com.github.shyiko.mysql.binlog.BinaryLogClient.readResultSet(BinaryLogClient.java:486)
at com.github.shyiko.mysql.binlog.BinaryLogClient.fetchBinlogFilenameAndPosition(BinaryLogClient.java:395)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:250)

Here's the full dump: https://gist.github.com/simao/8280188

I do have vagrant installed and I tried to run that command but the vm provisioning fails because it doesn't seem to have internet access, it can't download the required packages. I'll try to look into this to make sure it's not something weird with my OS, but looks like the problem is with the mysql-server?

Thanks

from mysql-binlog-connector-java.

Well, whatever the cause is - mysql-binlog-connector-java shouldn't get stuck without an explanation. Also, thank you for the stack trace. BinaryLogClient passed greeting & authentication and got stuck on binlog filename/position resolution.
Could you please execute show master status and tell me what the output is?

from mysql-binlog-connector-java.

here it is:

+------------------+-----------+--------------+------------------+-------------------+
| File             | Position  | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+-----------+--------------+------------------+-------------------+
| mysql-bin.000716 | 252731600 |              |                  |                   |
+------------------+-----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

from mysql-binlog-connector-java.

Looks OK. I hoped that user just don't has the necessary permissions, but it seems it has. Let's take another way. I've just committed support for custom SocketFactory. Could you please try enabling/disabling Nagle algorithm (as shown below)?

client.setSocketFactory(new SocketFactory() {

    override def createSocket = {
        val socket = new Socket()
        socket.setTcpNoDelay(true)
        socket
    }
})

from mysql-binlog-connector-java.

Actually, the user I use for mysqlbinlog doesnt have permissions for show master status! It has for enough permissions to read from the binlog but not for show master status.

from mysql-binlog-connector-java.

I see. Could you please grant REPLICATION CLIENT permission to whatever user you use for BinaryLogClient ("SHOW MASTER STATUS" isn't necessary only when binlogFilename & binlogPosition are specified)?

from mysql-binlog-connector-java.

According to this you could also try

client.setBinlogFilename("")
client.setBinlogPosition(4)

but I haven't tried it myself (note that this is different from the "last binlog filename/position" resolution).

from mysql-binlog-connector-java.

Yep, setting the file name and position explicitly works fine. I was not aware of those options, I should have tried that :(

I will add the REPLICATION CLIENT grant and chef provision the servers and try this with proper auth.

Thank you so much for your help.

from mysql-binlog-connector-java.

No problem. I'm sorry you had to go through all of this :)

from mysql-binlog-connector-java.

Hi!

I provisioned the servers and now the user has REPLICATION CLIENT and REPLICATION slave, but this still occurs if I don't explicitly set the filename and position. Do you know what the problem may be?

Thanks!

from mysql-binlog-connector-java.

Hello,

That's weird. I've created a new user (msandbox_slave) and granted him REPLICATION SLAVE and REPLICATION CLIENT privileges (as shown below). Integration tests passed (and failed once I revoke REPLICATION CLIENT, which is expected).

CREATE USER 'msandbox_slave' IDENTIFIED BY 'msandbox';
GRANT REPLICATION SLAVE ON *.* TO 'msandbox_slave';
GRANT REPLICATION CLIENT ON *.* TO 'msandbox_slave';

I would suggest to (in that order):

1. make sure mysql -hsame_host_as_for_binarylogclient -usame_user_as_for_binarylogclient -ppassword --port=9797 --protocol=TCP -e 'show master status' returns data without any errors
2. try disabling Nagle algorithm as I wrote before

from mysql-binlog-connector-java.

OK! My bad, it works now! Permissions were the culprit then :)

Thanks again for your help!

from mysql-binlog-connector-java.

Hello,

date format not converting to elasticsearch date format, getting exception java.util.date cannot cast to string, unsafe domain object conversion.
Please help me out regarding this issue.

from mysql-binlog-connector-java.