When the binlog file is larger than 4G, data loss occurs about gh-ost HOT 2 CLOSED

Thank you for your detailed report and the depth of understanding you've shown regarding gh-ost and MySQL functionalities.

We have carefully reviewed the implications of accepting binlog events that exceed the standard max_binlog_size and max_allowed_packet settings, as documented in the MySQL Reference Manual (max_binlog_size, max_allowed_packet). After this review, we have decided against implementing the requested change, primarily due to the way gh-ost interacts with the primary instance.

gh-ost functions by mimicking a replica to the primary database. This design means that in scenarios where a transaction exceeds 1GB, the likelihood of gh-ost successfully replicating this data without encountering errors is significantly reduced. MySQL's replication mechanism, which gh-ost relies upon, is not designed to handle individual transactions of this size, as they exceed the max_binlog_size and max_allowed_packet limit. For information on replication and max_allowed_packet, see this dev doc.

Our commitment with gh-ost is to provide a tool that not only enhances MySQL's capabilities but also aligns closely with its core principles and standards. Deviating from these established limits, even for edge cases, could lead to unpredictable challenges and potentially encourage practices that are outside MySQL’s recommended configurations.

In light of your report, we are considering ways to harden this check to move it from a soft failure to a hard failure. However, at this time, we cannot provide a specific timeline for this enhancement. We believe this approach will further safeguard the tool against unintended consequences and align it more closely with MySQL's standards.

We understand this might pose challenges in specific scenarios, especially with very large transactions. We recommend exploring alternate strategies, such as breaking down these transactions into smaller, more manageable sizes that align with MySQL's configuration guidelines.

Thank you again for your contribution and understanding.

from gh-ost.

Hi, Since the data table has a big row, assuming a single row is 4M and the chunk is 1000, the transaction generated by one copy is larger than 4G. There is nothing we can do to prevent this from happening. Losing data is a serious problem.

Thanks for the evaluation.

from gh-ost.