MySQL 5.6.13 : The incident LOST_EVENTS occured on the master. Message: error writing to the binary log

Oct 2, 2013

I received this error on my MySQL 5.6.13 slave
[ERROR] Slave SQL: The incident LOST_EVENTS occured on the master. Message: error writing to the binary log, Error_code: 1590

I issued a slave skip counter:

stop slave;
SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;
start slave;

Which somewhat fixed the issue, but then I kept running into Duplicate entries for key errors like this:
[Warning] Slave SQL: Could not execute Query event. Detailed error: Duplicate entry '19905692' for key 'PRIMARY';, Error_code: 1062

Since they were duplicate, I kept trying to skip them, but there was no end in sight. I thought this was odd and wondered if somehow MySQL had reset the slave. I don't know what position it stopped in, so I added a line to the MySQL configuration file, my.cnf, to skip the error code 1062, which skips Duplicate entries:
slave-skip-errors = 1062 #Skip duplicate entry

I have Percona tools sync script running daily, which will fix any inconsistencies.

This is how the error went down in the MySQL error log:

2013-10-01 15:30:47 29607 [ERROR] Slave SQL: The incident LOST_EVENTS occured on the master. Message: error writing to the binary log, Error_code: 1590
2013-10-01 15:30:47 29607 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'db-bin.000201' position 843441357

I came across a couple of articles after I went through all this which explains a bit more why this error occurs, except it doesn't explain my duplicate key errors:
Granting privileges may break replication in MySQL 5.6.10
Critical MySQL 5.6 bug: GRANTs and replication

I was issuing some GRANTs at the time replication stopped, and I'm sure I issued at least 1 of them incorrectly, so explains why it broke.

Comments

Avi Vainshtein -- Dec 12, 2013 10:37 AM
I've found the reason of the problem in my case. It's a bug as described here : http://bugs.mysql.com/bug.php?id=68892 Shortly: grant or revoke statement with syntax error cause replication fail (even revoke for non-existing grants). Listed as resolved in 5.6.15. regards, Avi
Reply
Avi Vainshtein -- Dec 12, 2013 8:51 AM
I've also met that behavior twice on 5.6.12 slaves. However, i didn't find the root cause of that. So, each time i just rebuild the slave. Did you find the reason of that ? And what is the long-time solution ? Best regards, Avi
Reply
New Comment