Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with date transformation using debezium #57

Open
guilhermeassad opened this issue Nov 3, 2021 · 0 comments
Open

Issue with date transformation using debezium #57

guilhermeassad opened this issue Nov 3, 2021 · 0 comments

Comments

@guilhermeassad
Copy link

guilhermeassad commented Nov 3, 2021

We use Debezium with MySQL connector to get data from a MariaDB server. A few months ago we upgraded our MariaDB to version 10.4 and after that, we started having this issue with the connectors.
Some colummns on a table are datetime(6).

Task threw an uncaught and unrecoverable exception
org.apache.kafka.connect.errors.ConnectException: Invalid value for MonthOfYear (valid values 1 - 12): -83
        at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:241)
        at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207)
        at io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onEventDeserializationFailure(BinlogReader.java:1150)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:965)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
        at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.time.DateTimeException: Invalid value for MonthOfYear (valid values 1 - 12): -83
        at java.base/java.time.temporal.ValueRange.checkValidValue(ValueRange.java:311)
        at java.base/java.time.temporal.ChronoField.checkValidValue(ChronoField.java:717)
        at java.base/java.time.LocalDate.of(LocalDate.java:269)
        at java.base/java.time.LocalDateTime.of(LocalDateTime.java:361)
        at io.debezium.connector.mysql.RowDeserializers.deserializeDatetime(RowDeserializers.java:378)
        at io.debezium.connector.mysql.RowDeserializers$WriteRowsDeserializer.deserializeDatetime(RowDeserializers.java:202)
        at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeCell(AbstractRowsEventDataDeserializer.java:181)
        at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:143)
        at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserializeRows(WriteRowsEventDataDeserializer.java:64)
        at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:56)
        at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:32)
        at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303)
        at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:232)
        at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:252)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:952)
        ... 3 more

We noticed from the binlog file that although the statement is correct the fields have a strange date format.

#Q> INSERT INTO `some_table` (`tsCreated`, `tsUpdated`, `label_id`, `customer_id`, `category_id`, `product_id`, `te_id`, `lot_id`, `page`) VALUES ('2021-11-01 10:46:25.219851', '2021-11-01 10:46:25.219878'
, 1, NULL, NULL, 139153, '017c8322f53e00059e3b6c9d48bb000780021070004a0', NULL, 'auction')
#211101 11:46:25 server id 204203  end_log_pos 170739039 CRC32 0x055ccef1       Table_map: `database`.`some_table` mapped to number 97
# at 170739039
#211101 11:46:25 server id 204203  end_log_pos 170739157 CRC32 0xc82864ea       Write_rows: table id 97 flags: STMT_END_F
### INSERT INTO `database`.`some_table`
### SET
###   @1=219190348
###   @2='85489092-40-24 33:28:01'
###   @3='280044596-30-48 38:70:73'
###   @4=1
###   @5='017c8322f53e00059e3b6c9d48bb000780021070004a0'
###   @6='auction'
###   @7=139153
###   @8=NULL
###   @9=NULL
###   @10=NULL
# Number of rows: 1
# at 170739157
#211101 11:46:25 server id 204203  end_log_pos 170739188 CRC32 0x8dec9c7a       Xid = 2376154080
COMMIT/*!*/;

This binlog is already from a replicated database, but the master shows the same behavior, and looking into the database itself, the data is all correct.
So, we believe that problem is not in MariaDB itself but in the client-side.
And the issue is pretty similar to this one: https://jira.mariadb.org/browse/MDEV-20860
We did try to upgrade java, Debezium, and our latest try was to upgrade only this binlog library to version 0.25.4, but nothing helped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant