Datacoral adds change data capture for MySQL sources to its slice catalog
I’m excited to announce that the Datacoral platform now supports change data capture for MySQL, which we call “MySQL CDC”. Change data capture (CDC) from operational systems is instrumental in modernizing your ability to capture incremental data inserts, updates and deletes. Without CDC, you are focused on the current snapshot of business data and not on how it has changed and evolved over time.
We all know that data is bringing in a massive wave of change and has become a driver for competitive business strategy. Becoming a data-driven business requires you to have a scalable modern data architecture as well as the ability to move data into that architecture and catalog, organize (transform) and harness its utility by publishing it anywhere you need. Datacoral’s MySQL CDC Collect Slice allows you to efficiently move such data from MySQL into a data warehouse like Redshift without losing how it has evolved. This enables you to perform analytics with data in the warehouse instead of putting load on your production MySQL server.
Features of our MySQL Change Data Capture slice
Datacoral’s MySQL CDC slice has no impact on the performance of your MySQL system because it reads the Row-Based-Replication log of MySQL. CDC can be implemented for various tasks such as auditing, copying data to another system or processing events.
MySQL data changes can be captured hourly. Once data is captured by any Datacoral Connectors, it is passed into S3 for persistence and then loaded into your data warehouse which makes it available for further data transformations, analysis and use (or publishing) downstream.
MySQL CDC is also fault tolerant and also allows you to recover from failures such that we can pick up from when the failure happened. Our paging mechanism allows quick recovery for each failed page which results in not having to re-read the whole binary log when a failure is encountered. Datacoral’s TimeLabel ClockTM ensures that we can reprocess the exact set of changes which were lost in order to ensure accuracy of data in the warehouse.
Beyond that, since a MySQL binary log contains change records from hundreds of tables, we use a complex mix of streams and fan-out processing which allows us to upload multiple tables into the warehouse in parallel. This gives you the ability to capture and store multiple schemas with hundreds of tables and millions of records immediately after you install MySQL CDC. And if that’s not enough, we also support the ability to apply CDC across multiple MySQL shards.
If you are looking to perform near-real time analytics on data stored in MySQL, checkout the Datacoral platform and our MySQL CDC slice documentation. We can help you get setup in an afternoon, like we did for our newest customer, this week.
