Syncing your Database To OpenSearch In Real-Time _Gunnar Morling

Publicado el: 31 mayo 2024
en el canal de: Java Day Istanbul
1,018
15

Title: Syncing your Database To OpenSearch In Real-Time

Description: You've been tasked with implementing a data streaming pipeline for propagating data changes from your operational Postgres database to a search index in OpenSearch. Data views in OpenSearch should be denormalized for fast querying, and of course there should be no noticeable impact on the production database.

In this session we'll discuss how to build this data pipeline using two popular open-source projects: Debezium for log-based change data capture (CDC) and Apache Flink for stream processing.

Setting up change data streams with Debezium
Efficiently building nested data structures from 1:n joins
Deployment options: Kafka Connect vs. Flink CDC

We'll also touch on some advanced aspects like observability and consistency checks for your realtime data pipeline.

Bio: Gunnar Morling is a software engineer and open-source enthusiast by heart, currently working at Decodable on stream processing based on Apache Flink. In his prior role as a software engineer at Red Hat, he led the Debezium project, a distributed platform for change data capture.

He is a Java Champion and has founded multiple open source projects such as JfrUnit, kcctl, and MapStruct. Gunnar is an avid blogger (morling.dev) and has spoken at various conferences like QCon, Java One, and Devoxx. He lives in Hamburg, Germany.


En esta página del sitio puede ver el video en línea Syncing your Database To OpenSearch In Real-Time _Gunnar Morling de Duración hora minuto segunda en buena calidad , que subió el usuario Java Day Istanbul 31 mayo 2024, comparta el enlace con amigos y conocidos, en youtube este video ya ha sido visto 1,018 veces y le gustó 15 a los espectadores. Disfruta viendo!