Syncing your Database To OpenSearch In Real-Time _Gunnar Morling

Pubblicato il: 31 maggio 2024
sul canale di: Java Day Istanbul
1,018
15

Title: Syncing your Database To OpenSearch In Real-Time

Description: You've been tasked with implementing a data streaming pipeline for propagating data changes from your operational Postgres database to a search index in OpenSearch. Data views in OpenSearch should be denormalized for fast querying, and of course there should be no noticeable impact on the production database.

In this session we'll discuss how to build this data pipeline using two popular open-source projects: Debezium for log-based change data capture (CDC) and Apache Flink for stream processing.

Setting up change data streams with Debezium
Efficiently building nested data structures from 1:n joins
Deployment options: Kafka Connect vs. Flink CDC

We'll also touch on some advanced aspects like observability and consistency checks for your realtime data pipeline.

Bio: Gunnar Morling is a software engineer and open-source enthusiast by heart, currently working at Decodable on stream processing based on Apache Flink. In his prior role as a software engineer at Red Hat, he led the Debezium project, a distributed platform for change data capture.

He is a Java Champion and has founded multiple open source projects such as JfrUnit, kcctl, and MapStruct. Gunnar is an avid blogger (morling.dev) and has spoken at various conferences like QCon, Java One, and Devoxx. He lives in Hamburg, Germany.


In questa pagina del sito puoi guardare il video online Syncing your Database To OpenSearch In Real-Time _Gunnar Morling della durata di ore minuti seconda in buona qualità , che l'utente ha caricato Java Day Istanbul 31 maggio 2024, condividi il link con amici e conoscenti, su youtube questo video è già stato visto 1,018 volte e gli è piaciuto 15 spettatori. Buona visione!