Title: Syncing your Database To OpenSearch In Real-Time
Description: You've been tasked with implementing a data streaming pipeline for propagating data changes from your operational Postgres database to a search index in OpenSearch. Data views in OpenSearch should be denormalized for fast querying, and of course there should be no noticeable impact on the production database.
In this session we'll discuss how to build this data pipeline using two popular open-source projects: Debezium for log-based change data capture (CDC) and Apache Flink for stream processing.
Setting up change data streams with Debezium
Efficiently building nested data structures from 1:n joins
Deployment options: Kafka Connect vs. Flink CDC
We'll also touch on some advanced aspects like observability and consistency checks for your realtime data pipeline.
Bio: Gunnar Morling is a software engineer and open-source enthusiast by heart, currently working at Decodable on stream processing based on Apache Flink. In his prior role as a software engineer at Red Hat, he led the Debezium project, a distributed platform for change data capture.
He is a Java Champion and has founded multiple open source projects such as JfrUnit, kcctl, and MapStruct. Gunnar is an avid blogger (morling.dev) and has spoken at various conferences like QCon, Java One, and Devoxx. He lives in Hamburg, Germany.
Sur cette page du site, vous pouvez voir la vidéo en ligne Syncing your Database To OpenSearch In Real-Time _Gunnar Morling durée heure minute seconde en bonne qualité , qui a été Téléchargé par l'utilisateur Java Day Istanbul 31 mai 2024, Partagez le lien avec vos amis et connaissances, sur youtube cette vidéo a déjà été regardée 1,018 fois et il a aimé 15 téléspectateurs. Bon visionnage!