628 questions
1
vote
1
answer
198
views
IIDR CDC Kafka Custom KCOP disable schema auto register
we are trying to replicate Db2 changes from z/OS to Confluent Kafka.
Everything works out fine for the production of simple JSON records in Kafka.
As our policies for Kafka Topics and Schemas are ...
- 76
0
votes
2
answers
101
views
How to capture delta records in Snowflake when a table is dropped and re-created every day?
I have a pipeline that drops and re-creates several Snowflake tables every day (effectively a full refresh using CREATE OR REPLACE TABLE). I want to capture daily deltas (inserts/updates/deletes) for ...
- 1
0
votes
0
answers
43
views
about EMQX Enterprise 6 Action Sink to Kafka cluster
I am a newbie in EMQX architecture. Recently, for test purposes, I deployed an EMQX enterprise 6 single node, with Kafka producer connector integration (Action Sink).
The compatition matrix of the ...
0
votes
1
answer
58
views
Flink CDC + Hudi isn't working as expect, found log said state is cleared
I'm using Flink CDC + Apache Hudi in Flink to sync data from MySQl to AWS S3. My Flink job looks like:
parallelism = 1
env = StreamExecutionEnvironment.get_execution_environment(config)
...
- 834
0
votes
1
answer
136
views
Why am I getting a null row when querying a CDC function in an ADF Lookup activity?
I am trying to build an ADF pipeline that incrementally updates a table using Change Data Capture (CDC). And I'm running into an issue when I try to use a CDC built in table function in an ADF Lookup ...
- 1
0
votes
0
answers
37
views
Debezium initial snapshot on demand
I have a debezium 3.0 in my python project. Postgres-debezium connector has the following structure:
{
"name": "dbz_name",
"config": {
"connector.class&...
2
votes
1
answer
136
views
Debezium + Flink Oracle CDC - "db history topic or its content is fully or partially missing" for some tables
I am using Flink with Debezium to consume CDC changes from Oracle DB tables via LogMiner.
For some tables, everything works fine. For example, the following table works without issues:
CREATE TABLE ...
- 23
0
votes
0
answers
48
views
Perform double duplication in Delta Live tables
I have a weird request. I have 2 sets of keys, one being pk and unique indices. I am trying to do 2 rounds of deduplication. 1 using pk to remove cdc duplicates and other to merge. DLT is not allowing ...
- 41
0
votes
2
answers
87
views
how have CDC on redis?
I'm using CDC services like (Debezium) on my Mongo or Postgres but somehow I came up with situation that I need have to CDC on Redis . for example get streams of event occur in Redis like adding new ...
3
votes
1
answer
422
views
How to perform real-time Postgres to Redis sync?
I want to sync a Postgres table (2 million records) to Redis.
Requirements:
Full table should sync initially
After that, realtime insert/update/delete should sync to Redis automatically (no polling)
...
- 81
0
votes
0
answers
53
views
MongoDB custom CDC handler using PySpark
I want to replicate a collection and sync in real time.
The CDC events are streamed to Kafka and I’ll be listening to it and based on operationType (insert/delete/update) I’ll have to process the ...
- 197
0
votes
0
answers
58
views
how to add NEW tables to the mysql source kafka debezium connector
I am using AWS MSK cluster and I have created a MySQL debezium connector that runs on EC2 instance and reads from a specific table, and it is working fine, but it could be the use case where the user ...
- 645
2
votes
0
answers
25
views
In windows, I can't see usb cdc descriptions
I'm developing a new project by using stm32 MCU and USB is one of most important features in my project. But unfortunately I always see manufacturer as "Microsoft" and comport name as "...
0
votes
0
answers
43
views
No CDC data in Cassandra binary logs (cdc_raw)
I encountered an issue while using CDC in Cassandra and need your help to resolve it.
I have set up a Change Data Capture (CDC) system to track changes in a specific table. CDC is enabled, and I can ...
- 1
0
votes
0
answers
100
views
Error 300 VIEW SERVER STATE permission when trying to disable CDC as the SA
Having an issue with the CDC process. I'm trying to add a column to a CDC table and the first step is to disable the current CDC table. I can't get past this step even though I'm logged in as 'sa'. ...
- 83
0
votes
0
answers
52
views
CDC ECM Library for ESP32 Not available anymore for ESP IDF V5.4
I need to work on my Code that basiccally does TLS with my ESP32 running in C Language. All I want to do is to USE USBCDC ECM Library. But for ESP IDF 5.4V they have not support USB CDC ECM. Can you ...
2
votes
1
answer
673
views
Read incremental data from iceberg tables using Spark SQL
I am trying to read incremental data between two snapshots
I have last processed snapshot (my day0 load) and below is my code snippet to read incremental data
incremental_df = spark.read.format("...
- 33
0
votes
1
answer
98
views
Flink CDC Task Frequently Failing - has no more allocated slots for job
I'm running a Flink CDC job (version 3.0.0) that synchronizes data from MySQL 8.0.35 to Apache Doris, but it keeps failing with the error Could not allocate the required slot within timeout. Has no ...
0
votes
1
answer
50
views
Triggers execution order inside transaction for change tracking
I have a Firebird table Orders with bunch of columns. I have after update triggers that log table information in events_history table in the following manner:
SET TERM ^ ;
CREATE OR ALTER TRIGGER ...
0
votes
0
answers
89
views
pyFlink job - Could not find any factory for identifier 'mysql-cdc' when creating a Flink-cdc table
I'm running a Flink cluster in Docker in my local env, and I've copied these jar files to the /opt/flink/lib/ of the image:
flink-cdc-dist-3.3.0.jar
flink-cdc-pipeline-connector-mysql-3.3.0.jar
flink-...
- 3
0
votes
0
answers
42
views
2 Stage Synchronizer confusion
In this picture, Ds goes metastable during the second rising edge of CLK-B because Din changed values during its aperture time at the first rising edge of CLK-B. However, Ds can go metastable during ...
0
votes
2
answers
339
views
Postgres CDC failure and WAL filling up disk space?
Suppose a CDC data loading tool (Fivetran, Airbyte, Debezium, etc) gets into a failure state and is not successfully running its incremental syncs of a Postgres data source. Also suppose that ...
0
votes
1
answer
48
views
Event Ordering Across Tables in Kafka
In Kafka, it's generally recommended to map one table to one topic. However, what if the order of events across multiple tables is important for a specific key, such as CompanyId?
For example, ...
- 108
1
vote
1
answer
771
views
Debezium connector for Postgres doest not work for Partitioned tables
I set up Postgres->kafka CDC using Debezium connector. Debezium CDC is able publish all changes to Kafka topic public.some_topic for regular postgres tables.
But it is not publishing changes to ...
- 29
0
votes
0
answers
56
views
How to Implement Maxwell CDC with AWS MSK?
I am trying to implement Change Data Capture (CDC) using Maxwell's Daemon with AWS Managed Streaming for Apache Kafka (MSK). My goal is to capture changes from a RDS MySQL database and stream them to ...
0
votes
1
answer
159
views
Java heap space error in Airbyte SQL Server source connector
My team migrated to Airbyte at the start of 2024 and mid-year, we started using the CDC capability of SQL Server.
However, one of the jobs has started failing again on a java heap space error. The CDC ...
- 63
0
votes
1
answer
122
views
Unable to delete a CDC resource in ADF
I had created a CDC resource in ADF to test out, how it works. After few tests, It became non-functional with a red cross mark against it. I cannot edit or view the activity anymore.
If I try to ...
0
votes
0
answers
108
views
Aws glue, db connection and data load based on timestamp
CDC -- Pipeline1 work is to load data (have list of tables) based on timestamp columns (creation_date, updation_date) from replica db (RDS) to S3 (landing_zone)
If I created rds connection on glue, ...
- 33
1
vote
2
answers
236
views
AWS DMS: Same S3 bucket as target for multiple SQL sources
I cannot find information about how to achieve what's in the title of this question.
Let's say I have different SQL databases from different departments in the org and I want to migrate all of them to ...
- 597
1
vote
0
answers
105
views
Flink SQL CDC connector do nothing to the setting "debizum.*" when create the source table
I have following table in the souce database:
CREATE TABLE `client_test` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(500) NOT NULL DEFAULT '',
`age` int(11) NOT NULL,
...
- 11
1
vote
0
answers
210
views
Kafka Debezium Oracle Source Connector Intermittent Lag & Data Delay
I am working with Kafka, and we are using Debezium to capture CDC (Change Data Capture) data from an Oracle source database. Below are my Debezium configuration parameters:
{
"name": &...
- 155
1
vote
0
answers
96
views
How to create debezium ElasticsearchSinkConnector connecting to AWS ES intance with IAM role
We were trying to connect to ES through debezium ElasticsearchSinkConnector but getting 401 Unauthorized
We were able to connect to ES instance with IAM role with no master cred and authentication. ...
- 145
0
votes
1
answer
209
views
USB CDC freezes (Host to Device transfers)
My setup
I try to use USB FS Host on GD32F405 (=STM32) microcontroller. I use CMSIS and operate directly on registers.
I have Android device, that is working as VCP (Virtual COM port) by default and ...
- 202
-1
votes
1
answer
155
views
Issue with Debezium Snapshot: DateTimeParseException in PostgreSQL Connector
I'm encountering an issue with Debezium version 2.3.1 while processing a snapshot from a PostgreSQL database. The error occurs during the snapshot phase and seems to be related to parsing a date/time ...
0
votes
1
answer
161
views
Amazon S3 as Source for Apache Flink using CDC
I need to publish the event using CDC whenever a file is added to S3, to Apache Flink, from Flink the newly added file has to be read and processed.
- 505
1
vote
0
answers
661
views
How to reduce CPU impact of sp_cdc_scan procedure in SQL Server when CDC is enabled on multiple databases?
I have enabled CDC on a few databases in my SQL Server instance. I noticed that enabling CDC on a database creates the sp_cdc_scan request, which runs continuously and consumes CPU time. This has ...
- 547
1
vote
1
answer
348
views
SQL Server CDC: cdc.fn_cdc_get_all_changes_ returns a row containing all null values - LSN = 0x00000000000000000000
I'm running the following query on a SQL Server capture instance:
DECLARE @from_lsn binary (10), @to_lsn binary (10)
SELECT @from_lsn = sys.fn_cdc_get_min_lsn ( 'dbo_ABC' );
SELECT @from_lsn;
...
- 5,472
0
votes
1
answer
569
views
debezium standalone server with http sink
we are running Debezium standalone server to capture the changes from postgresql database, and when a change is capture, we want it to send to a http endpoint. It works perfectly fine, however I can't ...
- 577
-1
votes
1
answer
605
views
Issues with executing PySpark / Delta MERGE statement due to special character in a table name
I am working on implementing CDC (Change Data Capture) which will use Azure Sql as the source and Azure Databricks delta lake as the target.
Azew Sql creates CDC tables with a very specific column ...
- 15.7k
0
votes
1
answer
162
views
How to implement CDC from Greenplum?
We need catch data changes from Greenplum(catch insert/update/delete from GP database)
Do you know any methods of software to help catch data from Greenplum?
- 166
0
votes
0
answers
167
views
DMS Source Data and Latency spike nightly
I'm using a SQL Server on-prem source for a DMS to replicate to a SQL Server RDS. Every night, at about midnight, I'm seeing a spike of ~1.6M CDC incoming changes on my DMS task, but nothing is making ...
0
votes
1
answer
459
views
Error on Airbyte using cdc incremental load with mariadb as source and duckdb as destination
I am using CDC incremental load from mariadb database to the duckdb as a destination. Since there is no any connector for mariadb, I have used mysql connector. There is not any issue during set up ...
0
votes
1
answer
717
views
CDC capture job not running after running sys.scan_job manually
I have a CDC enabled table used for replication. One of my stored procedure I have, runs this exec sys.sp_cdc_scan manually for a specific case (my stored procedure is invoked by a DDL trigger which I ...
- 312
0
votes
0
answers
277
views
Debezium - value.converter serializing problem
hello I use debezium outbox router for postgres
this is my json in payload column
{
"userId": 107385,
"chatId": "beb8faec-b75f-4eca-ace0-57b8621c7ca0",
"fromEcho&...
- 215
0
votes
1
answer
161
views
How to stop clean up scheduler in Azure SQL DB CDC
I have requirement where i need to perform Audit of data for that i have planned to use Azure DB change data capture(CDC).
In Azure CDC it works on scheduler unlike on premise where SQL Server Agent ...
0
votes
0
answers
82
views
How to read master->detail rows?
I have 2 tables T1(master), T2(detail). CDC works on these both
tables.
I'd like to restore history tables to th new 2 tables T11->T21 using stored procedure.
The question is: how to process the ...
- 6,592
2
votes
0
answers
334
views
Debezium automatic topic creation not working as expected
I'm working with CDC using a debezium connector managed by Confluent.
I was trying to use the topics creation group feature offered by debezium, to create topics automatically based on the size of ...
- 794
1
vote
1
answer
1k
views
Debezium signal table at connector startup
I have a doubt regarding Ad hoc snapshot signals in debezium
Let say that I have created the signal table in my database before creating my CDC connector:
-- Creating the signal table
CREATE TABLE ...
- 794
0
votes
2
answers
450
views
I do not find the cdc.fn_cdc_get_net_changes_*
I am developing SQL Server business logic using CDC.
The CDC works fine, I see the changes in cdc tables but I cannot find function cdc.fn_cdc_get_net_changes*. But other functions exist: cdc....
- 6,592
0
votes
1
answer
1k
views
Getting error in AWS DMS Source Test endpoint connection
Test Endpoint failed: Application-Status: 1020912, Application-Message: Failed to connect Network error has occurred, Application-Detailed-Message: RetCode: SQL_ERROR SqlState: 08001 NativeError: 101 ...
- 143