kafka connector HTTP/API source

Question

I am actually aware on how to capture data from any data source, such as a specific API (e.g HTTP GET request) and ingest them in specific kafka connector.

{
    "name": "localfileSource", 
    "config": {
        "connector.class": "FileStreamSourceConnector",
        "tasks.max": "1",
        "file": "test.txt",
        "topic": "connectSource"
    }
}

I would need something similar to this(FileStreamSourceConnector), that can be used with API sources.

Cástor Rodríguez · Accepted Answer · 2020-06-09 07:11:41Z

5

It's probably a bit late for the question asker, but I had a similar need and I decided to implement my own connector.

The particularity of the connector is that it implements some sort of Change Data Capture

In case it can help anyone else, you can find it here: castorm/kafka-connect-http

answered Jun 9, 2020 at 7:11

Cástor Rodríguez

511 silver badge3 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Pegerto · Accepted Answer · 2018-04-13 09:51:13Z

1

You can always build your own connectors, to stream from http source in a set of intervals is relatively simple.

I am aware that you may have solved your problems, a naive implementation can be found: https://github.com/pegerto/kafka-connect-http

Regards.

answered Apr 13, 2018 at 9:51

Pegerto

6241 gold badge5 silver badges19 bronze badges

Collectives™ on Stack Overflow

kafka connector HTTP/API source

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related