how to pull data from DynamoDB and insert them into Elasticsearch

Question

I m using aws elasticsearch and i want to import data from dynamodb to elasticsearch,I don t want use river,Because Elasticsearch river API is deprecated, any Alternatives ?

I don t want to use logstash output elasticsearch, i need logstash output dynamodb but this plugin doesn t exist. — pho
– pho, Commented Dec 4, 2015 at 21:19
Ok, so you should update your question which says "from dynamodb to elasticsearch", that's why I suggested a logstash dynamodb input. — Val
– Val, Commented Dec 4, 2015 at 22:22

Ryan Fitzgerald · Accepted Answer · 2015-12-03 17:56:34Z

3

I would enable dynamodb streams on your table and then write a lambda function which reads data from the stream and inserts it into elasticsearch. Amazon has a sample lambda function which streams data from kinesis to elasticsearch. It should be very easy to modify to work with a dynamodb stream.

answered Dec 3, 2015 at 17:56

Ryan Fitzgerald

8395 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Tofig Hasanov Over a year ago

Won't this only import newly created/updated entries? How about the ones, which are already in the database?

koxon · Accepted Answer · 2016-11-17 04:16:13Z

I made it work with Python.

See a sample below which authenticate your Lambda function to ES. You can then use the ElasticSearch Python library.

from __future__ import print_function

import json
import boto3
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

def lambda_handler(event, context):

    session = boto3.session.Session()
    credentials = session.get_credentials()

    print(credentials.access_key)
    print(credentials.secret_key)
    print(credentials.token)
    print(session.region_name)

    awsauth = AWS4Auth(credentials.access_key,
                       credentials.secret_key,
                       session.region_name, 'es',
                       session_token=credentials.token)

    es = Elasticsearch(
        [ "YOUR ES ENDPOINT" ],
        http_auth=awsauth,
        use_ssl=True,
        verify_certs=True,
        connection_class=RequestsHttpConnection
    )

    print(es.info())

    [...]

You need proper role for your Lambda function. I suggest reading this blog which is not fully updated but works for the IAM roles.

https://aws.amazon.com/blogs/compute/indexing-amazon-dynamodb-content-with-amazon-elasticsearch-service-using-aws-lambda/

I which the Lambda Blueprint mentioned in the blog was still available but it is not at this time.

Hope this help

Collectives™ on Stack Overflow

how to pull data from DynamoDB and insert them into Elasticsearch

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related