Async connection to AWS elasticsearch

Question

I am struggling to connect to AWS elasticsearch with async connection.

Regular sync connection works fine:

from requests_aws4auth import AWS4Auth
from elasticsearch import Elasticsearch, RequestsHttpConnection

AWS_ACCESS_KEY_ID = "<id>"
AWS_SECRET_ACCESS_KEY = "<key>"
AWS_REGION = "us-east-1"
awsauth = AWS4Auth(AWS_ACCESS_KEY_ID,
                   AWS_SECRET_ACCESS_KEY,
                   AWS_REGION, 'es')


es = Elasticsearch(
        ['https://url-to-elastic-in-aws/'],
        http_auth=awsauth,
        use_ssl=True,
        verify_certs=True,
        connection_class=RequestsHttpConnection
    )

print(await es.info())

But async connection with connection_class=RequestsHttpConnection or connection_class=RequestsHttpConnection throw an exception

from requests_aws4auth import AWS4Auth
from elasticsearch import AsyncElasticsearch, AIOHttpConnection

AWS_ACCESS_KEY_ID = "<id>" 
AWS_SECRET_ACCESS_KEY = "<key>" 
AWS_REGION = "us-east-1" 
awsauth = AWS4Auth(AWS_ACCESS_KEY_ID,
                   AWS_SECRET_ACCESS_KEY,
                   AWS_REGION, 'es')


es = AsyncElasticsearch(
        ['https://url-to-elastic-in-aws/'],
        http_auth=awsauth,
        use_ssl=True,
        verify_certs=True,
        connection_class=AIOHttpConnection
    )

print(await es.info())

It is either: TypeError: object tuple can't be used in 'await' expression it connection_class=RequestsHttpConnection

or

AttributeError: 'AWS4Auth' object has no attribute 'encode'

I search the Internet and could not find a solution for this. Is there any way to connect to AWS elasticsearch with AsyncElasticsearch class.

Stefan · Accepted Answer · 2021-12-22 14:45:55Z

1

I believe this is a known limitation with RequestsHttpConnection, and has been since last year: https://github.com/elastic/elasticsearch-py/issues/1333#issuecomment-676278567

The recommendation is to not use the async elasticsearch/opensearch client until RequestsHttpsConnection is updated.

edited Dec 22, 2021 at 14:45

Stefan

1,96222 silver badges38 bronze badges

answered Dec 22, 2021 at 2:11

Jim

111 silver badge1 bronze badge

Sign up to request clarification or add additional context in comments.

Comments

Perye LI · Accepted Answer · 2024-06-06 05:46:25Z

0

This answer solves my problem https://stackoverflow.com/a/77082892. You can use AWSV4SignerAsyncAuth rather than AWSV4SignerAuth for auth and AsyncHttpConnection as the connection class in async scenarios.

edited Jun 6, 2024 at 5:46

answered Jun 5, 2024 at 3:28

Perye LI

11 bronze badge

1 Comment

Vitalizzare Over a year ago

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review

Collectives™ on Stack Overflow

Async connection to AWS elasticsearch

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related