Connecting to AWS Elasticsearch from non-AWS node.js app

Question

I'm working on puzzling out an infrastructure-ish issue with a project I'm working on. The service that I'm developing is hosted on a transient, containerized platform w/o a stable IP — only a domain name (api.example.com). I'm utilizing Elasticsearch for search, so requests go to something like /my-search-resource and then use ES to find results to return. It's written in node and uses the supported elasticsearch driver to connect to ES.

The issue I'm having is in trying to use an AWS Elasticsearch domain. This project is bootstrapped, so I'm taking advantage of the free-tier from AWS, even though the other services are hosted/deployed on another platform (think: heroku, GCP, etc. — containerized and transient resources).

Since I can't just whitelist a particular IP, I'm not sure what I should do to enable the service to have access to the service. I do need to sign every request sent to the domain? This isn't ideal, since it would require monkey-patching the ES driver library with that functionality. Ideally, I'd like to just use username & pw to connect to the domain, but I know IAM isn't really oriented for something like that from an external service. Any ideas? Is this even something possible?

Hello, facing the same issue right now. Did you find a way?

Jürgen Simon
– Jürgen Simon

2016-10-26 09:04:02 +00:00
Commented Oct 26, 2016 at 9:04 — Jürgen Simon
– Jürgen Simon, Commented Oct 26, 2016 at 9:04
Yes I did! I'll put the solution up later today :)

markthethomas
– markthethomas

2016-10-26 14:15:15 +00:00
Commented Oct 26, 2016 at 14:15 — markthethomas
– markthethomas, Commented Oct 26, 2016 at 14:15

score 12 · Accepted Answer · 2019-05-08 15:32:51Z

12

In my current project we connect to AWS Elastic by using the normal elasticsearch NPM package, and then use http-aws-es to create a specific AWS connection header when connecting.

So for example we have something like this:

const es = require( 'elasticsearch' );
const httpAwsEs = require( 'http-aws-es' );

const esClient = es.Client( { 
  hosts: 'somehostonaws',
  connectionClass: httpAwsEs,
  awsConfig: {
    region: 'some-aws-region',
    accessKey: 'some-aws-access-key',
    secretKey: 'some-aws-secret-key'
  }
} );

That wouldn't require the whole AWS SDK, but it would allow you to connect to Elastic's that are behind the AWS. Is that a solution to your issue?

edited May 8, 2019 at 15:32

answered Oct 26, 2016 at 10:19

user771071

Sign up to request clarification or add additional context in comments.

5 Comments

user771071 Over a year ago

No problem, shame I didn't see this question earlier then :)

Shushu Over a year ago

Does not work for me. With Node v6.9.5, elasticseach 13.2.0 and http-aws-es v2.0.4 I always get an error saying: "(node:18475) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Data must be a string or a buffer". I tried to debug it and see where it comes from, but wasted too much time on it.

user771071 Over a year ago

That seems like an unrelated issue though. Make sure your promises are handled and do a stack trace to find out where it happens.

Dhaval Chaudhary Over a year ago

here all the fields from amazonES should be in awsConfig.

user771071 Over a year ago

Thanks @DhavalChaudhary, I've updated the solution.

Jürgen Simon · Accepted Answer · 2016-10-26 09:47:41Z

This is not a solution to the problem, but a few thoughts on how to approach it. We're in the same pickle at the moment: we wish to use AWS but we do not want to tie in with AWS SDK. As far as I understand it, AWS offers 3 options:

Open to public (not advisable)
Fixed IP addresses (whitelist)
AWS authentication

Option 1 is not an option.

Option 2 presents us with the problem that we have to teach whatever we use to log there to go through a proxy so that the requests appear to come from the same IP address. Our setup is on Heroku and we use QuotaGuard for similar problems. However: i checked the modules I was going to use to interact (we're trying to log there, either to logstash or elasticsearch directly using winston transports) and they offer no support for proxy. Perhaps this is different in your case.

Option 3 is also not supported in any way by winston transports at this time. Which would leave us to use aws-sdk modules and tie in with AWS forever or write our own.

Collectives™ on Stack Overflow

Connecting to AWS Elasticsearch from non-AWS node.js app

2 Answers 2

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related