440 questions
1
vote
1
answer
96
views
Use AWS DMS to replicate data to pre-existing OpenSearch index
I'm planning to use AWS DMS to replicate data from AWS RDS (PostgreSQL) to OpenSearch.
I want to create an index in OpenSearch before replication to adjust primary/replica shards count. And then ...
0
votes
1
answer
80
views
Node.js - AWS X-Ray and Elasticsearch Client, are these compatible?
Is it possible to instrument a node.js application with AWS X-ray SDK simply by using the client wrapper without having to use segments and subsegments? The app is sort of legacy and it is using @...
1
vote
1
answer
550
views
How do I specify an AWS Elastic Search Ingestion Pipeline in a SAM template as a YAML string?
I have a sam template with an OpenSearch Ingestion Pipeline resource defined below. According to the documentation PipelineConfigurationBody is meant to be a string of YAML. But this string could be ...
0
votes
0
answers
1k
views
How to retrieve http.max_content_length from cluster settings in Elasticsearch on AWS
My task is to retrieve the max_content_length from cluster settings to control the size of the bulk request I am sending to AWS Elasticsearch.
I learned that I can do this by using the GET _cluster/...
0
votes
0
answers
101
views
How can I make object properties with certain depth always be visible?
I'm working on a project with Nest.js where I have to make a query object like this with my dynamic filter builder. I encountered a problem where I have to make an object with some depth to send it as ...
-1
votes
3
answers
66
views
Store and Search Stock Symbols AWS Architecture - Elasticsearch or DynamoDB
I have an external API to synchronize all the Stock Symbols from all markets (NASDAQ, NYSE...).
I have an AWS Lambda to get this data from the external API.
Then I want a frontend website with a ...
1
vote
0
answers
598
views
Getting invalid host while reindexing in aws opensearch
I've two domains of Elastic search v 7.7, hosted under same VPC. I'm trying to reindex data between two domain by following steps mentioned in aws doc reindex. But getting below error
"error&...
1
vote
1
answer
3k
views
Issue connecting Springboot 2.7.8 with AWS Elasticsearch : ElasticsearchException[Invalid or missing build flavor [oss]]
I've upgraded my Springboot version to 2.7.8, spring-data-elasticsearch version to 4.4.7 and org.elasticsearch --- elasticsearch to 7.17.3
Here are the dependencies in my pom.xml
<parent>
&...
1
vote
1
answer
83
views
Elastic Search AND and OR condition is not returning correct data
I have data from two sources of document one which have status purchased and another does not have status purchased. In order to identify the status purchased i am using a nested object called ...
1
vote
2
answers
950
views
SocketTimeoutException while searching documents on AWS ElaticSearch
Context
My team has a REST service running on AWS Elastic Container Service. It is connected with AWS Elastic Search to save the data.
The following is the Elastic Search client setup:
YAML Config:
...
1
vote
1
answer
171
views
Deleting old indices on ElasticSearch Kibana?
I did my research and I am aware you can use ILM on proper Kibana for that. However, ES Kibana doesn't have that option, and I'm not sure how to write a policy for it. Google and GPT didn't help. Any ...
1
vote
1
answer
450
views
Boto3 for fetching the cluster health of OpenSearch domain
I am trying to fetch the cluster health of Opensearch domain through the boto3, but I do not find any method for the same. How to solve this problem?
1
vote
1
answer
2k
views
How to convert Elasticsearch index field of string type to json?
I have an Opensearch index with a string field message defined as below:
{"name":"message.message","type":"string","esTypes":["text"],"...
0
votes
1
answer
201
views
If I upgrade from ElasticSearch 6.8 to 7.4 we can not read the data
(Please bear in mind that I am new to ElasticSearch and I am not a web programmer)
So we have a site that uses ElasticSearch (in AWS) to look up names for our site.
It is on version 6.8 and works fine....
1
vote
0
answers
3k
views
How can we improve our AWS OpenSearch performance?
We are using AWS OpenSearch to ingest and merge data and see that the process is quite slow. Wondering if someone here can help:
Essentially we have 850 GB of data, 400 MM rows with indexing on a ...
0
votes
1
answer
800
views
Permissions required to restore automated AWS elasticsearch snapshot
I have an AWS Opensearch/ElasticSearch domain with a cluster configuration to take hourly snapshots. I'm trying to automate the restoration of the snapshot when the cluster goes down.
At the moment I'...
2
votes
0
answers
4k
views
aws-sdk node equivalent of boto3 credentials python calls
I have inherited a python lambda which I need to convert to node js. I am a bit stuck with the code that handles authentication and so I am wondering what the JS equivalent would be for the python ...
0
votes
1
answer
194
views
AWS Elasticsearch cluster upgarde from 6.3 to 7
Presently AWS Elasticsearch cluster version is 6.3 and I am planning to upgrade it to 7. reindexing is also have to be done. reindexing is required
to have _doc as type for the indices instead of our ...
2
votes
1
answer
2k
views
OpenSearch: how to define parent/child relationship between documents?
Due to AWS migration I have to migrate from ElasticSearch to OpenSearch. The OpenSearch will be provisioned by AWS OpenSearch Service.
The existing documents in ES index has parent/child relationship ...
2
votes
0
answers
747
views
change logo OpenSearch dashboards - kibana
Need to change OpenSearch logo in the dashboard.
Found i way to change if I run it local host or using docker changing the opensearch_dashboards.yml. But how could i change it if i hosted this in AWS?...
1
vote
1
answer
77
views
How to update _index value to make it indexName_{currentDate} using index template?
I am trying to create index template which will help me to update index field value from indexName to indexName{current_date}.
I found one solution using processors but it's not working for index ...
1
vote
0
answers
371
views
Connect AWS elasticsearch to Django
I have a django application which I succesfully linked to elasticsearch that is running on AWS.
So I created a GET API using elasticsearch_dsl and elasticsearch_dsl_drf and more
while this API now ...
0
votes
2
answers
752
views
Can AWS Glue connect to Elasticsearch using IAM authentication?
Followed this tutorial: https://docs.aws.amazon.com/glue/latest/ug/tutorial-elastisearch-connector.html
I know using this connector can connect to Aws Elasticsearch, but this still needs AWS Secrets ...
2
votes
1
answer
1k
views
Denied AWS Opensearch write permission
I'm trying to connect a spring boot application from AWS EKS to AWS Opensearch both of which reside in a VPC. Though the connection is successful im unable to write any data to the index.
All the AWS ...
0
votes
1
answer
36
views
How to get matching doc for term A and term B separately in AWS elastic search with single Query
Currently I'm able to do query on AWS elastic search to get matching doc's for single term. for that I'm using below query.
Now I have a requirement to do query for multiple term and get there ...
0
votes
1
answer
815
views
Elasticsearch NEST SDK throws System.OutOfMemoryException
I am getting this exception from Elasticsearch .net client - NEST v7.10.0.
Exception of type 'System.OutOfMemoryException' was thrown. Elasticsearch.Net.UnexpectedElasticsearchClientException ...
0
votes
1
answer
962
views
How to make an asynchronous search (client call) to AWS Elasticsearch (7.10) over ESDomain host
Asynchronous Search is available in AWS Elasticsearch starting with version: 7.10. According to the documentation operation /_opendistro/_asynchronous_search
is supported. However, is there a ...
0
votes
1
answer
214
views
Automating source IP of OpenSearch using Boto3/Lambda
Every time our IPs change, I have to keep updating this policy to access Kibana. I thought I could automate this, but is there any way I can delete an existing policy and create a new one on Lambda? I'...
5
votes
0
answers
1k
views
AWS OpenSearch Service connection with MongoDB
Is there any way we can connect AWS OpenSearch with MongoDB without using DocumentDb? I have an application based on Nest.js and ReactJs where I want to use OpenSearch Service
5
votes
3
answers
15k
views
How to create an index pattern in Opensearch using API?
I want to create an index pattern using Opensearch API. I tried to replicate what could be made graphically in the following image window, using as index pattern name cwl-* and then as time field @...
1
vote
2
answers
5k
views
How to calculate the actual byte size of an Array in nodejs?
I'm doing a bulk indexing in AWS OpenSearch within my node application and it's failing because the chunk size are too big. Below is the error
Request size exceeded 104857600 bytes
So I need to ...
0
votes
1
answer
477
views
Spring Native with RestHighLevelClient not connecting to AWS OpenSearch
I have a Lambda configured and running on "Custom runtime on Amazon Linux 2". GraalVM is the runtime used for our application.
When Lambda tries to make a connection with OpenSearch, it ...
7
votes
0
answers
1k
views
Is is possible to pretty print a JSON log message in ElasticSearch/Kibana
I have logs being indexed by ElasticSearch from stdout of our application server. Without having to modify the source writing the logs, is it possible to "pretty print" any JSON log message? ...
1
vote
1
answer
770
views
AWS Elasticsearch - Delete_by_query, how to find task id so can continue with code when delete is done, and when to do wait_for _completion
I'm using AWS Lambda to do a delete_by_query on an Elasticsearch index so I get rid of everything older than 7 days. That works, but I noticed that the count of the documents is the same before and ...
0
votes
1
answer
1k
views
TypeError: trying to use aws-elasticsearch-connector
I'm attempting to update a legacy elastic-search node app, using the the package aws-elasticsearch-connector
and for some reason I'm unable to get it to work at all, even the simplest provided example....
0
votes
1
answer
628
views
How to migrate to OpenSearch with CloudFormation without outage?
I found this tutorial, however this tutorial has two main drawbacks.
A brand new CloudFormation stack would be created
There would be outage.
Is there a way to upgrade ES without outage and new ...
1
vote
0
answers
115
views
Injest Pipeline Script for nested list of array - retrieve and copy data data from old field to new field in Kibana
My goals -
Find the existing field if present (responseBody.results[i].statusCode) and copy it's value into a new field (embedded_error_code ) at runtime
My original field is a list of array. Below ...
1
vote
1
answer
1k
views
Cannot See Workflows while Running Temporal on AWS ECS with AWS Elasticsearch
I deployed Temporal along with Temporal Web on AWS ECS. I replicated the configuration using this Docker Compose file: https://github.com/temporalio/docker-compose/blob/main/docker-compose.yml
I used ...
1
vote
1
answer
1k
views
OpenSearch compute knn index size properly
I am using OpenSearch service in AWS for my research.
Task: I want to compute the index size for N records in the index.
Input: I have only one node in AWS [r6g.4xlarge.search] with 128 RAM. The index ...
0
votes
1
answer
276
views
Handling Array Indexes When Searching In ElasticSearch
I have a problem with my ES queries where they fail because of having array indexes in their query strings. This happens because my following approach.
I flatten the JSON requests that I get with the ...
1
vote
0
answers
117
views
How to calculate api latency from tomcat server weblogs via elastic search Kibana?
I have been asked to help getting latency KPI put together for key API endpoints. And transformed into a format and location where we can more easily track api latency.
BUT,
I am not able to get any ...
5
votes
1
answer
2k
views
Why is Elasticsearch/Opensearch query returning everything?
I'm using AWS's OpenSearch, and I'm having trouble getting any queries or filters to only return matching results.
To test, I'm using sample ecommerce data that includes the field "...
0
votes
1
answer
788
views
How to reindex fileds in kibana when using AWS Opensearch service
I want to resolve conflicts in the kibana fields when we are using AWS elasticsearch / AWS Opensearch service.
We have 3 fields showing as conflicted in Kibana. How I can resolve that?
0
votes
0
answers
960
views
Opensearch integration tests with Jest
I am using Elasticsearch (Opensearch) in a Nx monorepo in a node js Serverless app that connects to aws services. I would like to use jest to test but so far I have not been able to get responses from ...
0
votes
0
answers
403
views
logs are not sent to the aggregator server
I have 2 ec2 instance
first instance in availability zone a, secon in in availability zone b
first instance must forward logs to second, but i have an error
2022-01-10 20:15:42 +0000 [warn]: detached ...
0
votes
1
answer
1k
views
Use AWS authentication for Amazon Open Search Service (AOSS)
We have an AOSS cluster and I'd like to use regular AWS console authentication to authenticate users. I've reviewed (and followed much of) the following:
Configuring Amazon Cognito authentication ...
0
votes
1
answer
539
views
ElasticSearch : constant_score query vs function_score query
I recently upgraded my ElasticSearch version from version 5.3 to version 5.6
"query" : {
"constant_score" : {
"query" : {
"bool" : {
...
0
votes
0
answers
763
views
AWS Elastic Search ConnectException: Operation timed out
I'm trying to access AWS open search cluster present in different AWS account(lets say ES cluster present in account A and we are trying to access it from Account B). I have performed following steps ...
0
votes
1
answer
782
views
elasticsearch reindex from remote cluster - null_pointer_exception
I have some data in index "A_1" of instance "A". I've created an index "B_1" in instance "B", with the same mappings as "A_1".
I'm trying to move data ...
1
vote
1
answer
2k
views
Unable to get OpenSearch to publish alerts to Amazon SNS
I am having great difficulties in getting OpenSearch to publish alerts to Amazon SNS. I have set up the SNS topic and permissions properly, and set up the role for the SNS destination in OpenSearch ...