Elasticsearch Search Query using python subprocess: Illegal Argument Exception

Question

I am trying to run a curl command using the run command in the subprocess module in python. But it is failing. However, the curl command for my elasticsearch search query API is working fine-

curl -k XGET -H "Content-Type: application/json" -H "Authorization: ApiKey Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw==" 'https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/_search?index=.fleet-enrollment-api-keys-7&size=20&pretty' -d '{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}},"fields" : ["_id", "_index"]}'

This is the code that I am trying to run-

import json
from builtins import int
from pydantic import BaseModel, Field
from typing import List, Dict
from subprocess import PIPE, run

def elasticsearch_search_query(host: str, 
                               port: int, 
                               api_key: str, 
                               path: str, 
                               query: dict,
                               index: str,
                               size: int,
                               fields: List):

    es_path = host + ":" + str(port) + path +"?index="+index+"&"+str(size)+"&pretty"
    es_header = "Authorization: ApiKey" + " " + api_key
    es_dict = {"query": query, "fields": fields}
    es_json = json.dumps(es_dict)
    es_replaced = " "+"'"+ es_json.replace("'", "\"")+"'"

    result = run(["curl", "-k", "-XGET", "-H", "Content-Type: application/json", "-H",
                  es_header,
                  es_path,
                  "-d",
                  es_replaced]
                 # ,stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=False
                )
    print(es_path, es_header, es_replaced)
    print(result.stdout)


elasticsearch_search_query(host="https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/_search", api_key="Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw==",
                           port=9200, size=5, query={"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, fields=["_id", "_index"], path="/_search",index=".fleet-enrollment-api-keys-7")

Here is the error-

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "request [/_search:9200/_search] contains unrecognized parameter: [5]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "request [/_search:9200/_search] contains unrecognized parameter: [5]"
  },
  "status" : 400
}
https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/_search:9200/_search?index=.fleet-enrollment-api-keys-7&5&pretty Authorization: ApiKey Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw== '{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, "fields": ["_id", "_index"]}'
None

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   350  100   350    0     0  10294      0 --:--:-- --:--:-- --:--:-- 10294
curl: (3) nested brace in URL position 12:
'{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, "fields": ["_id", "_index"]}'

UPDATE

Output of repr(result):

CompletedProcess(args=['curl', '-k', '-XGET', '-H', 'Content-Type: application/json', '-H', 'Authorization: ApiKey Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw==', 'https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200:9200/_search?index=.fleet-enrollment-api-keys-7&size=5&pretty', '-d', '{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, "fields": ["_id", "_index"]}'], returncode=3)

es_replaced is inherently broken. You're adding single quotes -- that's shell quotes -- to your data; but when you run a shell command the shell removes those quotes before it hands anything off to curl; they are not, and should not be, part of your data. — Charles Duffy
– Charles Duffy, Commented Sep 8, 2022 at 15:18
So when there's no shell, you shouldn't be adding shell quotes. Take the 's out, and leave the double quotes exactly the way json.dumps() made them. — Charles Duffy
– Charles Duffy, Commented Sep 8, 2022 at 15:19
@CharlesDuffy I removed the es_replaced variable and I am using es_json as it is. After which I am getting this error- curl: (3) URL using bad/illegal format or missing URL — Shloka Bhalgat
– Shloka Bhalgat, Commented Sep 8, 2022 at 16:18
Please log the repr() of the array you're feeding to run(). — Charles Duffy
– Charles Duffy, Commented Sep 8, 2022 at 16:27
@CharlesDuffy, Done, please check the UPDATE part in the question. Also, as an update, I checked the URL I had specified the port number twice. Thank you so much! It got resolved. — Shloka Bhalgat
– Shloka Bhalgat, Commented Sep 8, 2022 at 16:48

Val · Accepted Answer · 2022-09-09 03:50:04Z

1

Since the error was

contains unrecognized parameter: [5]

The problem is not in the query but in the way you construct es_path. Instead of

&"+str(size)+

it should be

&size="+str(size)+

UPDATE:

Also from what I see your URL now looks like this

https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200:9200/_search?index=.fleet-enrollment-api-keys-7&size=5&pretty

and it's wrong for the following reasons:

the port is specified twice
the index is not at the right location

First, host should just be this (i.e. without port and path)

host="https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com"

Then you can build the URL like this instead:

es_path = host + ":" + str(port) + "/" + index + path +"?size="+str(size)+"&pretty"

i.e. it should look like this instead:

https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/.fleet-enrollment-api-keys-7/_search?size=5&pretty

edited Sep 9, 2022 at 3:50

answered Sep 8, 2022 at 15:11

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Charles Duffy Over a year ago

The query is also wrong, insofar as the OP is shell-quoting data that should be passed literally.

Val Over a year ago

The first error shown is about the query string having an unknown parameter. But you're probably right that maybe once the query string is corrected, the next issue will be the query itself as hinted by the curl error

Shloka Bhalgat Over a year ago

Thanks, the issue was resolved. But now I am getting this error- curl: (3) URL using bad/illegal format or missing URL

Shloka Bhalgat Over a year ago

I also removed the single quotes in es_replaced

Val Over a year ago

I think es_path must also be wrapped within quotes because of the & sign

|

Collectives™ on Stack Overflow

Elasticsearch Search Query using python subprocess: Illegal Argument Exception

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related