0

He or she who answers this question will have a very special place in my heart. I'm trying to update many records at once using Curl with PHP on elasticsearch 7.13. It seems that no matter what I do, I get the error "The bulk request must be terminated by a newline". I have tried many, many renditions of this code. Is there perhaps a way to overload what PHP CURL is doing to force it to use binary data? I believe it is stripping the newline characters, but if I change to plain text submission it won't be accepted by elasticsearch.

    public function bulkUpdate(){
        $query_json_store = '';
        foreach($this->records['hits']['hits'] as $person){
        
            $entry1 = [
                "update"=>[ 
                    "_id"=> $person['_id'],
                    "type"=> "doc",
                    "_index"=> "human_data",
                ]
                ];
            
                $entry2 = [
                    "doc"=>[
                        "scanned"=> true
                    ]
                ];              


        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, 'localhost:9200/human_data/doc/_bulk');
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
        curl_setopt($ch, CURLOPT_POSTFIELDS, '\n'.json_encode($params)).'\n';
        curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
        $headers = array();
        $headers[] = 'Content-Type: application/x-ndjson';
        curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
        $result = curl_exec($ch);
        if (curl_errno($ch)) {
            echo 'Error:' . curl_error($ch);
        }
        curl_close($ch);
        \Log::info($result);
        }
    }
4
  • The bulk endpoint does not expect a proper JSON body, but rather a multi-line body where each line contains a JSON object (e.g. line 1: the command, line 2: the document, line 3: next command, line 4: next document, etc.) where each line is terminated by a newline character (which must be in double quotes in PHP, i.e. "\n" instead of '\n'). See this answer for an example. You will not get that request body from json_encode(). Commented Jun 23, 2021 at 19:52
  • @rickdenhaan great answer 👍 Commented Jun 23, 2021 at 19:55
  • Christopher it's not clear how you construct your $params but as Rick stated, you cannot use json_encode. See the answer Rick pointed at for how the body should be formatted Commented Jun 23, 2021 at 19:58
  • The problem is that PHP passes through new lines incorrectly with single quotes. I through doubles over it and it works. Commented Jun 24, 2021 at 0:48

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.