How to send scroll_id to ElasticSearch with Curl

Question

I can't figure out how I should send scroll_id to ElasticSearch using Curl.

This is what I have tried so far, but it doesn't seem to work.

$url = "http://distribution.virk.dk/cvr-permanent/virksomhed/_search?scroll=2m&_scroll_id=".$_POST["scroll_id"];
        $data = array(
            "_scroll_id" => $_POST["scroll_id"],
            "scroll_id" => $_POST["scroll_id"],
            "size" => 10, 
            "_source" => array(
                "Vrvirksomhed.cvrNummer",
                "Vrvirksomhed.elektroniskPost",
                "Vrvirksomhed.livsforloeb",
                "Vrvirksomhed.hjemmeside",
                "Vrvirksomhed.virksomhedMetadata.nyesteNavn.navn",
                "Vrvirksomhed.hovedbranche",
                "Vrvirksomhed.penheder",
                "Vrvirksomhed.telefonnummer",
                "Vrvirksomhed.virksomhedMetadata.nyesteBeliggenhedsadresse"
            ), 
            "query" => array (
                "bool" => array (
                    "must_not" => array (
                        "exists" => array (
                            "field" => "Vrvirksomhed.livsforloeb.periode.gyldigTil"
                        )
                    )
                )
            )
        );

The ElasticSearch returns the same 10 posts every time, so I think it doesn't get the scroll_id right.

Updated code after trying Val's suggestion. Using the setHosts I get a timeout after a long time. Leaving out the setHosts, I get the error saying that No alive nodes found in your cluster.

use Elasticsearch\ClientBuilder;

    require 'vendor/autoload.php';


    $username = "MY_USERNAME";
    $password = "MY_PASSWORD";
    
    $hosts = [
        'host' => 'distribution.virk.dk',
        'scheme' => 'http',
        'path'  => '/cvr-permanent',
        'port' => '80',
        'user' => $username,
        'pass' => $password
    ];

    $client = ClientBuilder::create()->setHosts($hosts)->build();
    $params = [
    'scroll' => '30s',
    'size'   => 50,
    'type'   => '/cvr-permanent/virksomhed',
    'index'  => 'virksomhed',
    'body'   => [
        'query' => [
            'match_all' => new \stdClass()
        ]
    ]
    ];

    // Execute the search
    // The response will contain the first batch of documents
    // and a scroll_id
    $response = $client->search($params);

    // Now we loop until the scroll "cursors" are exhausted
    while (isset($response['hits']['hits']) && count($response['hits']['hits']) > 0) {

        // **
        // Do your work here, on the $response['hits']['hits'] array
        // **

        // When done, get the new scroll_id
        // You must always refresh your _scroll_id!  It can change sometimes
        $scroll_id = $response['_scroll_id'];

        // Execute a Scroll request and repeat
        $response = $client->scroll([
            'body' => [
                'scroll_id' => $scroll_id,  //...using our previously obtained _scroll_id
                'scroll'    => '30s'        // and the same timeout window
            ]
        ]);
    }

Val · Accepted Answer · 2020-09-02 08:52:16Z

1

There are two steps for using the scroll API.

In the first step you need to send the query and the duration of the scroll context.

In the second step, you don't need to send the query again, but only the scroll id you got from the previous scroll search.

You can find a full-blown example here

answered Sep 2, 2020 at 8:52

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Jeppe Donslund Over a year ago

I have tried to follow the examples in your link. First I get "No alive nodes found in your cluster" and then I thought that I needed to fill in my username and password somewhere, så I added setHosts to the ClientBuilder line, but then I get a timeout error after a long time.

Val Over a year ago

Maybe update your question with the new code you're trying

Val Over a year ago

According to your first code snippet, I think you need to use port 80 and not 9200 as in your second snippet

Jeppe Donslund Over a year ago

Using port 80 doesn't give much change. POST /cvr-permanent/virksomhed/_search?search_type=scan&scroll=1m { "_source":[ "Vrvirksomhed.cvrNummer", "Vrvirksomhed.virksomhedMetadata.nyesteNavn.navn", "Vrvirksomhed.virksomhedMetadata.nyesteBeliggenhedsadresse" ], "query":{"match_all":{}}, "size":200 } Does that give any hint about what I should try?

Val Over a year ago

I was talking about this: 'port' => '9200', which should be 'port' => '80',

|

Collectives™ on Stack Overflow

How to send scroll_id to ElasticSearch with Curl

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related