Bulk collection inside the right Index path in ElasticSearch using NEST in a .NET Core application

Question

I am trying to bulk a collection of elements inside an index of ElasticSearch using NEST inside a .NET Core application.

Currently what I have is working, and the elements are saved, but Is not saved where I try to do

My client creation:

protected ElasticClient GetClient()
{
    var node = new Uri("http://localhost:9200/");
    var settings = new ConnectionSettings(node)
        .DefaultIndex("TestIndex")
        .PrettyJson(true);

    return new ElasticClient(settings);
}

Here is how I create the descriptor for bulk all the data

protected BulkDescriptor GenerateBulkDescriptor<T>(IEnumerable<T> elements, string indexName) where T: class, IIndexable
{
    var bulkIndexer = new BulkDescriptor();

    foreach (var element in elements)
        bulkIndexer.Index<T>(i => i
          .Document(element)
          .Id(element.Id)
          .Index(indexName));

    return bulkIndexer;
}

Finally, once I have this, here is how I index the data

var descriptor = GenerateBulkDescriptor(indexedElements, "indexed_elements");

var response = GetClient().Bulk(descriptor);

But, If I see how It's stored in the Elastic index using this, that is what I have:

How can I know if is created under TestIndex index? Because as far as I can see, there is just one index created

Thank you a lot in advance

Russ Cam · Accepted Answer · 2019-02-25 23:37:00Z

1

When defining the index operations on the BulkDescriptor, you are explicitly setting the index to use for each operation

foreach (var element in elements)
    bulkIndexer.Index<T>(i => i
      .Document(element)
      .Id(element.Id)
      .Index(indexName));

where indexName is "indexed_elements". This is why all documents are indexed into this index and you do not see any in "TestIndex".

The Bulk API allows multiple operations to be defined, which may include indexing documents into different indices. When the index is specified directly on an operation, that will be the index used. If all index operations on a Bulk API call are to take place against the same index, you can omit the index on each operation and instead, specify the index to use on the Bulk API call directly

var defaultIndex = "default_index";
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));

var settings = new ConnectionSettings(pool)
    .DefaultIndex(defaultIndex);

var client = new ElasticClient(settings);

var people = new [] 
{
    new Person { Id = 1, Name = "Paul" },
    new Person { Id = 2, Name = "John" },
    new Person { Id = 3, Name = "George" },
    new Person { Id = 4, Name = "Ringo" },
};

var bulkResponse = client.Bulk(b => b
    .Index("people")
    .IndexMany(people)
);

which sends the following request

POST http://localhost:9200/people/_bulk
{"index":{"_id":"1","_type":"person"}}
{"id":1,"name":"Paul"}
{"index":{"_id":"2","_type":"person"}}
{"id":2,"name":"John"}
{"index":{"_id":"3","_type":"person"}}
{"id":3,"name":"George"}
{"index":{"_id":"4","_type":"person"}}
{"id":4,"name":"Ringo"}

Note that the URI is /people/bulk and that each JSON object representing an operation does not contain an "_index".

If you omit the .Index() on Bulk API call, it will use the DefaultIndex configured on ConnectionSettings:

var bulkResponse = client.Bulk(b => b
    .IndexMany(people)
);

which yields

POST http://localhost:9200/_bulk
{"index":{"_id":"1","_index":"default_index","_type":"person"}}
{"id":1,"name":"Paul"}
{"index":{"_id":"2","_index":"default_index","_type":"person"}}
{"id":2,"name":"John"}
{"index":{"_id":"3","_index":"default_index","_type":"person"}}
{"id":3,"name":"George"}
{"index":{"_id":"4","_index":"default_index","_type":"person"}}
{"id":4,"name":"Ringo"}

You can also specify a default index to use for a given POCO type on ConnectionSettings with DefaultMappingFor<T>(), where T is your POCO type.

answered Feb 25, 2019 at 23:37

Russ Cam

126k34 gold badges211 silver badges275 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

xaberue Over a year ago

Hello Russ, thanks for your help, the first approach that you have explained me is giving me the same result, however the second one is not indexing anything. I don't know why. Then, in addition, If I try to perform a search with the client is looking for a different index name (Tha one that I want /TestIndex/indexedelement/) Sorry, I know that I am doing too many questions... I don't know If I am not understanding or If I am doing something wrong in my code. Thank you again

xaberue Over a year ago

Hello again, after several tests and attemps, I have found a solution for what I want to implement... I wanted to thank you again for your help, it was a very clear answer and it was helpful at the end in order to find a solution

xaberue · Accepted Answer · 2019-02-27 15:10:07Z

0

After som tests and attemps, I have found a solution.

First of all, it was a problem with the index configured, once I set it in lower case, the index was working fine indexing data inside.

Then, I had the problem of index data in a specific "path" inside the same index, finalyy I found the Type solution from NEST, taking also advantage of the DefaultMappingFor suggested by Russ in the previous answer.

Client definition:

var node = new Uri(_elasticSearchConfiguration.Node);
var settings = new ConnectionSettings(node)
    .DefaultMappingFor<IndexedElement>(m => m
        .IndexName(_elasticSearchConfiguration.Index)
        .TypeName(nameof(IndexedElement).ToLower()))
    .PrettyJson(true)
    .DisableDirectStreaming();

var client = new ElasticClient(settings);

Then, the BulkDescriptior creation:

var bulkIndexer = new BulkDescriptor();

foreach (var element in elements)
    bulkIndexer.Index<IndexedElement>(i => i
        .Document(element)
        .Type(nameof(IndexedElement).ToLower()))
        .Id(element.Id)
    );

And finally, data bulk:

client.Bulk(bulkIndexer);

Now, If I perform a call to the index, I can see this

{
"testindex": {
    "aliases": {},
    "mappings": {
        "indexedelement": {

[...]

}

Thank you Russ for your help and for who have had a look to the post.

UPDATE

Finally, it seems that the unique problem was regarding the default index, that it must be lowercase, so, specify the type with the name of the POCO itself is not neccesary, like @RussCam has truly detected in comments above. After changing thedefault index to lowercase, all the different possibilities worked fine.

Thank you all again

edited Feb 27, 2019 at 15:10

answered Feb 26, 2019 at 23:52

xaberue

3465 silver badges23 bronze badges

2 Comments

Russ Cam Over a year ago

.Type(nameof(IndexedElement).ToLower())) is not needed as this is what NEST does by default, so you don't need it in DefaultMappingFor<IndexedElement> or in each in bulk call.

xaberue Over a year ago

Hello @RussCam, I have tested and you are right, so at the end I guess that it was just a problem with the index that it should was in lowercase. After that, my previous implementations and also your pruposals works fine. I will edit the post to add your info, thank you a lot

Collectives™ on Stack Overflow

Bulk collection inside the right Index path in ElasticSearch using NEST in a .NET Core application

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related