Separate indices or use type field in Elasticsearch

Question

I'm developing an Elasticsearch service and we have multiple sources like our support ticket portal and a forum. Currently, I'm segregating each source into it's own index as each will have a child type. The ticket portal will of course search tickets (with nested replies) but also search users and such so there are multiple types under the portal index. Simple stuff so far.

However, I'm starting to think of merging the indices and prefix the type (portalTicket, portalUser, forumThread, forumUser, etc) as I'm wanting to search across both sources, but maybe there is a way to query them and bring it all back together. I'm just working with tickets and threads at the moment to start small, here are the two simples mappings I'm using thus far:

{
    ticket : {
        properties : {
            replies : {
                type : 'nested'
            }
        }
    }
}

{
    thread : {
        properties : {
            posts : {
                type : 'nested'
            }
        }
    }
}

Wanted to show that to show I'm using nested objects with different names. I can of course have same names but there will also be other meta data attached to the ticket and thread mappings that will be nested types also and that takes me to the issue. When I search without specifying the index, I get issues with some not having the nested type, as expected. The thread mapping doesn't have a replies property, it's posts. I can get around it using index in a filter like so:

{
    filter : {
        indices : {
            index          : 'portal',
            no_match_query : 'none',
            query          : {
                bool : {
                    should : [
                        {
                            match : {
                                title : 'help'
                            }
                        },
                        {
                            nested : {
                                path  : 'replies',
                                query : {
                                    match : {
                                        'replies.text' : 'help'
                                    }
                                }
                            }
                        }
                    ]
                }
            }
        }
    }
}

Ok, that works for the portal index but working it to include the forum index is making me feel like I'm just fighting elasticsearch and not using it properly.

So should I keep them on separate indices and get a filter that will return both indices results or should I merge them into a single index, use a field to hold the source and likely normalize the nested properties or is there a way to work with multiple indices in a faceted way? (I know, aggregates in ES 2)

See here for the "one index vs many" tradeoff: stackoverflow.com/questions/14465668/… As for types, you can easily filter by type in queries by adjusting your REST endpoint(s) and query structure. — ryanlutgen
– ryanlutgen, Commented Aug 30, 2016 at 18:11
Thank you @ryanlutgen, from my take away there is I'll need to stick with multiple indices as I do have a several millions documents for each source. Can you say whether I'm on the right path to searching nested objects and how to properly search across the different indices like I am above? — Mitchell Simoens
– Mitchell Simoens, Commented Aug 30, 2016 at 18:24

Community · Accepted Answer · 2017-05-23 11:51:41Z

0

Reading these two posts (thanks to the commenters for pointing these out):

I have decided that my data is too different and the amount of documents that I anticipate (and future additions) means that I should go with different indices.

Now to learn how to search across the different indices but this post was more about which strategy I should use so I'm going to open a new question for that.

edited May 23, 2017 at 11:51

CommunityBot

11 silver badge

answered Aug 31, 2016 at 12:50

Mitchell Simoens

2,5462 gold badges20 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Separate indices or use type field in Elasticsearch

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related