Return specific fields for elasticsearch in rails

Question

This seems like a really easy issue, but everything I've tried from other solutions and websites is not working. I have three fields I do not want indexed or queried--:p_s, :gender, and :part_of_speech--but elasticsearch is still returning values from those fields even though I don't specify that they should be indexed or queried. About halfway down, this article says to say no to indexing, but they don't indicate where this would occur.

Term Controller:

  def search
    @terms = Term.search(params[:query]).page(params[:page])
  end

Model:

require 'elasticsearch/model'

class Term < ActiveRecord::Base

include Elasticsearch::Model
include Elasticsearch::Model::Callbacks

  settings index: { number_of_shards: 1, number_of_replicas: 0 },
    do

    mappings dynamic: 'false' do
      indexes :id, index: :not_analyzed
      indexes :name, analyzer: :spanish_analyzer
      indexes :definition, analyzer: :combined_analyzer
      indexes :etymology1, analyzer: :combined_analyzer
      indexes :etymology2, analyzer: :combined_analyzer
      indexes :uses, analyzer: :combined_analyzer
      indexes :notes1, analyzer: :combined_analyzer
      indexes :notes2, analyzer: :combined_analyzer
    end
  end

  def self.search(query)
    __elasticsearch__.search(
      {
        query: {
          multi_match: {
            query: query,
            fields: ['name^7', 'definition^6', 'etymology1^5', 'etymology2^4', 'uses^3', 'notes1^2', 'notes2^1'],
            operator: 'and'
          }
        }
      }
    )
  end
end

# Delete the previous term index in Elasticsearch
Term.__elasticsearch__.client.indices.delete index: Term.index_name rescue nil

# Create the new index with the new mapping
Term.__elasticsearch__.client.indices.create \
  index: Term.index_name,
  body: { settings: Term.settings.to_hash, mappings: Term.mappings.to_hash }

# Index all term records from the DB to Elasticsearch
Term.import(force: true)

Nic Nilov · Accepted Answer · 2016-06-29 10:23:36Z

1

To mark a field as non-indexed use this:

mappings dynamic: 'false' do
    ...
    indexes :p_s, index: :no
    indexes :gender, index: :no
    indexes :part_of_speech, index: :no
    ...
end

By default elasticsearch returns all document fields under "_source" key. To only get specific fields you can either specify fields array on the top query level like this

def self.search(query)
    __elasticsearch__.search(
      {
        query: {
          multi_match: {
            query: query,
            fields: ['name^7', 'definition^6', 'etymology1^5', 'etymology2^4', 'uses^3', 'notes1^2', 'notes2^1'],
            operator: 'and'
          }
        },
        fields: ['name', 'definition', 'etymology1', 'etymology2', 'uses', 'notes1', 'notes2']
      }
    )
  end

or filter "_source"

def self.search(query)
    __elasticsearch__.search(
      {
        query: {
          multi_match: {
            query: query,
            fields: ['name^7', 'definition^6', 'etymology1^5', 'etymology2^4', 'uses^3', 'notes1^2', 'notes2^1'],
            operator: 'and'
          }
        },
        '_source': ['name', 'definition', 'etymology1', 'etymology2', 'uses', 'notes1', 'notes2']
      }
    )
end

See Elasticsearch source filtering docs for more.

When using multi_match clause, the inner fields element specifies the fields to run the search on and, optionally, the boost like in your example. The outer fields or '_source' clause in turn determines which fields to return and this is the one you're after.

To have a better visibility into what's going on while debugging elasticsearch queries, use a tool like Sense. When you get the result you want it may be much easier to transfer the query to ruby code than vice versa.

edited Jun 29, 2016 at 10:23

answered Jun 24, 2016 at 19:10

Nic Nilov

5,1652 gold badges24 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

evanejin Over a year ago

Okay, I tried specifying fields in the top level query, but I receive this error undefined method slug' for #<Elasticsearch::Model::Response::Result` and it points to this is my search view template <a href="/term/<%=term.slug %>"><%= term.name %></a> I deleted the link and performed the query again but received the same error for term.name

evanejin Over a year ago

I tried the _source method as well, but I think I'm putting it in the wrong location/not typing it correctly. I've read tons of documentation and try hard to understand, but as a newbie, I find a lot of this goes over my head. For instance, how am I supposed to apply this line "term" : { "user" : "kimchy" } to my code? I don't really get what I'm supposed to replace it with.

Nic Nilov Over a year ago

Please see the updated example on "_source" usage.

evanejin Over a year ago

I tried both '_source' and _source for good measure, but now it's returning the same error that I'm having with the other person helping me. I even added Term.__elasticsearch__.refresh_index!(force: true) at the end of my model per his suggestion, but that didn't help.

Nic Nilov Over a year ago

What was this error? Note that template errors related to <a href="/term/<%=term.slug %>"><%= term.name %></a> may have nothing to do with the query working correctly. This is because depending on your query configuration the resulting dataset structure may be different. This is why it is important to check the raw outcome of the query before trying to use it down the line.

|

oreoluwa · Accepted Answer · 2016-06-24 18:58:47Z

0

I think using the included elasticsearch methods makes a lot of sense. However, in my own case, in my model I did something like this, modified for your own case:

def as_indexed_json
  as_json(only: [:id, :name, :definition, :etymology1, :etymology2, :uses, :notes1, :notes2])
end

This should work because by default Elasticsearch would call the as_indexed_json method in your model to get the data it needs to index.

answered Jun 24, 2016 at 18:58

oreoluwa

5,6532 gold badges22 silver badges28 bronze badges

5 Comments

evanejin Over a year ago

Sorry for the delay, I had a new issue come up that I needed to solve first...Your solution is one of the methods I tried that didn't work for me. To be clear, this code should take the place of my def self.search(query) method, no? It also should go after my mappings, right?

oreoluwa Over a year ago

With this method in place you wouldn't need to do the mappings anymore, because those fields would not be indexed anymore and therefore cannot be queried at all. Your search can still remain the way it is and I guess all other configurations can be done in your settings.

evanejin Over a year ago

hmm, well I tried just as you said, but it's still returning those extra fields that I don't want included

oreoluwa Over a year ago

Did you reindex the table?

evanejin Over a year ago

I placed Term.__elasticsearch__.refresh_index!(force: true) after Term.import(force: true) at the bottom of my model, but that didn't change anything. I even closed down elasticsearch and fired it back up as well as dopped the db and migrated and seeded it again to no avail

Collectives™ on Stack Overflow

Return specific fields for elasticsearch in rails

2 Answers 2

7 Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related