How to index binary file in ElasticSearch without using Base64

Question

I'm using the NodeJS elasticsearch package to interact with ElasticSearch. I have a document that has a file field. I want to be able to upload a file to the index but the only way that I have found is by using the elasticsearch-mapper-attachment plugin.

The problem is that if I use it, I have to load the whole file in memory, encode it to Base64 and then pass the String to ElasticSearch.

I'd like to be able to pass a Stream to ElasticSearch (referencing any binary file: pdf, xls, doc, ppt).

Mmm so the only way is with a base64 string? I'm not sure about the file size. Let's say 1GB, but if 10000 users uploaded a big file, I'd have to load a lot into memory. — iamdeit
– iamdeit, Commented Sep 29, 2016 at 16:42
Do you want that attachment just stored along the index or actually indexed and searchable? — Andrey
– Andrey, Commented Sep 29, 2016 at 18:10
The files are stored in S3, I want to be able to perform searches. — iamdeit
– iamdeit, Commented Sep 29, 2016 at 20:10

Andrey · Accepted Answer · 2016-09-29 18:20:31Z

3

The elasticsearch-mapper-attachment plugin parses the uploaded binary file and extracts text for further indexing using built-in Tika extractor.

What some applications do (for example Search Technology's Aspire) - they run binaries thru Tika locally, extract text and upload just that text with the documents to index.

It might not be the answer you are looking for but you really have just two options - use Elastic plugin (and convert the binary to base64 in yoru code prior to uploading the document to elastic), or parse the binary and extract text in your code and then upload just that text to elastic. Former is easier, latter gives you more control over the process

answered Sep 29, 2016 at 18:20

Andrey

21.4k30 gold badges119 silver badges181 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

iamdeit Over a year ago

Using Tika in my application is out of scope. I found a related issue about this: github.com/elastic/elasticsearch-mapper-attachments/issues/146 Apparently, they don't want to consume the files from external data sources.

AKS Over a year ago

@Andrey, if I use Tika and extract the document content as text, then uploading that text to elastic will have any limitation? I mean in terms of if pdf file is huge then some issue will come up?

Andrey Over a year ago

@AKS - standard ES document size limit of 2GB per document will apply, so unless your pdf + all other fields are less then 2GB you are good

Collectives™ on Stack Overflow

How to index binary file in ElasticSearch without using Base64

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related