13

I have a large set off files (hdf) that I need to enable search for. For Java I would use Lucene for this, as it's a file and document indexing engine. I don't know what the python equivalent would be though.

Can anyone recommend which library I should use for indexing a large collection of files for fast search? Or is the prefered way to roll your own?

I have looked at pylucene and lupy, but both projects seem rather inactive and unsupported, so I am not sure if should rely on them.

Final notes: Woosh and pylucene seems promising, but woosh is still alpha so I am not sure I want to rely on it, and I have problems compiling pylucene, and there are no actual releases off it. After I have looked a bit more at the data, it's mostly numbers and default text strings, so as off now an indexing engine won't help me. Hopefully these libraries will stabilize and later visitors will find some use for them.

Improve this question
1
  • I used Whoosh in 2018 and it was solid :) Used it to index several thousand resumes. Commented Sep 2, 2022 at 2:14

5 Answers 5

Reset to default
9

Lupy has been retired and the developers recommend PyLucene instead. As for PyLucene, its mailing list activity may be low, but it is definitely supported. In fact, it just recently became an official apache subproject.

You may also want to look at a new contender: Whoosh. It's similar to lucene, but implemented in pure python.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

5

I haven't done indexing before, however the following may be helpful :-

  1. pyIndex - http://rgaucher.info/beta/pyIndex/ -- File indexing library for Python
  2. http://www.xml.com/pub/a/ws/2003/05/13/email.html -- Thats a script for searching Outlook email using Python and Lucene
  3. http://gadfly.sourceforge.net/ - Aaron water's gadfly database (I think you can use this one for indexing. Haven't used it myself.)

As far as using HDF files goes, I have heard of a module called h5py.

I hope this helps.

Improve this answer

2 Comments

I can read the hdf5 files fine using pytables, I just need to find the right tool to index the information I extract.
I have little experience in the area. Since you can already read hd5 files, I think that pyIndexer might work for you. I have little experience in the area and I hope your project works out well.
4

I'd suggest Sphinx. It's very active, has much more features and seems faster than Lucene.

Improve this answer

1 Comment

Sphinx is great, and IMHO, easier to install, configure etc, than pylucene.
2

A popular C++ based information retrieval library that is often used with Python is Xapian http://xapian.org/

It's incredibly quick and can happily manage large amounts of data, however it's not quite as easily extensible as Lucene.

Improve this answer

Comments

0

Elastic search can be used to index documents and search by keywords
Elasticsearch can be integrated with graph db and hadoop as well Some urls below:
1) https://www.elastic.co/products/elasticsearch
2) https://towardsdatascience.com/getting-started-with-elasticsearch-in-python-c3598e718380

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.