Mongodb Indexing Issue

It seems to me that you have not fully understood the power of document based databases such as MongoDB.

Bellow are just a few thoughts:

• you have 1 million records
• you have 1 million index values for that collection
• you have to RAM available to store 1 million index values in-memory, otherwise the benefits of indexing would not be so keen to show up
• yes you can have sharding but you need lots of hardware to accommodate basic needs

What you for sure need is something that can make dynamically link random text to valuable indexes and that allows you to search in vast amounts of text very fast. And for that you should use a tool like ElasticSearch.

Note that you can and should store your content in a NoSQL database and yes MongoDB is a viable option. And for the indexing part ElasticSearch has plugins available to enhance the communication between the two.

P.S. If I recall correctly the plugin is called MongoDB River

EDIT:

I've also added a more comprehensive definition for ElasticSearch. I won't take credit for it since I've grabbed it from Wikipedia:

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents

EDIT 2:

I've scaled down a bit on the numbers since it might be far-fetched for most projects. But the main idea remains the same. Indexes are not recommended for the use-case described in the question.

Based on what you want to query, you will end up indexing those fields. You can also have secondary indexes in MongoDB. But beware creating too many indexes may improve your query performance but consume additional disk space and make inserts slower due to re-indexing.

MongoDB indexes

Short answer: you can't. Use Elastic Search. Here is a good tutorial to setup MongoDB River on Elastic Search

The reason is simple, MongoDB does not work like that. It helps you store complex schemaless sets of documents. But you cannot index dozens of different fields and hope to get good performance. Generally a max of 5-6 indices are recommended per collection.

Elastic Search is commonly used in the fashion described above in many other use-cases, so it is an established pattern. For example, Titan Graph DB has the built-in option to use ES for this purpose. If I were you, I would just use that and would not try to make MongoDB do something it is not built to do.

If you have the time and if your data structure lends itself to (I think it might from the json above), then you could also use rdbms to break down these pieces and store them on-the-fly with an EAV like pattern. Elastic Search would be easier to start and probably easier to achieve performance quickly.

Well, there are lots of problems w.r.t having many indexes and has been discussed here. But if at all you need to add indexes for dynamic fields you actually create index from you mongo db driver.

So, lets say if you are using the Mongodb JAVA driver then you could create an index like below: http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-java-driver/#creating-an-index

coll.createIndex(new BasicDBObject("i", 1));  // create index on "i", ascending

PYTHON

http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.create_index

So, when you are populating data using any of the drivers and you find a new field which has come thru then you could fire index creation using driver itself and not have to do it manually.

P.S.: I have not tried this and it might not be suitable or advisable.

Hope this helps!

Indexing of dynamic fields is tricky. There is no such thing as wildcard-indexes. Your options would be:

Option A: Whenever you insert a new document, do an ensureIndex with the option sparse:true for each of its fields. This does nothing when the index already exists and creates a new one when it's a new field. The drawback will be that you will end up with a very large number of indexes and that inserts could get slow because of all the new and old indexes which need to be created/updated.

Option B: Forget about the field-names and refactor your documents to an array of key/value pairs. So

    "General Features" : {
        "Brand" : "Sony",
        "Form" : "Bar"
    },
    "Platform/Software" : {,
        "Processor" : "1.5 GHz Qualcomm",
        "Graphics" : "Adreno 320"
    }

becomes

 properties: [
     { category: "General Features", key: "Brand", value: "Sony" },
     { category: "General Features", key: "Form", value: "Bar" },
     { category: "Platform/Software", key: "Processor", value: "1.5 GHz Qualcomm" },
     { category: "Platform/Software", key: "Graphics", value: "Adreno 320" }
 ]

This allows you to create a single compound index on properties.category and properties.key to cover all the array entries.