Mongo db read performance improvement

Question

In mongodb we are storing flat data (our Event Object) containing below fields :

EventReceivedTimeStamp
EventStatus (SENT, FAILED, REJECTED)
UniqueId
System (5 systems)
Trigger (20 triggers)
ErrorMessage

Data Size & Updates

Data Count: 270 million entries
Updates per day: 3 million (3 million new added and 3 million old removed per day)
Document size: approx 2-3 kb

Search criteria:

EventReceivedTime Range and Status
EventReceived Range and System and trigger and status
Unique Id

We created compound indexes/index on the above fields.

We are not getting response times in milliseconds while performing search on the data set. How can we improve the performance of mongodb in our case?

SampleDocument

{
    "_id" : ObjectId("570cd8a55b66f3af1b4bd17e"),
    "_class" : "com.companyName.commons.dto.EventRecord",
    "uuid" : BinData(3,"10OXD+C51CEXS4E8jz0vnQ=="),
    "moc" : "APP_NOTIFICATION",
    "eventReceivedDateTime" : ISODate("2016-04-12T11:14:43.766Z"),
    "retryAttemptsMade" : 0,
    "retryAttemptDateTime" : ISODate("2016-04-12T11:14:43.766Z"),
    "eventStatus" : "FAILED",
    "systemName" : "OMS",
    "triggerName" : "COD_ORDER_NIGHT",
    "referenceId" : "39437515",
    "aggregatorBased" : true,
    "sentToDndQueue" : false,
    "lastProcessedDateTime" : ISODate("2016-04-12T11:14:44.294Z"),
    "errorMessage" : "Exception in validating attributes ",
    "suborderCodes" : [
        "39437515"
    ],
    "dndEvent" : false
}

**db.collection_name.getIndices()**

    [
        {
            "v" : 1,
            "key" : {
                "_id" : 1
            },
            "name" : "_id_",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "uuid" : 1
            },
            "name" : "uuid",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "moc" : 1
            },
            "name" : "moc",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "eventReceivedDateTime" : 1
            },
            "name" : "eventReceivedDateTime",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "retryAttemptsMade" : 1
            },
            "name" : "retryAttemptsMade",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "retryAttemptDateTime" : 1
            },
            "name" : "retryAttemptDateTime",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "eventStatus" : 1
            },
            "name" : "eventStatus",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "systemName" : 1
            },
            "name" : "systemName",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "triggerName" : 1
            },
            "name" : "triggerName",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "referenceId" : 1
            },
            "name" : "referenceId",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "aggregatorBased" : 1
            },
            "name" : "aggregatorBased",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "salesforceTagged" : 1
            },
            "name" : "salesforceTagged",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "sentToDndQueue" : 1
            },
            "name" : "sentToDndQueue",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "senderId" : 1
            },
            "name" : "senderId",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "lastProcessedDateTime" : 1
            },
            "name" : "lastProcessedDateTime",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "errorMessage" : 1
            },
            "name" : "errorMessage",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "suborderCodes" : 1
            },
            "name" : "suborderCodes",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "dndEvent" : 1
            },
            "name" : "dndEvent",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "eventStatus" : 1,
                "eventReceivedDateTime" : 1
            },
            "name" : "SDR",
            "ns" : "admin.event_record"
        },
        {
            "v" : 1,
            "key" : {
                "triggerName" : 1,
                "eventStatus" : 1,
                "systemName" : 1,
                "eventReceivedDateTime" : 1
            },
            "name" : "STSDR",
            "ns" : "admin.event_record"
        }
    ]

Note : will share the explain output

Are you using indexes? Can you provide us with example data flows? — Valeriy Solovyov
– Valeriy Solovyov, Commented Apr 23, 2016 at 6:22
We need hard data here: sample document, query, output of yourQuery.explain(), output of db.yourcoll.getIndices() — Markus W Mahlberg
– Markus W Mahlberg, Commented Apr 23, 2016 at 10:34
@MarkusWMahlberg : I have shared the sample document along with the indices. I will share the explain output in some time as our data is on cloud and i am not able to access it as of now. Thanks ! — Sahil Gupta
– Sahil Gupta, Commented Apr 23, 2016 at 15:54

Lakmal Vithanage · Accepted Answer · 2016-04-23 12:04:43Z

1

1). You can use Bulk operations to read, write and updates. It reduces the time drastically.

2). Create the schemas so that fields which used together are in one collection.

3). You have to make the search fast since even when write data need to search existing data. So make indexes bravely. If you create more indexes, data write operations goes slower.(Additionally need to update indexes)

4). use the uniqeId as the _id

answered Apr 23, 2016 at 12:04

Lakmal Vithanage

2,7878 gold badges47 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Lakmal Vithanage Over a year ago

please, consider to upvote the answer if you got help. It will helpful other with the same kind of issue.

Sahil Gupta · Accepted Answer · 2016-04-28 04:16:17Z

1

We improved performace of our queries by :

1) Changed the pagination logic. Instead of using pageNumber/PageSize , we started using id of the last element send in the last response.

2) secondly we started giving hints to our queries so that it should use proper compound indexes.

answered Apr 28, 2016 at 4:16

Sahil Gupta

2,18616 silver badges25 bronze badges

Comments

Rolson Quadras · Accepted Answer · 2016-04-23 10:03:54Z

0

It all depends on how you have created the compound indexes. While creating compound indexes, make sure to check how the query would be run on the collection. Define compound index such that the the first keys of index are match (in query - the equal statement), then add keys which are range based in the query. Refer https://docs.mongodb.org/manual/tutorial/create-queries-that-ensure-selectivity.

Also, make sure that you have sufficient memory(RAM) to process the datasets. You can refer the official MongoDB docs - https://docs.mongodb.org/manual/tutorial/ensure-indexes-fit-ram.

answered Apr 23, 2016 at 10:03

Rolson Quadras

4463 silver badges8 bronze badges

Collectives™ on Stack Overflow

Mongo db read performance improvement

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related