ReplaceOne throws duplicate key exception

Question

My app receives data from a remote server and calls ReplaceOne to either insert new or replace existing document with a given key with Upsert = true. (the key is made anonymous with *) The code only runs in a single thread.

However, occasionally, the app crashes with the following error:

Unhandled Exception: MongoDB.Driver.MongoWriteException: A write operation resulted in an error.                             
  E11000 duplicate key error collection: ****.orders index: _id_ dup key: { : "****-********-********-************" } ---> MongoDB.Driver.MongoBulkWriteException`1[MongoDB.Bson.BsonDocument]: A bulk write operation resulted in one or more errors.                                                                                                              
  E11000 duplicate key error collection: ****.orders index: _id_ dup key: { : "****-********-********-************" }                                                                                                                  
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)                                                                                                               
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)                                                                                       
   --- End of inner exception stack trace ---                                                                                
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)                                                                                       
   at Dashboard.Backend.AccountMonitor.ProcessOrder(OrderField& order)                                                       
   at Dashboard.Backend.AccountMonitor.OnRtnOrder(Object sender, OrderField& order)                                          
   at XAPI.Callback.XApi._OnRtnOrder(IntPtr ptr1, Int32 size1)                                                               
   at XAPI.Callback.XApi.OnRespone(Byte type, IntPtr pApi1, IntPtr pApi2, Double double1, Double double2, IntPtr ptr1, Int32 size1, IntPtr ptr2, Int32 size2, IntPtr ptr3, Int32 size3)                                                                   
Aborted (core dumped)

My question is, why is it possible to have dup key when I use ReplaceOne with Upsert = true options?

The app is working in the following environment and runtime:

.NET Command Line Tools (1.0.0-preview2-003121)

Product Information:
 Version:            1.0.0-preview2-003121
 Commit SHA-1 hash:  1e9d529bc5

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64

And MongoDB.Driver 2.3.0-rc1.

notionquest · Accepted Answer · 2016-09-13 08:23:32Z

14

Upsert works based on the filter query. If the filter query doesn't match, it will try to insert the document.

If the filter query finds the document, it will replace the document.

In your case, it could have gone in either way i.e. insert/update. Please check the data to analyze the scenario.

Insert scenario:-

The actual _id is created automatically by upsert if _id is not present in filter criteria. So, _id shouldn't create uniqueness issue. If some other fields are part of unique index, it would create uniqueness issue.

Replace scenario:-

The field that you are trying to update should have unique index defined on it. Please check the indexes on the collection and its attributes.

Optional. When true, replaceOne() either: Inserts the document from the replacement parameter if no document matches the filter. Replaces the document that matches the filter with the replacement document.

To avoid multiple upserts, ensure that the query fields are uniquely indexed.

Defaults to false.

MongoDB will add the _id field to the replacement document if it is not specified in either the filter or replacement documents. If _id is present in both, the values must be equal.

edited Sep 13, 2016 at 8:23

answered Sep 13, 2016 at 8:13

notionquest

39.4k6 gold badges120 silver badges112 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Kun Ren Over a year ago

My filter is only on _id and the document to upsert/replace has exactly the same _id.

notionquest Over a year ago

Did you check the fields that you are trying to update whether any unique index defined on it?

Kun Ren Over a year ago

The pattern is that the app receives order records from the server. Each order has a unique id and will return several times with unchanged or changed status. Each time the app receives an order, it calls ReplaceOne to insert or replace to ensure that the database stores the latest data of all orders received. To my knowledge, there shouldn't be dup-key error in this case at all.

notionquest Over a year ago

Then, you need to check the source stream. I think it is sending some duplicate data. The replaceOne adds the new _id only if _id is not present in filter or replacement document. I have updated the same in the answer. I don't see any other issue in replaceOne. Worth to check the source feed to ensure that it is not sending dup data.

Kun Ren Over a year ago

You mean if the server sends data that is exactly the same with the existing document in the database, it will throw dup-key error? For example, in db, the doc is { _id: 1, a: 2, b: 3}, replaceOne with exactly the same document would produce an error?

|

smoore4 · Accepted Answer · 2020-05-01 15:22:27Z

4

I could not get IsUpsert = true to work correctly due to a unique index on the same field used for the filter, leading to this error: E11000 duplicate key error collection A retry, as suggested in this Jira ticket, is not a great workaround.

What did seem to work was a Try/Catch block with InsertOne and then ReplaceOne without any options.

try
{
    // insert into MongoDB
    BsonDocument document = BsonDocument.Parse(obj.ToString());
    collection.InsertOne(document);
}
catch
{
    BsonDocument document = BsonDocument.Parse(obj.ToString());
    var filter = Builders<BsonDocument>.Filter.Eq("data.order_no", obj.data.order_no);
    collection.ReplaceOne(filter, document);
}

answered May 1, 2020 at 15:22

smoore4

4,9363 gold badges43 silver badges64 bronze badges

Comments

Rashad Ibrahimov · Accepted Answer · 2016-09-13 08:04:35Z

0

There is not enough information from you, but probably the scenario is the following: You receive data from server, replaceOne command doesn't match any record and try to insert new one, but probably you have a key in a document that is unique and already exists in a collection. Review and make some changes in your data before trying to update or insert it.

answered Sep 13, 2016 at 8:04

Rashad Ibrahimov

3,3193 gold badges21 silver badges41 bronze badges

Comments

jman · Accepted Answer · 2020-07-28 12:06:24Z

0

There is a bug in older MongoDB drivers, for example v2.8.1 has this problem. Update your MongoDB driver and the problem will go away. Please note when you use a new driver the DB version also needs to be updated and be compatible.

answered Jul 28, 2020 at 12:06

jman

4154 silver badges9 bronze badges

Comments

ultra909 · Accepted Answer · 2017-07-07 07:55:02Z

-1

I can co-sign on this one:

    public async Task ReplaceOneAsync(T item)
    {
        try
        {
            await _mongoCollection.ReplaceOneAsync(x => x.Id.Equals(item.Id), item, new UpdateOptions { IsUpsert = true });
        }
        catch (MongoWriteException)
        {
            var count = await _mongoCollection.CountAsync(x => x.Id.Equals(item.Id)); // lands here - and count == 1 !!!
        }
    }

answered Jul 7, 2017 at 7:55

ultra909

1,7481 gold badge23 silver badges26 bronze badges

1 Comment

Evgeny Gorbovoy Over a year ago

MongoWriteException can indicate another very different error, not related to the topic.

Collectives™ on Stack Overflow

ReplaceOne throws duplicate key exception

5 Answers 5

7 Comments

Comments

Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

7 Comments

Comments

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related