CosmosDB (mongo/mongoose) Bulk Write Error

Question

I am trying to use insertMany() to put 50 documents in a collection. On the local emulator this works fine. When I try the same code in Azure with a CosmosDB I get the following error:

2019-12-05T00:12:51.320 [Information] Connected to Cosmsos DB
2019-12-05T00:12:51.576 [Error] failed to create users, err: BulkWriteError: Error=16500, RetryAfterMs=72, Details='
2019-12-05T00:12:51.320 [Information] Connected to Cosmsos DB
2019-12-05T00:12

Here is the connection mechanism, schema, and the code that populates the collection initially.

const personSchema = new mongoose.Schema({
    firstName: String,
    lastName: String,
    addressNumber: Number,
    streetName: String,
    city: String,
    email: String
})

async open() {
    let host = this.conn
    this.log.info(`Host: ${JSON.stringify(host)}`)
    const url = `mongodb://${host.host}:${host.port}/${host.db}?ssl=true&replicaSet=globaldb&retryWrites=false`
    const options = {
        auth: {
            user: host.user,
            password: host.password
        }
    }
    try {
        await mongoose.connect(url, options)
        this.log.info('Connected to Cosmsos DB')
        this.db = mongoose
    } catch (err) {
        this.log.error('Unable to connect to DB', err)
    }
    this.users = mongoose.model('person', personSchema)
}

async populate(logger) {
    await this.connect(logger)
    try {
        await this.users.deleteMany({})
    } catch (err) {
        this.log.error(`cannot empty collection: ${err}`)
    }
    const uList = userData.map(u => this.users(u))
    try {
        const result = await this.users.collection.insertMany(uList)
        this.log.info(`users created: ${result}`)
    } catch (err) {
        this.log.error(`failed to create users, err: ${err}`)
    } finally {
        await this.close()
    }
}

Does this answer your question? CosmosDb Request rate is large with insertMany — David Makogon
– David Makogon, Commented Dec 6, 2019 at 5:21

David Makogon · Accepted Answer · 2019-12-06 04:37:47Z

6

The error you're getting, 16500, is "Too Many Requests." TL;DR you've hit your RU limit and are being throttled. Each insert is costing some level of RU, and your per-second RU rate isn't set high enough to handle a rapid-insert scenario like what you've set up.

You can work around this by either increasing your RU capacity or reducing the number of items you're trying to insert at the same time.

FYI I just recreated this error condition by calling db.collection.insertMany() from the mongo shell with a large array of documents (in this case, about 150 small documents), combined with a very low RU count (400 in my case).

answered Dec 6, 2019 at 4:37

David Makogon

71k22 gold badges145 silver badges204 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Dr.YSG Over a year ago

Would bulk insert help? Not quite clear how it is invoked. I have 50 tiny documents. I thought 400 RU would be plenty.

Dr.YSG Over a year ago

The documentation seems to indicate that in 3.6, as long as one uses an array. insertMany is the same as bulk writes. I guess increasing RU is the only way.

Collectives™ on Stack Overflow

CosmosDB (mongo/mongoose) Bulk Write Error

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related