Updating and saving thousands of objects at once on Parse Server using cloud code. Code doesn't work for more than 2 objects. (Parse.Object.saveAll())

Question

In my parse server I have a class called Stats which contains the columns secondsPlayed (number) and timeScore (number)

I am using cloud code to update all the rows in the column timeScore

The code below works only when updating and saving 1 or 2 objects results.length. If Parse.Query returns more than 2 results the code crashes and I get the following error.

error: Failed running cloud function timeScore for user undefined with:
Input: {}
Error: {"code":101,"message":"Object not found."} functionName=timeScore, code=101, message=Object not found., , user=undefined
error: Error generating response. ParseError { code: 101, message: 'Object not found.' } code=101, message=Object not found.
error: Object not found. code=101, message=Object not found.

This is a problem as I need to update and save thousands of objects. What is the best and fastest way to do this?

Why does my code work for 2 objects but not for more than 2? How can I fix this?

Here is my code

    var _ = require("underscore");

Parse.Cloud.define("timeScore", function(request, response) {

    var query = new Parse.Query("Stats");
    query.greaterThan("secondsPlayed", 1000);
    query.find().then(function(results) {

        _.each(results, function(result) {
            var secondsPlayed = result.get("secondsPlayed") || 0;
            result.set("timeScore", secondsPlayed*2);

        });
        return Parse.Object.saveAll(results);

    }).then(function(results) {

        response.success(results);
    }, function(error) {

        response.error(error);
    }); });

Here is how I call it

#!/usr/bin/env node
var Parse = require("parse/node");
Parse.initialize("xx",   "xx");
Parse.serverURL = "http://randomapp.herokuapp.com/parse";
Parse.Cloud.run('timeScore');

UPDATE:

Below is my latest code. Everything works well except for the fact that I get the following error for no apparent reason.

heroku[router]: at=error code=H12 desc="Request timeout" method=POST path="/parse/functions/timeScore

I get the timeout error regardless of what batchSize I choose and I get it every 30 seconds. I get it a total of 5 times. After the 5th time I dont get it anymore. I get the 5th and last error at around 2.5 minutes into the process (30seconds*5). This error does not effect the process in any way. All 250k objects are updated and saved regardless of batchSize.

I thought maybe because I never call results.error or results.success the server thinks I am still doing some work and shows the error. However I updated my code as the following and I still get the timeout errors.

Also after each timeout error processBatch() is called once again from the beggining. Since I get 5 timeout errors processBatch() gets called 5 times. So after the 5th timeout error there is 5 processBatch() functions running simultaneously (I confirmed this with logs).

What is causing the heroku timeout errors I am getting? How do I fix it?

var _ = require("underscore");
Parse.Cloud.define("timeScore", function(request, response) {
var counter = 0;
function processBatch(query, batchSize, startingAt, process) {
    query.limit(batchSize);
    query.skip(startingAt);

    return query.find().then(results => {

        return process(results).then(() => results.length);
    }).then(length => {

        return (length === batchSize)? processBatch(query, batchSize, startingAt+length, process) : {};
    });
}

function setTimeScores(stats) {
        console.log("LENGTH " + stats.length);
    _.each(stats, stat => {

        counter ++;
        stat.set("timeScore", counter);

    });
    return Parse.Object.saveAll(stats);
}

var query = new Parse.Query("Stats");

processBatch(query, 2500, 0, setTimeScores).then(results => {
        response.success(results);
    }).catch(error => {
        response.error(error);
    });

});

The code looks okay. Sometimes parse errors aren't very instructive. One guess is that either Stats (or an object related to stats via a relation) has an ACL that requires a permissioned user. Here's a debug step: don't run this code on "Stats". Make a brand new class with no access control. Give a single property, an integer value which you increment by 1 in this code. Does it still break? — danh
– danh, Commented Jun 27, 2018 at 14:32
Incidentally, there is a problem in the code that will prevent it from running on thousands of objects: the max limit on the query is 1000. I can show you how to fix that once the more basic problem is solved. — danh
– danh, Commented Jun 27, 2018 at 14:40
Hi, ACL is public read and write for Stats and every Stats object. I don't think the problem is read and write permissions as the code does work when I set the query so that only 2 objects are returned. When 3 or more objects are returned I get the error. Could the solution have something to do with promises? docs.parseplatform.org/js/guide/#promises — Murat Kaya
– Murat Kaya, Commented Jun 27, 2018 at 14:56
Nope. Your promise code looks impeccable. Does the Stats object have any relations to other objects? That's where I've seen the mysterious error before: the save of the stats object succeeds, but the save of a related object that fails (parse implicitly saves an object's relations on save). My suggestion is to remove any doubt about the data model / permissions / or other errors in the data by trying this code on a brand new class that has no relations (no properties of any sort) except a single int property that you increment. — danh
– danh, Commented Jun 27, 2018 at 17:49
Hi, I did what you suggested. Tried running the code on a new class with new columns and my code worked. You were right. I realized that the problem was the format of the data I imported to mongodb was incompatible with parse. Once I fixed that my code worked. Now the only issue remaining is the query limit. The maximum number of objects that can be returned from my code is 100 not even 1000. How can I remove the query limit so I can run the code on thousands of objects? — Murat Kaya
– Murat Kaya, Commented Jun 28, 2018 at 12:48

danh · Accepted Answer · 2018-07-11 05:38:17Z

2

To handle numbers of objects greater than the max query limit, build a more abstract function that uses query's limit() and skip() to cursor through the data:

function processBatch(query, batchSize, startingAt, process) {
    query.limit(batchSize);
    query.skip(startingAt);
    return query.find().then(results => {
        return process(results).then(() => results.length);
    }).then(length => {
        return (length === batchSize)? processBatch(query, batchSize, startingAt+length, process) : {};
    });
}

This says, get one, batchSize-long batch of objects specified by query, then do something with the retrieved objects, then, if there might be more, do the same thing again, skipping objects we've processed already.

Your process step looks like this:

function setTimeScores(stats) {
    _.each(stats, stat => {
        var secondsPlayed = stat.get("secondsPlayed") || 0;
        stat.set("timeScore", secondsPlayed*2);
    });
    return Parse.Object.saveAll(stats);
}

Call it like this:

let query = new Parse.Query("Stats");
query.greaterThan("secondsPlayed", 1000);
processBatch(query, 100, 0, setTimeScores);

EDIT in the context of a cloud function, call it like this...

Parse.Cloud.define("computeStats", function(request, response) {
    let query = new Parse.Query("Stats");
    query.greaterThan("secondsPlayed", 1000);
    processBatch(query, 100, 0, setTimeScores).then(results => {
        response.success(results);
    }).catch(error => {
        response.error(error);
    });
});

edited Jul 11, 2018 at 5:38

answered Jun 28, 2018 at 14:53

danh

62.7k10 gold badges100 silver badges139 bronze badges

Sign up to request clarification or add additional context in comments.

20 Comments

Murat Kaya Over a year ago

Hi, I tried your code. Can you please check out my code posted above as an update and tell me why it doesnt work properly? My code only updates and saves 4204 objects each time and gives the error "heroku[router]: at=error code=H12 desc="Request timeout" method=POST path="/parse/functions/timeScore" 5 times during its run. Always at the same place. I indicated in my code when this error happens with console.log. This error appears 5 times on the logs and setTimeScores() is only run 5 times. After the 5th time the code just stops running without having updated all 250000 objects.

danh Over a year ago

Looks like a timeout, not a code problem. Is this running as a web request or a scheduled job. A web request has timeouts built in. The other possibility is heroku imposing a resource limitation. One test would be try try different batch sizes. Will it get ~42 runs with 100? or ~420 with 10? These won't be exact because of different overhead for different batch sizes, but it would be useful to know if the behavior happens at roughly 42 batches, and repeatably so.

danh Over a year ago

@MuratKaya - I don't want to be a pest about this, but I'm waiting for you to realize that it is unwise to design a system that visits tens of thousands of records all the time, updating each with a function based on wall clock time. You're not breaking any laws of physics here, but you are on your way to breaking laws of economics.

Murat Kaya Over a year ago

Hi, I know what you mean and understand your POV. I am just trying to make a different kind of leaderboard that is time dependent. On a traditional leaderboard you would have just scores. The early adapters of the game would always be a the top as their scores has added up over time and the new user cant compete with them as it takes a lot of time to accumulate big scores. With a time dependent scoring the new user can compete with the long term users. As the score is a fraction of time spend in app/time since they downloaded the app.

Murat Kaya Over a year ago

I was mistaken before the updated code above only updates and saves 1000 objects (equal to batchsize I set). The setTimeScores() is called 5 times but always for the same 1000 objects. If I set the batch size to 100 then the same thing happens. Only 100 objects are updated and saved and setTimeScores() is called 5 times (gets recalled after the timeout error) but always for the same 100 objects. So no matter the batch size only one batch is updated and saved with the above code.

|

Collectives™ on Stack Overflow

Updating and saving thousands of objects at once on Parse Server using cloud code. Code doesn't work for more than 2 objects. (Parse.Object.saveAll())

1 Answer 1

20 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

20 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related