Spring JDBC - Batch DELETE and INSERT

Question

In my application, I have a table with around 200K records that I need to update in the database. Instead of checking for every record if a matching record exists in DB and then inserting or updating, I thought a faster approach would be to just delete all the matching records in DB and insert them. I am using the Spring JDBC framework. To delete, I used the Jdbctemplate batchUpdate method along with a ParameterizedPreparedStatementSetter and for insert I am using SimplJdbcInsert. The insert works fine, however, the batch delete performance is very slow. I am not quite sure as what other approach I should take to delete the records in DB and insert them. Any suggestions would be very helpful. I am using SQL Server 2008 R2

ParameterizedPreparedStatementSetter<Order> vSetter = 
            new ParameterizedPreparedStatementSetter<Order>() {
                @Override
                public void setValues(PreparedStatement ps,
                        Order order) throws SQLException {
                    ps.setInt(1, order.getOrderNum());
                }
    };

getJdbcTemplate().batchUpdate("DELETE FROM Order WHERE OrderNum = ?",
            aDemandOrders,
            50000,
            vSetter);

I'd suggest using MERGE instead of deleting and then inserting. — Mark Rotteveel
– Mark Rotteveel, Commented Oct 5, 2013 at 10:43
I ended up doing an update and based on update count performed an insert. Helped improve performance a lot — user320587
– user320587, Commented Oct 14, 2013 at 10:18

Brett Ryan · Accepted Answer · 2016-01-31 09:08:56Z

The reason for the slow performance is that the DB will receive the batch of statements but still execute them one by one.

An alternative is to use the in () clause and manually batch the statements to allow the DB to execute one statement with each of the batch sizes.

To still get the benefits of query caching however you must not simply send all in a single in () clause and should batch them appropriately.

private static final int MIN = 1;
private static final int SML = 4;
private static final int MED = 11;
private static final int MAX = 51;
private static final String DEL_ORDERS_QRY
    = "DELETE FROM Order WHERE OrderNum in (:orders)";

public void deleteOrders(Collection<Integer> origIds) {
    int done = getJdbcTemplate().execute((Connection con) -> {
        // Reuse this query, `:orders` is a placeholder for the in-clause.

        LinkedList<Integer> ids = new LinkedList<>(origIds);
        int remainder = ids.size();
        int updated = 0;

        while (remainder > 0) {
            // identify the batch size for this execution.
            int batchSize;
            if (remainder >= MAX) {
                batchSize = MAX;
            } else if (remainder >= MED) {
                batchSize = MED;
            } else if (remainder >= SML) {
                batchSize = SML;
            } else {
                batchSize = MIN;
            }
            remainder -= batchSize;

            // Build the in-clause parameters.
            StringBuilder inClause = new StringBuilder(batchSize * 2);
            for (int i = 0; i < batchSize; i++) {
                if (i > 0) {
                    inClause.append(',');
                }
                inClause.append('?');
            }

            try (PreparedStatement ps = con.prepareStatement(
                    DEL_ORDERS_QRY.replace(":orders", inClause.toString()))) {
                for (int i = 0; i < batchSize; i++) {
                    ps.setInt(i + 1, ids.pop());
                }
                updated += ps.executeUpdate();
            } catch (SQLException ex) {
                log.error("Couldn't execute batch", ex);
                throw new RuntimeException(ex.getMessage(), ex);
            }
        }
        return updated;
    });
}

Collectives™ on Stack Overflow

Spring JDBC - Batch DELETE and INSERT

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related