0

I have a 500MB file which contains a data that I need to use it to update a table that contains 10M rows so I use this code in using QT SQL library

QSqlDatabase db = QSqlDatabase::addDatabase("QMYSQL");
// my database info 
db.transaction(); // start the transaction
    QSqlQuery q;
    QString sql = ""; // to append many sql statment to this string

    QFile file(dataFile);
    file.open(QIODevice::Text | QIODevice::ReadOnly);
    QTextStream stream(&file);

    int counter = 0;
    clock_t tStart = clock();
    while (!stream.atEnd()) {
        counter++;
        QString str = stream.readLine();
        QStringList list = str.split(';');
        QString id = list[0];
        QString streetName = list[1];
        QString HouseName = list[2];
        QString HouseNumber = list[3];
        QString city = list[4];
        QString postalCode = list[5];

        QString temp_sql = "UPDATE temp_table_copy SET street_name='!1', house_number='!2',postal_code='!3',address_city='!4', house='!5' where id=" + id + ";";
        temp_sql = temp_sql.replace("!1", streetName);
        temp_sql = temp_sql.replace("!2", HouseNumber);
        temp_sql = temp_sql.replace("!3", postalCode);
        temp_sql = temp_sql.replace("!4", city);
        temp_sql = temp_sql.replace("!5", HouseName);

        sql += temp_sql;
        if (sql.size() >= 100000) {
            if (!q.exec(sql)) {
                 qDebug() << "Excute Error: " << q.lastError().text();
            }
            sql = "";
            printf("Finished: %d sql statment\n", counter);
        }

        if (counter == Number)
            break;
    }
    if (!sql.isEmpty()) {
        if (!q.exec(sql)) {
            qDebug() << "Excute Error: " << q.lastError().text();
        }
        printf("Finished: %d sql statment\n", counter);
    }
    db.commit(); // commit and


    printf("Time taken: %.2fs\n", (double)(clock() - tStart)/CLOCKS_PER_SEC);
    db.close();

the idea is I append the big SQL string with the SQL statement and when it reaches the 100,000 in size I execute it and empty the string to avoid bad allocation when the string is too much size in memory, and also I read that MySQL database has a certain size to the SQL statement to be executed and I get its limit using

SHOW VARIABLES LIKE 'max_allowed_packet';

the problem here is that not all the data are updated some ids have a value in the text file and not updated inside the MySQL database table so what can be the problem? those columns are null before the update so when I try to count how many not update I found 7M only which means some ids are not updated and also I did not get any error from the query, any help or suggestion to know why this happened, or a better way to do it.
Thanks in advance.

5
  • Is the any really good cause, why you didnÄ't use prepared statements and opted for an unsecure text concatination? see doc.qt.io/qt-5/sql-sqlstatements.html Commented Apr 5, 2020 at 11:37
  • or just LOAD DATA INFILE ... Commented Apr 5, 2020 at 11:52
  • when I try to execute one command at time using statements the 1000 only take 287 seconds, which will take days to make the 10M rows update, i read that to save time to minimize excutes and try to make it in one query. Commented Apr 5, 2020 at 11:52
  • Suggest making the file as SQL file instead of text. overhead at parsing text and execute query row by row. Commented Apr 5, 2020 at 13:10
  • See the answer stackoverflow.com/a/6889129/4149835 Commented Apr 5, 2020 at 13:12

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.