0

I'm currently trying to make a software that downloads a lot of files from Google Drive. Downloading is currently not a problem.

Nevertheless, I encounter an issue when launching 500+ simultaneous downloads. I use a slightly modified version of this tutorial : https://wiki.qt.io/Download_Data_from_URL.

Here is the .h file :

class FileDownloader : public QObject
{
    Q_OBJECT
public:
    explicit FileDownloader(QUrl url, QObject *parent = 0, int number = 0);
    QByteArray downloadedData() const;
    void launchNewDownload(QUrl url);
    QByteArray m_DownloadedData;
    QNetworkReply* reply;

    static QNetworkAccessManager *m_WebCtrl;

signals:
    void downloaded();

private slots:
    void fileDownloaded(QNetworkReply* pReply);
};

And here is the .cpp file :

QNetworkAccessManager* FileDownloader::m_WebCtrl = nullptr;

FileDownloader::FileDownloader(QUrl url, QObject *parent) :
    QObject(parent)
{
    if (m_WebCtrl == nullptr) {
        m_WebCtrl = new QNetworkAccessManager(this);
    }
    connect(m_WebCtrl, SIGNAL (finished(QNetworkReply*)),this, SLOT (fileDownloaded(QNetworkReply*)));

    launchNewDownload(url);
}

void FileDownloader::launchNewDownload(QUrl url) {
    QNetworkRequest request(url);
    this->reply = m_WebCtrl->get(request);
}

void FileDownloader::fileDownloaded(QNetworkReply* pReply) {
    m_DownloadedData = pReply->readAll();

    //emit a signal
    pReply->deleteLater();
    emit downloaded();
}

QByteArray FileDownloader::downloadedData() const {
    return m_DownloadedData;
}

The issue is "QThread::start: Failed to create thread ()" when reaching about the 500th download. I tried to limit the number of downloads which run at the same time - but I always get the same issue. Besides, I tried to delete every downloader when finishing its task - it did nothing else than crashing the program ;)

I think that it is coming from the number of threads allowed for an only process, but I'm not able to solve it !

Does anyone have an idea that could help me ?

Thank you !

7
  • 3
    Are you starting your own threads to run requests? Or do you have multiple instances of QNetworkAccessManager (maybe one for each request)? You don't need both of the above for your objective. You just need one instance of QNetworkAccessManager and your main thread (and nothing more). Use the asynchronous API QNetworkAccessManager provides. Let Qt handle low level details of parallelizing requests when possible and You should be fine. Commented Mar 5, 2017 at 21:11
  • I have multiple instances of QNetworkAccessManager, but only the main thread. When I try to use only one (static) instance of QNetworkAccessManager, my program has a strange behavior. It doesn't work anymore, files are instantly downloaded without any content... And there is much more files than expected ! Commented Mar 5, 2017 at 21:26
  • You need to add an MCVE to your question to be answerable. You must be doing something wrong in your code for that to happen. Commented Mar 5, 2017 at 21:29
  • Sorry for the MCVE, I'm not used with StackOverFlow ! Here are the files, I hope there are clear enough. Commented Mar 5, 2017 at 21:35
  • 3
    " When I connect the signal to the slot, it disconnects the previous slot.". No, this is not true. When connecting new signals/slots, Qt never disconnects old slots. You can even connect the same signal/slot pair more than once (this will result in the slot being called many times whenever the signal is emitted). Commented Mar 6, 2017 at 1:56

2 Answers 2

2

QNetworkAccessManager::finished signal is documented to be emitted whenever a pending network reply is finished.

This means that if the QNetworkAccessManager is used to run multiple requests at a time (and this is perfectly normal usage). finished signal will be emitted once for every request. Since you have a shared instance of QNetworkAccessManager between your FileDownloader objects, the finished signal gets emitted for every get call you have made. So, all the FileDownloader objects get a finished signal as soon as the first FileDownloader finishes downloading.

Instead of using QNetworkAccessManager::finished, you can use QNetworkReply::finished to avoid mixing up signals. Here is an example implementation:

#include <QtNetwork>
#include <QtWidgets>

class FileDownloader : public QObject
{
    Q_OBJECT
    //using constructor injection instead of a static QNetworkAccessManager pointer
    //This allows to share the same QNetworkAccessManager
    //object with other classes utilizing network access
    QNetworkAccessManager* m_nam;
    QNetworkReply* m_reply;
    QByteArray m_downloadedData;

public:
    explicit FileDownloader(QUrl imageUrl, QNetworkAccessManager* nam,
                            QObject* parent= nullptr)
        :QObject(parent), m_nam(nam)
    {
        QNetworkRequest request(imageUrl);
        m_reply = m_nam->get(request);
        connect(m_reply, &QNetworkReply::finished, this, &FileDownloader::fileDownloaded);
    }
    ~FileDownloader() = default;

    QByteArray downloadedData()const{return m_downloadedData;}

signals:
    void downloaded();
private slots:
    void fileDownloaded(){
        m_downloadedData= m_reply->readAll();
        m_reply->deleteLater();
        emit downloaded();
    }
};

//sample usage
int main(int argc, char* argv[]){
    QApplication a(argc, argv);

    QNetworkAccessManager nam;
    FileDownloader fileDownloader(QUrl("http://i.imgur.com/Rt8fqpt.png"), &nam);
    QLabel label;
    label.setAlignment(Qt::AlignCenter);
    label.setText("Downloading. . .");
    label.setMinimumSize(640, 480);
    label.show();
    QObject::connect(&fileDownloader, &FileDownloader::downloaded, [&]{
        QPixmap pixmap;
        pixmap.loadFromData(fileDownloader.downloadedData());
        label.setPixmap(pixmap);
    });

    return a.exec();
}

#include "main.moc"

If you are using this method to download large files, consider having a look at this question.

Sign up to request clarification or add additional context in comments.

1 Comment

Thanks, it worked ! I didn't know QNetworkReply had a signal "finished"...
0

One solution could be to uses a QThreadPool. You simply feed it tasks (QRunnable) and it will handle the number of threads and the task queue for you.

However in your case this is not perfect because you will be limiting the number of simultaneous downloads to the number of threads created by QThreadPool (generally the number of CPU core you have).

To overcome this you will have to handle the QThread yourself and not use QThreadPool. You would create a small number of thread (see QThread::idealThreadCount()) and run multiple FileDownloader on each QThread.

1 Comment

Please note that there is generally no need to use a QThreadPool when using QNetworkAccessManager. From the docs: " QNetworkAccessManager queues the requests it receives. The number of requests executed in parallel is dependent on the protocol. Currently, for the HTTP protocol on desktop platforms, 6 requests are executed in parallel for one host/port combination.". QNetworkAccessManager already handles these low level details, It is internally designed to suit the way QtWebKit works, see here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.