4

Is there a way to upload large files to server?

I am using MultipartRequest with MultipartFile like:

  List<int> fileBytes) async {
  var request = new http.MultipartRequest("POST", Uri.parse(url));
  request.files.add(http.MultipartFile.fromBytes(
    'file',
    fileBytes,
    contentType: MediaType('application', 'octet-stream'),
    filename: fileName));
  request.headers.addAll(headers);
  var streamedResponse = await request.send();
  return await http.Response.fromStream(streamedResponse);

and reading the file like:

    html.InputElement uploadInput = html.FileUploadInputElement();
    uploadInput.multiple = false;
    uploadInput.draggable = true;
    uploadInput.click();

    uploadInput.onChange.listen((e) {
      final files = uploadInput.files;
      final file = files[0];

      final reader = new html.FileReader();

      reader.onLoadEnd.listen((e) {
        setState(() {
          _bytesData =
              Base64Decoder().convert(reader.result.toString().split(",").last);
          _selectedFile = _bytesData;
        });
      });

      reader.readAsDataUrl(file);
    });

It is OK for files around 30 MB but for more than that, I am getting Error code: Out of Memory. enter image description here

Am I doing something wrong? I saw somewhere

MultipartFile.fromBytes will give you some issues on bigger files, as the browser will limit your memory consumption.

And I think his solution is:

There’s a fromStream constructor. Usually, for bigger files, I just use HttpRequest, and put the File object in a FormData instance.

I used MultipartFile and MultipartFile.fromString and both times (for 150 MB file) that happened again. How can I use this solution? or Is there a better way to do that for files more than 500 MB?

Update

Added an answer using Worker. This is not a great solution but I think this might help someone.

7
  • 1
    indeed, dont use fromBytes named constructor as it needs a 500 MB byte buffer - instead use other constructors Commented Aug 29, 2020 at 4:15
  • I used MultipartFile and MultipartFile.fromString and both time(for 150 MB file) that happened again. That is why I think I am doing something wrong here. For fromString I can use reader.result.toString() right? Commented Aug 29, 2020 at 4:25
  • try pub.dev/documentation/http/latest/http/MultipartFile/… then Commented Aug 29, 2020 at 4:29
  • Doesn't work on web. Commented Aug 30, 2020 at 17:03
  • did you figure out a solution for this? Commented Dec 10, 2020 at 15:47

3 Answers 3

4

Update

A project to show how to do this task using this solution with a progress indicator is available here.

Currently, I solved the problem:

Import:

import 'package:universal_html/html.dart' as html;

Flutter part:

class Upload extends StatefulWidget {
  @override
  _UploadState createState() => _UploadState();
}

class _UploadState extends State<Upload> {
  html.Worker myWorker;
  html.File file;

  _uploadFile() async {
    String _uri = "/upload";

    myWorker.postMessage({"file": file, "uri": _uri});
  }

  _selectFile() {
    html.InputElement uploadInput = html.FileUploadInputElement();
    uploadInput.multiple = false;
    uploadInput.click();

    uploadInput.onChange.listen((e) {
      file = uploadInput.files.first;
    });
  }

  @override
  void initState() {
    myWorker = new html.Worker('upload_worker.js');
    myWorker.onMessage.listen((e) {
      setState(() {
        //progressbar,...
      });
    });

    super.initState();
  }

  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        RaisedButton(
          onPressed: _selectFile(),
          child: Text("Select File"),
        ),
        RaisedButton(
          onPressed: _uploadFile(),
          child: Text("Upload"),
        ),
      ],
    );
  }
}

Javascript part:

In the web folder (next to index.html), create the file 'upload_worker.js' .

self.addEventListener('message', async (event) => {
    var file = event.data.file;
    var url = event.data.uri;
    uploadFile(file, url);
});

function uploadFile(file, url) {
    var xhr = new XMLHttpRequest();
    var formdata = new FormData();
    var uploadPercent;

    formdata.append('file', file);

    xhr.upload.addEventListener('progress', function (e) {
        //Use this if you want to have a progress bar
        if (e.lengthComputable) {
            uploadPercent = Math.floor((e.loaded / e.total) * 100);
            postMessage(uploadPercent);
        }
    }, false);
    xhr.onreadystatechange = function () {
        if (xhr.readyState == XMLHttpRequest.DONE) {
            postMessage("done");
        }
    }
    xhr.onerror = function () {
        // only triggers if the request couldn't be made at all
        postMessage("Request failed");
    };

    xhr.open('POST', url, true);

    xhr.send(formdata);
}
Sign up to request clarification or add additional context in comments.

4 Comments

Hey, I used your solution and it is working absolutely fine but I am having one problem. The progress percentage is not working correctly. The progress reaches to 100% very quickly even though the file is still uploading. I am not able to figure out anything to fix it.
Did you try to log the progress in js? I'll try to add a working example with a progress bar but I think the code is fine.
A simple project to show how to use this solution with a progressbar is available here: github.com/warjeh/flutter_web_upload
I'm getting issue while uploading video file more than 8gb-10gb
1

I solved the problem using only Dart code: The way to go is to use a chunk uploader. This means to manually send the file in little parts. I send 99MB per request for example. There is already a basic implementation of this online: https://pub.dev/packages/chunked_uploader

You have to get a stream, this is possible with the file_picker or the drop_zone library. I used the drop_zone library because it provides the file picker and the drop zone functionality. In my code the dynamic file objects come from the drop_zone library.

Maybe you have to adjust the chunk uploader functionality depending one your backend. I use a django backend where I wrote a simple view that saves the files. In case of small files it can receive multipart requests with multiple files, in case of large files it can receive chunks and continiues to write a file if a previous chunk was received. Here some parts of my code:

Python backend:

@api_view(["POST"])
def upload(request):
    basePath = config.get("BasePath")
    
    targetFolder = os.path.join(basePath, request.data["taskId"], "input")
    if not os.path.exists(targetFolder):
        os.makedirs(targetFolder)

    for count, file in enumerate(request.FILES.getlist("Your parameter name on server side")):
        path = os.path.join(targetFolder, file.name)
        print(path)
        with open(path, 'ab') as destination:
            for chunk in file.chunks():
                destination.write(chunk)

    return HttpResponse("File(s) uploaded!")

flutter chunk uploader in my version:

import 'dart:async';
import 'dart:html';
import 'dart:math';
import 'package:dio/dio.dart';
import 'package:flutter_dropzone/flutter_dropzone.dart';
import 'package:http/http.dart' as http;

class UploadRequest {
  final Dio dio;
  final String url;
  final String method;
  final String fileKey;
  final Map<String, String>? bodyData;
  final Map<String, String>? headers;
  final CancelToken? cancelToken;
  final dynamic file;
  final Function(double)? onUploadProgress;
  late final int _maxChunkSize;
  int fileSize;
  String fileName;
  late DropzoneViewController controller;

  UploadRequest(
    this.dio, {
    required this.url,
    this.method = "POST",
    this.fileKey = "file",
    this.bodyData = const {},
    this.cancelToken,
    required this.file,
    this.onUploadProgress,
    int maxChunkSize = 1024 * 1024 * 99,
    required this.controller,
    required this.fileSize,
    required this.fileName,
    this.headers
  }) {
    _maxChunkSize = min(fileSize, maxChunkSize);
  }

  Future<Response?> upload() async {
    Response? finalResponse;
    for (int i = 0; i < _chunksCount; i++) {
      final start = _getChunkStart(i);
      print("start is $start");
      final end = _getChunkEnd(i);
      final chunkStream = _getChunkStream(start, end);
      
      
      var request = http.MultipartRequest(
        "POST",
        Uri.parse(url),
      );

      //request.headers.addAll(_getHeaders(start, end));
      request.headers.addAll(headers!);

      //-----add other fields if needed
      request.fields.addAll(bodyData!);

      request.files.add(http.MultipartFile(
        "Your parameter name on server side",
        chunkStream,
        fileSize,
        filename: fileName// + i.toString(),
        )
      );


      //-------Send request
      var resp = await request.send();

      //------Read response
      String result = await resp.stream.bytesToString();

      //-------Your response
      print(result);

      
    }
    return finalResponse;
  }

  Stream<List<int>> _getChunkStream(int start, int end) async* {
    print("reading from $start to $end");
    final reader = FileReader();
    final blob = file.slice(start, end);
    reader.readAsArrayBuffer(blob);
    await reader.onLoad.first;
    yield reader.result as List<int>;
  }


  // Updating total upload progress
  _updateProgress(int chunkIndex, int chunkCurrent, int chunkTotal) {
    int totalUploadedSize = (chunkIndex * _maxChunkSize) + chunkCurrent;
    double totalUploadProgress = totalUploadedSize / fileSize;
    this.onUploadProgress?.call(totalUploadProgress);
  }

  // Returning start byte offset of current chunk
  int _getChunkStart(int chunkIndex) => chunkIndex * _maxChunkSize;

  // Returning end byte offset of current chunk
  int _getChunkEnd(int chunkIndex) =>
      min((chunkIndex + 1) * _maxChunkSize, fileSize);

  // Returning a header map object containing Content-Range
  // https://tools.ietf.org/html/rfc7233#section-2
  Map<String, String> _getHeaders(int start, int end) {
    var header = {'Content-Range': 'bytes $start-${end - 1}/$fileSize'};
    if (headers != null) {
      header.addAll(headers!);
    }
    return header;
  }

  // Returning chunks count based on file size and maximum chunk size
  int get _chunksCount {
    var result = (fileSize / _maxChunkSize).ceil();
    return result;
  }
}

    

Upload code that decides whether to upload multiple files in one request or one file divided to many requests:

//upload the large files

Map<String, String> headers = {
  'Authorization': requester.loginToken!
};

fileUploadView.droppedFiles.sort((a, b) => b.size - a.size);

//calculate the sum of teh files:

double sumInMb = 0;
int divideBy = 1000000;

for (UploadableFile file in fileUploadView.droppedFiles) {
    sumInMb += file.size / divideBy;
}

var dio = Dio();

int uploadedAlready = 0;
for (UploadableFile file in fileUploadView.droppedFiles) {

  if (sumInMb < 99) {
    break;
  }

  var uploadRequest = UploadRequest(
    dio,
    url: requester.backendApi+ "/upload",
    file: file.file,
    controller: fileUploadView.controller!,
    fileSize: file.size,
    fileName: file.name,
    headers: headers,
    bodyData: {
      "taskId": taskId.toString(),
      "user": requester.username!,
    },
  );

  await uploadRequest.upload();

  uploadedAlready++;
  sumInMb -= file.size / divideBy;
}

if (uploadedAlready > 0) {
  fileUploadView.droppedFiles.removeRange(0, uploadedAlready);
}

print("large files uploaded");

// upload the small files

//---Create http package multipart request object
var request = http.MultipartRequest(
  "POST",
  Uri.parse(requester.backendApi+ "/upload"),
);


request.headers.addAll(headers);

//-----add other fields if needed
request.fields["taskId"] = taskId.toString();

print("adding files selected with drop zone");
for (UploadableFile file in fileUploadView.droppedFiles) {

  Stream<List<int>>? stream = fileUploadView.controller?.getFileStream(file.file);

  print("sending " + file.name);

  request.files.add(http.MultipartFile(
      "Your parameter name on server side",
      stream!,
      file.size,
      filename: file.name));
}


//-------Send request
var resp = await request.send();

//------Read response
String result = await resp.stream.bytesToString();

//-------Your response
print(result);

Hopefully this gives you a good overview how I solved the problem.

5 Comments

Where did you get "file" objet on Web platform ?
This is from the flutter drop_zone library.
Can you help me how to upload on S3 bucket if I have List<int> bytes for flutter web..I am getting bytes but when add multipart it crashes or invalid length. ....can you provide solution for S3 bucket
Sorry, don't know a solution for your S3 bucket problem.
what si this param for? "Your parameter name on server side" can we do the same thing with a cloud function?
0

I think if you are looking for the solution for flutter web, universal_html is yet having no support for web flutter.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.