Golang net/http uploading large files, undefined errors

Question

I'm having an error that is rather hard to debug when uploading large files to a server made using golang's default net/http package. The upload code looks like this:

    uploadForm.onsubmit = () => {
        const formData = new FormData(uploadForm);
        const isPublic : boolean = (<HTMLInputElement>document.getElementById('public_switch')).checked;
        formData.append('file', (<HTMLInputElement>document.getElementById('file')).files[0]);
        formData.append('compression', (<HTMLInputElement>document.getElementById('compression')).value);
        formData.append('public', String(isPublic));
        const xhr = new XMLHttpRequest();
        xhr.open("POST", "/upload/");
        xhr.send(formData);
        xhr.onreadystatechange = function() {
            console.log(xhr.responseText);
        }
    }

I have a server written in golang which I start as follows:

var server = &http.Server{
    Addr:         ":" + Configuration.Port,
    ReadTimeout:  300 * time.Second,
    WriteTimeout: 300 * time.Second,
    ReadHeaderTimeout: 300 * time.Second,
    MaxHeaderBytes: 500000000}

http.HandleFunc("/upload/", uploadFile)

server.ListenAndServe()

Finally I accept the and parse the file using the following code

func uploadFile(w http.ResponseWriter, r *http.Request) {
    //Parsing the upload arguments into the values we shall be working with
    r.ParseMultipartForm(5000000000000000)

    file, _, err := r.FormFile("file")
    //etc

Now, the code itself fails at 'r.FormFile("file")' with the very descriptive error message: "multipart: NextPart: EOF"

Is there some sort of setting on file limit or timeouts which I might not be setting either in the go code or in javascript ? The file I'm trying to upload is ~1.7GB so clearly fits within the limits supported by http.

Any idea how I could debug this issue a bit better without having to delve into FormFile or capture the request itself ? The code works just fine with smaller files ( a few Mb's).

After setting ReadTimeout to 5 mins, are you able to transfer 1.7GB file before timing out there? — Yadvendar
– Yadvendar, Commented Jun 10, 2017 at 17:51
The error appears within ~10 seconds of starting the transfer. Its done on lo so I don't think timeout is the problem, I just set it at 5 mins to get it out of the way. — George
– George, Commented Jun 10, 2017 at 18:17
It seems content is not sent to server. Test the server separately. please have a look stackoverflow.com/a/33809474 and try it. — jeevatkm
– jeevatkm, Commented Jun 10, 2017 at 19:32
Two troubleshooting tips: You can try to upload the data with a different client. Wireshark will tell you which side dropped the connection. — vbence
– vbence, Commented Jun 10, 2017 at 22:51
Ok, so, apparently the problem is (quite obviously now that I think about it), here: formData.append('file', (<HTMLInputElement>document.getElementById('file')).files[0]); — George
– George, Commented Jun 11, 2017 at 22:36

Chandan · Accepted Answer · 2022-06-02 16:41:37Z

OK, so, to answer my own question in case this stuff comes up again.

The problem is poorly javascript side and I'm afraid it has to do with the fact that the web apis for form&files aren't really up to scruff.

This is the solution I came up with javascript side:

const uploadFile = (form_id: string) => {
    const uploadForm: HTMLFormElement = <HTMLFormElement>document.getElementById(form_id);
    document.getElementById("submit_form").addEventListener("click", function(e) {
        e.preventDefault()

        let reader: FileReader = new FileReader();
        reader.readAsArrayBuffer((<HTMLInputElement>document.getElementById('file')).files[0]);

        reader.onload = function(evt) {

            const formData = new FormData(uploadForm);
            const isPublic: boolean = (<HTMLInputElement>document.getElementById('public_switch')).checked;
            formData.append('file', (<any>evt.target).result);
            formData.append('compression', (<HTMLInputElement>document.getElementById('compression')).value);
            formData.append('public', String(isPublic));
            const xhr = new XMLHttpRequest();
            xhr.open("POST", "/upload/");
            xhr.send(formData);
            xhr.onreadystatechange = function() {
                console.log(xhr.responseText + '  \n status is: ' + xhr.statusText);
            }

        };
    });

}

Quite simple really, but it took a long time to find since in the typical javascipt-community spirit all online examples are cluttered with shit.

The handling of large files should be done using a FileReader object which is basically called as such:

let reader = new FileReader();
reader.readAsArrayBuffer(dom_element_with_file_input.files[0]);

Then its used much like any other async web api:

reader.onload = function(e) {
    const your_file = e.target.result
}

The variable 'your_file' is then used for whatever you want to do with the file. In my case it was appending to form data which is done like this:

reader.onload = function(e) {
    const your_file = e.target.result
    const formData = new FormData(dom_element_of_type_form);
    formData.append('my_file', your_file);
    formData.append('other_stuff', 'a string with metadata');
}

I'm leaving this as an example since I could not actually find any so post with a clear example of the file Reader API, expect for ones cluttered in literally hundreds of lines of code. The example I followed almost to the T is the one here:

https://developer.mozilla.org/en-US/docs/Using_files_from_web_applications. But I used 'readAsArrayBuffer' instead for speed purposes.

One last note is that the reader API seems to be very slow and consumes a lot of ressource, which means that for arbitrary file sizes browser may simply crash, I had this problem with chromioum and I still can't fix it, so I'm open to suggestions, if I find a fix for browser crashes (or at least a way to handle them gracefully) I will update the question and mark it as the answer, in the meanwhile if anyone find a duplicate of my answer or a way to fix the browser crashes please tell me.

Collectives™ on Stack Overflow

Golang net/http uploading large files, undefined errors

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related