LWP PUT of a Large File Upload Without Form Data

Question

I'm trying to implement Google's YouTube video uploading API in Perl, which, alas has no library for such. I've run into an issue with the actual PUT to upload the video file. LWP::UserAgent has a very helpful function to avoid having to stream the upload of a large file by simply referencing the file's name in an arrayref like so:

my $ua = LWP::UserAgent->new;
my $upload_url = "https://myapi";
my $localPath = "myBigVideoFile.mp4";

my $upload_response = $ua->put( $upload_url,  'Content_Type' => 'form-data', 'Content' => [ 'file' => [ $localPath ] ] );

However, that isn't the format Google's API expects. It wants the Content-Type to be 'video/*' and it wants the entire body of the response to be the file, not have it tucked away as the "file" field in a form. But changing the code to match Google's expectations disables LWP's handy file loading function. For example:

my $upload_response = $ua->put( $upload_url,  'Content_Type' => 'video/*', 'Content' => [ $localPath ] );

In that case, the LWP request object shows just the file name as the content, rather than streaming out the content.

Is there any way to activate LWP's file loading magic, or easily simulate it, so that I can achieve Google's required format without preloading the entire file (obviously not a good idea!).

Here's Google needed HTTP format:

PUT API_ADDRESS_GOES_HERE HTTP/1.1
Authorization: Bearer AUTH_TOKEN_GOES_HERE
Content-Length: CONTENT_LENGTH_GOES_HERE
Content-Type: video/*

BINARY_FILE_DATA

(In the actual code, I'm using LWP::Authen::OAuth2 on top of LWP::UserAgent, but everything I outlined above happens when I send the data to my own endpoint using just UserAgent.

Timothy R. Butler · Accepted Answer · 2024-06-01 05:38:34Z

2

This may not be the best answer (I'd love to hear a suggestion that is more elegant and triggers LWP's built in functionality), but I have managed to get a working solution by opening a file handle and passing the handle to $ua->content indirectly with an anonymous subroutine to read it (passing the file handle directly did not work as I had hoped):

$request->content(sub {
    my $buffer;
    my $bytes_read = read($fh, $buffer, 8192);
    return $bytes_read ? $buffer : undef;
});

In the final code, this requires getting the user agent from the OAuth2 object and finishing the job manually.

my $ua = $oauth2->user_agent();
my $upload_response = $ua->request($request);

LWP::Authen::OAuth2 has a built in request function that is intended to relay to LWP::UserAgent, but using that (rather than retrieving $ua and making the call directly) results in an error:

Can't store CODE items at /Users/timothybutler/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/LWP/Authen/OAuth2/AccessToken/Bearer.pm line 13.

answered Jun 1, 2024 at 5:38

Timothy R. Butler

1,1397 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ikegami Over a year ago

sysread makes more sense than read for large chunks of binary data

Collectives™ on Stack Overflow

LWP PUT of a Large File Upload Without Form Data

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related