Unable to return binary data from rails API

Question

I have stored a base64url_encoded string into a postgres database by using binary data type in rails. Here is the migration

def change
  add_column :gmail_attachments, :base64_data, :binary
end

The data I'm storing is coming as a base64 url encoded string from gmail API. When I tried to store the data as a string data type in postgres, I got

ArgumentError (string contains null byte)

So, I went with binary data type and it was stored successfully into database. Now, when I try

render status: 200, json: gmail_attachment_record

I get the following error

Encoding::UndefinedConversionError ("\xFF" from ASCII-8BIT to UTF-8):

How do I get rid of this error and return the stored data? Have I stored it in a wrong data type? What would be the best choice in rails ActiveRecord data types for achieving this?

nishantnigam.in/2012/01/… - check if this article helps you — Sampat Badhe
– Sampat Badhe, Commented Oct 15, 2020 at 14:38

Masroor · Accepted Answer · 2020-10-16 15:36:20Z

1

When you save a model with a :binary column, Rails is going to do all the work for you making sure that encodings are set appropriately to ensure your data is persisted correctly.

I believe you're getting the encoding error because the to_json method (implicitly called via render) is attempting to convert your binary string to UTF-8 via JSON.encode. That's why you're getting the UndefinedConversionError. (ASCII-8BIT is a special encoding in ruby that essentially means BINARY.)

To get to a working state, I think you're going to want to:

persist your data to a binary column, like you did above - consider naming it data. (Even better, use ActiveStorage and persist it to a file elsewhere.) I'd recommend you first convert your data from base64 to the raw binary representation:

# Not sure what the api exactly looks like, but you get the idea
require('base64')

base64_data = Gmail.get_attachment(...)

gmail_attachment.data = Base64.decode64(base64_data) # now our data field is just the raw bytes.
gmail_attachment.save

Then you'll want to serialize that back to base64 for transmission via to_json. You can do that through the as_json method like this:

# models/gmail_attachment.rb
require('base64')

class GmailAttachment < ApplicationRecord

  def base64_data
    Base64.encode64(self.data)
  end
end

# controllers/your_controller.rb
render json: gmail_attachment_record.as_json(except: [:data], methods: [:base64_data])

Pretty sure that should get you headed in the right direction!

edited Oct 16, 2020 at 15:36

Masroor

1,6243 gold badges16 silver badges30 bronze badges

answered Oct 15, 2020 at 15:41

Mark G.

3,2801 gold badge31 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Masroor Over a year ago

thanks Mark G. it worked perfectly. However, I'm having trouble with syntax in the actual code that I'm writing. Please see my code with the error here 0bin.net/paste/…

Masroor Over a year ago

I can now return data but looks like it is not in the form that I intended to store it as. The data I receive is in ASCII-8BIT(i.e binary, do I still decode it using Base64.decode64(..)?) I tried both with and without decoding, and when I read it from db and encode with base64 it does not really take the form it is supposed to be. I verified this will online encoder/decoders and the data I'm getting is no where near to where its supposed to be,

Mark G. Over a year ago

Glad to hear it's almost working. I thought the Gmail API returned base64 encoded data? If that's the case, you need to decode it when you persist it to your database, as I suggested above. When you send it through your rails controller, you then need to encode it. Do you have an example that shows the discrepancy? Happy to check out a gist/pastebin.

Masroor Over a year ago

Please have a look here, it contains the actual Image's URL and the Gmail Attachment data I receive from the Gmail API. 0bin.net/paste/…

Collectives™ on Stack Overflow

Unable to return binary data from rails API

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related