More effective way to convert file to byte array to string and back

Question

I have following way of converting file to byte array and than to string and back.

        InputStream is = new FileInputStream(new File("c:/original.png"));
        String temp = Hex.encodeHexString(IOUtils.toByteArray(is));
        System.out.println(temp);
        byte[] b = Hex.decodeHex(temp .toCharArray());
        OutputStream out = new FileOutputStream(new File("c:/copy.png"));
        IOUtils.write(b, out);

It works OK. The problem is the size of temp string. If the c:/original.png file is 1523KB than the temp size is 3046KB. Is there more effective way of converting the file to the string that would not double the size of the file? (BTW I understand why it is about twice the size)

Alternatively, how would I go about compressing the temp string?

As far as the reason for the string. It is being stored in cache that will take only strings. The file is actually upload to the web server. Once the upload is about to be downloaded it will be pulled from cache rather than the database. And the cache is there to improve search performance via predictions that I don't want calling the database each time someone searches.

Please could you clarify why you need to store the binary data in a String? — NPE
– NPE, Commented Oct 19, 2011 at 15:05
Why do you want it as a string? Is it just to render on the screen? — Jeff Foster
– Jeff Foster, Commented Oct 19, 2011 at 15:05
Do you even need the hex encoded string? If not, skip that step and use the byte array directly. — millimoose
– millimoose, Commented Oct 19, 2011 at 15:07
You might try and compress the string somehow (RLE etc.), but since a single byte has to be represented by two hex digits there's no way to reduce the size of a plain copy. — Thomas
– Thomas, Commented Oct 19, 2011 at 15:07
Also, if it's just to make a copy, you don't need to store the whole file in memory. Read 2K bytes and write them, then read the 2K subsequent bytes and write them, etc. — JB Nizet
– JB Nizet, Commented Oct 19, 2011 at 15:09

MatBanik · Accepted Answer · 2011-10-19 15:23:34Z

3

A more efficient encoding than Hex is Base64, the overhead tending toward a minimum of about 37% for large files. Unfortunately there's no standard library for it, but Apache Commons contains a class to do this.

        String temp = Base64.encodeBase64String(IOUtils.toByteArray(is));            
        byte[] b = Base64.decodeBase64(s);

edited Oct 19, 2011 at 15:23

MatBanik

26.9k40 gold badges119 silver badges179 bronze badges

answered Oct 19, 2011 at 15:06

Daniel Earwicker

117k38 gold badges209 silver badges289 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ed Staub Over a year ago

@Mat, two points: first - if you encoded/decoded correctly, you should get the input file back verbatim - not 50% bigger. Second - the memory usage is still 3x the input size, because String chars are 16 bits. I'm sure an encoding could be made that would use most of the unicode space and bring this down to near unity. I can't find one off the shelf, though.

Collectives™ on Stack Overflow

More effective way to convert file to byte array to string and back

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related