matlab packing bits into byte array

Question

in Matlab, I'm trying to pack arbitrary bit-length unsigned integers (e.g., an array of 3-bit integers) into an array of uint8. Given the tip here, I can generate code that works for "small" arrays (say 10,000 elements), but it consumes all memory for large arrays (such as 16 million elements). The code I use is below, borrowing from previous postings:

function x_bytes = stuff_bits(x, n)
    r = dec2bin(x,n);                 % bitstring for each uint in x
    s = reshape(r',[],1);             % one continuous string of bits
    t = reshape(str2num(s),8,[])';    % array of 8-bit numbers (stuffed)
    u = t*(2.^(size(t,2)-1:-1:0))';   % array of bytes representing all the bits stuffed together
    x_bytes = uint8(u);              % should be compressed byte stream
end

I realize that I'm taking a uint, converting it to a string, then converting it back to a bit; I've also read that dec2bin is not terribly efficient.

When I try this with 16mil elements (on a 64-bit windows box with 8 GB of memory), all memory is consumed. Blah. So I loop over subsections and it requires about 10 minutes to complete the 16mil elements. So, something is very inefficient.

Anyone got a better way to generate bit strings like python's BitArray?

thanks,

So you start with an array of x decimal numbers, and you want to convert each one to a binary number that is a constant length (3-bits) and then you want to compact all the 3-bit numbers into a sequence of bits and then separate it into uint8s? so x=[6 2 5 4] -> [110,010,101,100] -> [110010101100] ->[00001100,10101100] -> [12,172] ? Or am I not understanding the question? — user1860611
– user1860611, Commented Jan 23, 2013 at 23:38
Also, you are converting t into 8 very large numbers, not a bunch of 8bit numbers. Try u=(2.^(7:-1:0))*t; — user1860611
– user1860611, Commented Jan 23, 2013 at 23:45

Community · Accepted Answer · 2017-05-23 11:49:01Z

1

It seems to be similat to this one and this one

At the first one, it was suggested to use dec2bitvec inside a for loop. This may be enought to you (altought slow).

The second one sugests to create a lookup table using bitget and then use it (instead of using dec2bit or dec2bitvec)

You can try to use something 'in the middle'.

B = 3; % Number of bits per int.
A = randi(7, 16000000, 1); % 16M random elements between 1 and 7 (3bits).

tic
% get each group of bits in a column of K.
K = cell2mat(arrayfun(@(bit)bitget(A, B+1-bit), 1:B, 'UniformOutput', 0))';
% reshape to have them in 8 packs
K = reshape(K, [8, numel(K)/8])';
% get the uint8 vec.
U = K*(2.^(size(K,2)-1:-1:0))';
toc

Mine was elapsed in 3.5 seconds. (Win8 64bits, i5 4GB ram)

Instead of creating a lookup table, this code is creating a matrix (K) with bit values of each integer (stored in columns), reshaping it (to create 8bin value) and then using the same math as you used before to create the uint8 vector.

edited May 23, 2017 at 11:49

CommunityBot

11 silver badge

answered Jan 23, 2013 at 23:59

Mikhail

764 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

rocketman Over a year ago

excellent solution - thanks. Converting to the string and back must have been killing the process. Your solution keeps everything binary. Since I don't have a fixed number of bits in the input array (sometimes I have 3-bit numbers, sometimes 5-bit, sometimes 12-bit), I needed something generalized.

Mikhail Over a year ago

@rocketman I just noticed that this will only works if the total amount of bits is divisible by 8 (eg. 3bits * 16M elements), because of the reshape function. To make it work in any case you may need to convert it to a vector (1 x N) and then stuff a few bits on the left side of it prior to reshape.

rocketman Over a year ago

yes - I agree that it only works for B*numel(A)/8, but that's easy to ensure. The application is for a unique compression algorithm of image data. Dramatically increased speed... from 450 seconds to 0.23 seconds. Thanks much!

bonapart3 · Accepted Answer · 2014-12-17 19:11:56Z

0

This is the code I created for turning a matrix of bits to n bit long numbers:

function [ uD10 ] = bits_to_n_bit_integers( A, n)
%bits_to_n_bit_integersTurns vector matrix of bits in A into a vector matrix of 
%n bits long numbers. 
%B is 1 for a bit matrix
%   Detailed explanation goes here

B = 1;
% get each group of bits in a column of K.
K = cell2mat(arrayfun(@(bit)bitget(A, B+1-bit), 1:B, 'UniformOutput', 0))';
%make sure there is multiple of B
K = K(:);
while ~(mod(numel(K),n) == 0)
    K = [0;K];
end
K = K(:);
% reshape to have them in 8 packs
K = reshape(K, [n, numel(K)/n])';
% get the uint8 vec.
UD = K*(2.^(size(K,2)-1:-1:0))';

uD10=bi2de(K);

end

:)

answered Dec 17, 2014 at 19:11

bonapart3

5191 gold badge7 silver badges20 bronze badges

Collectives™ on Stack Overflow

matlab packing bits into byte array

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related