Efficient way to build julia array element by element

Question

What's the best way to build a Julia array element by element when I am not sure of the final array size? In my problem, I am building the array in a single for loop.

So far, I have figured out I can either push into the array or index into a pre-allocated array:

using BenchmarkTools

function push_to_array!(array, N)
    for i in 1:N
        # some computation
        push!(array, rand(Int64))
    end
end

function fill_array!(array, N)
    for i in 1:N
        # some computation
        array[i]=rand(Int64)
    end
end

N = 100_000_000 # unknown in the real problem

empty = Vector{Int64}()
preallocated = Vector{Int64}(undef, 2*N) # 2*N represents some upper bound on N

@btime push_to_array!(empty, N)
#  28.272 s (6 allocations: 0 bytes)
@btime fill_array!(preallocated, N)
#  2.449 s (0 allocations: 0 bytes)

So filling a pre-allocated array is a lot faster than pushing, however, it's a bit cumbersome as I need to trim the output with correct_size = preallocated[1:N].

Is there a faster/better way of doing this?

If you have a rough idea of the final size, you can use sizehint! on the output array before you start pushing. — DNF
– DNF, Commented Apr 13, 2021 at 18:33

Przemyslaw Szufel · Accepted Answer · 2021-04-13 14:22:03Z

5

In Julia you have views - no trimming required. Simply do:

correct_size = @view preallocated[1:1_000_000]

or

correct_size = view(preallocated, 1:1_000_000)

Now this operation is very cheap (BTW note also that you are not bencharking correctly as @time measures both compile time and execution time):

julia> using BenchmarkTools

julia> @btime correct_size = @view $preallocated[1:1_000_000];
  1.800 ns (0 allocations: 0 bytes)

In conclusion making a view is basically free.

Finally, note that you can just resize! the array (the time cost is the same as in case of a @view):

resize!(preallocated, 1_000_000)

answered Apr 13, 2021 at 14:22

Przemyslaw Szufel

42.5k3 gold badges42 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

BoZenKhaa Over a year ago

Thanks for the tips, I have updated the question with @btime timings. So I take it that the best way to build the array is by preallocating or using sizehint! and then using @view or resize!.

BoZenKhaa Over a year ago

Btw, what's the dollarsign for in @view $preallocated? I couldn't find it in manual.

Przemyslaw Szufel Over a year ago

The dollar sign $ is understood by the @btime macro - it makes the global variable type stable otherwise we would be measuring unboxing time of it.

Collectives™ on Stack Overflow

Efficient way to build julia array element by element

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related