Finding faster ways to concatenate huge strings

Question

I'm making an app that converts images into textual RRR GGG BBB string arrays.

It works very fast with small images, but when the pixel count of input image is very high, the app slows down progressively.

The application runs x,y loop through all the pixels of input image, scans each pixel and adds its RGB formatted values to the final string which will be saved as text after the whole image is scanned.

With help of built-in profiler I found out that the System.String.Concat(string,string) takes more and more time the bigger the final string gets.

I then tried making a temporary string that will save the result of calculations of 1 row and before entering the next row, add it to the final string. Now it works about ten times faster, but still closer to the end, performance drops.

But in the end all my test images are smaller than the real ones are going to be. How do I keep concatenation speed high with even bigger images?

What do you do with the string? Can you save/send a portion of it? — Lukasz Madon
– Lukasz Madon, Commented Apr 23, 2012 at 12:03

abatishchev · Accepted Answer · 2012-04-23 12:02:00Z

8

Use System.Text.StringBuilder:

var sb = new StringBuilder();
sb.Append(r);
sb.Append(g);
sb.Append(b);

string result = sb.ToString();

This class is designed especially for fast string concatenation. I don't know anything faster (for general case) that it.

answered Apr 23, 2012 at 12:02

abatishchev

101k88 gold badges304 silver badges443 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Lukasz Madon Over a year ago

Open the link and scroll down :)

user1306322 Over a year ago

That's it, now it works even faster. Huge images are also processed fast enough.

Will Over a year ago

For optimal performance, if you roughly know the size you need, include it in the constructor, eg new StringBuilder(sizeEstimate). It then preallocates the estimated size instead of having to resize too much if starting from the (possibly much smaller) default size.

user1306322 Over a year ago

@Will my final string consists of preformatted strings of equal length and the number is vertical*horizontal image dimensions, should it improve if I pre-allocate exact space needed when initializing? (could be about 6 million symbols in a 16384*32 image)

Will Over a year ago

Yes, if you can easily precalculate the exact size, go for it! It also helps avoid memory fragmentation. A quick test shows approx 12%+ speed improvement, YMMV.

Avner Shahar-Kashtan · Accepted Answer · 2012-04-23 12:05:16Z

4

To elaborate on @abatishchev's answer:

When you concatenate strings, you're actually creating a new string instance for each concatenation, so you're allocating thousands (millions?) of tiny strings all the time. A StringBuilder, however, uses an internal character buffer to manage the string as it is built, and avoids these frequent allocations.

Usage is like this:

StringBuilder sb = new StringBuilder();
foreach (Pixel pixel in myPixels)
{
    sb.Append(ConvertToString(pixel));
}

string myString = sb.ToString();

answered Apr 23, 2012 at 12:05

Avner Shahar-Kashtan

14.8k3 gold badges42 silver badges66 bronze badges

Comments

abatishchev · Accepted Answer · 2012-04-24 07:03:22Z

3

As everyone said, yes, use StringBuilder.

BUT

If you already have the strings to concatenate in the form of some collection, String.Join() proved to be faster.

Just note that.

edited Apr 24, 2012 at 7:03

abatishchev

101k88 gold badges304 silver badges443 bronze badges

answered Apr 23, 2012 at 12:11

YoryeNathan

14.6k5 gold badges42 silver badges61 bronze badges

4 Comments

Avner Shahar-Kashtan Over a year ago

It would be surprising if String.Join is faster, considering the fact it uses StringBuilder internally, according to Reflector. EDIT: Strike that. It seems that some overloads of Join use their own unsafe implementations, and some use StringBuilder. Interesting.

Avner Shahar-Kashtan Over a year ago

I dug a bit more into string.Join, and it seems that if you have an existing array of strings (string[]), string.Join will use an internal unsafe implementation. However, if you have any other collection of strings (The IEnumerable<string> overload) or an object[], then string.Join will use a StringBuilder internally. Apparently they could squeeze some performance advantage when they know the strings are laid out sequentially in an array.

YoryeNathan Over a year ago

Makes sense. Either way, string.Join can only improve. But of course, of not in a collection beforehand - StringBuilder is the way to go.

YoryeNathan Over a year ago

By the way, turns out you can implement your own StringBuilder a lot more efficient than the framework one. Some improvements can be made on the framework StringBuilder which improve performance, although I don't remember the details.

Nikhil Agrawal · Accepted Answer · 2012-04-23 12:03:14Z

2

Use String.Join("", myarray);

answered Apr 23, 2012 at 12:03

Nikhil Agrawal

48.8k25 gold badges134 silver badges212 bronze badges

Comments

Stian · Accepted Answer · 2012-04-23 12:05:57Z

0

Use StringBuilder like this:

//add reference
using System.Text

//create string builder
StringBuilder sb = new StringBuilder();
//add items to it
sb.Append("some text");
sb.Append(" more text");

answered Apr 23, 2012 at 12:05

Stian

1,2992 gold badges19 silver badges38 bronze badges

Comments

Valeriy Gorbatikov · Accepted Answer · 2012-04-23 12:38:45Z

0

StringBuilder designed to work with huge and complex strings.

answered Apr 23, 2012 at 12:38

Valeriy Gorbatikov

3,5291 gold badge17 silver badges9 bronze badges

Collectives™ on Stack Overflow

Finding faster ways to concatenate huge strings

6 Answers 6

5 Comments

Comments

4 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

5 Comments

Comments

4 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related