UTF8 string variable in c#

Question

I am using PostgreSQL to power a C# desktop application. When I use the PgAdmin query analyzer to update a text column with a special character (like the copyrights trademarks) it works pefectly:

update table1 set column1='value with special character ©' where column2=1

When I use this same query from my C# application, it throws an error:

invalid byte sequence for encoding

After researching this issue, I understand that .NET strings use the UTF-16 Unicode encoding.

Consider:

string sourcetext = "value with special character ©";
// Convert a string to utf-8 bytes.
byte[] utf8Bytes = System.Text.Encoding.UTF8.GetBytes(sourcetext);

// Convert utf-8 bytes to a string. 
string desttext = System.Text.Encoding.UTF8.GetString(utf8Bytes);

The problem here is both the sourcetext and desttext are encoded as UTF-16 strings. When I pass desttext, I still get the exception.

I've also tried the following without success:

Encoder.GetString, BitConverter.GetString

Edit: I even tried this and doesn't help:

unsafe
{
  String utfeightstring = null;
  string sourcetext = "value with special character ©";
  Console.WriteLine(sourcetext);
  // Convert a string to utf-8 bytes. 
  sbyte[] utf8Chars = (sbyte[]) (Array) System.Text.Encoding.UTF8.GetBytes(sourcetext); 
  UTF8Encoding encoding = new UTF8Encoding(true, true);

  // Instruct the Garbage Collector not to move the memory
  fixed (sbyte* pUtf8Chars = utf8Chars)
  {
    utfeightstring = new String(pUtf8Chars, 0, utf8Chars.Length, encoding);
  }
  Console.WriteLine("The UTF8 String is " + utfeightstring); 
}

Is there a datatype in .NET that supports storing UTF-8 encoded string? Are there alternative ways to handle this situation?

May be foolish question to you. but why not just use System.Text.Encoding.GetString(byte[]) — HatSoft
– HatSoft, Commented Jul 11, 2012 at 13:50
As I explained encoder.getstring is not working either. By the way there is no such function available system.text.encoding.getstring. — Esen
– Esen, Commented Jul 11, 2012 at 13:51
No I am suggesting you to use System.Text.Encoding.GetString not System.Text.Encoding.UTF8.GetString — HatSoft
– HatSoft, Commented Jul 11, 2012 at 13:52
Is there a function System.Text.Encoding.GetString available in .NET 4.0? am I missing something here. It gives compilation error. — Esen
– Esen, Commented Jul 11, 2012 at 13:53
I suspect the problem is that you misconfigured your database adapter to use a wrong encoding (ANSI or so) — CodesInChaos
– CodesInChaos, Commented Jul 11, 2012 at 14:05

Peter M · Accepted Answer · 2012-07-11 14:14:56Z

5

As per this page from the mono project PostgreSQL they suggest that if you have errors with UTF8 strings that you can set the encoding to unicode in the connection string (if you are using the Npgsql driver):

Encoding: Encoding to be used. Possible values: ASCII(default) and UNICODE. Use UNICODE if you are getting problems with UTF-8 values: Encoding=UNICODE

And I have been looking in the official Npgsql docs and it isn't mentioned. NpgsqlConnection.ConnectionString

answered Jul 11, 2012 at 14:14

Peter M

7,6153 gold badges55 silver badges97 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

CodesInChaos Over a year ago

Why would they use ASCII as default -_-

Esen Over a year ago

Thanks Peter, this seems will work. I'll check and accept this as answer if it does

Peter M Over a year ago

@Esen Well I have never used postgresql so I have no idea if this will really fix your issue - but it does seem reasonable.

Esen Over a year ago

forums.devart.com/… this link is also talking similar. I am trying to set unicode=true in web.config and see if that works. Any ways your answer showed me a path to move on.

Esen Over a year ago

Yes it works. I have to set Unicode=true in the connection string. Thanks to peter for his help to resolve this issue.

|

gyluo · Accepted Answer · 2012-07-11 14:08:00Z

-1

I think it may not cause by utf-8 or 16 ,it may cause by de special character,you can replace the char with entity char which like '&amp';

answered Jul 11, 2012 at 14:08

gyluo

1

1 Comment

Esen Over a year ago

I narrowed down the issue. This query works in query analyzer and not through code. can you please explain. update table1 set column1='©' where column2=1

user2250475 · Accepted Answer · 2014-01-21 22:16:51Z

-1

Just put in your ConnectionString on the end a "...... ;Unicode=true"

answered Jan 21, 2014 at 22:16

user2250475

35 bronze badges

1 Comment

Esen Over a year ago

I already figured out the answer, commented it at Peter M answer. Why are you duplicating that again.

Collectives™ on Stack Overflow

UTF8 string variable in c#

3 Answers 3

6 Comments

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related