6

I am working on writing some code to scrub user input to my ASP.NET site. I need to scrub input to remove all references to ASCII characters 145, 146, 147, 148 which are occasionally getting input from my mac users who are copying and pasting content they write in a word processor on their macs.

My issue is the following three strings I am led to believe should output the same text.

string test1 = Convert.ToChar(147).ToString();
string test2 = String.Format("'{0}'", Convert.ToChar(147));

char[] characters = System.Text.Encoding.ASCII.GetChars(new byte[] { 147 });
string test3 = new string(characters);

Yet when I set an ASP TextBox to equal the following

txtShowValues.Text = test1 + "*" + test2 + "*" + test3;

I get a blank value for test1, test2 works correctly, and test3 outputs as a '?'.

Can someone explain what is happening differently. I am hoping this will help me understand how .NET is using ASCII values for characters over 128 so that I can write a good scrubbing script.

EDIT
The values I mentioned (145 - 148) are curly quotes. So single left, single right, double left, double right.

By "works correctly" I mean it outputs a curly quote to my browser.

SECOND EDIT
The following code (mentioned in an answer) outputs the curly quotes as well. So maybe the problem was using ASCII in test 3.

char[] characters2 = System.Text.Encoding.Default.GetChars(new byte[] { 147 });
string test4 = new string(characters2);

THIRD EDIT
I found a mac that I could borrow and was able to duplicate the problem. When I copy and paste text that has quote symbols in them from Word into my web app on the mac it pastes curly quotes (147 and 148). When I hit save curly quotes are saved to the database, so I will use the code you all helped me with to scrub that content.

FOUTH EDIT
Spent some time writing more sample code based on the responses here and noticed it has something to do with MultiLine TextBoxes in ASP.NET. There was good info here, so I decided to just start a new question: ASP.NET Multiline textbox allowing input above UTF-8

Improve this question
4
  • but then why does test2 work? If anything I expected test3 to work. Commented Feb 5, 2010 at 19:37
  • What do you mean by 'works correctly'? It's an invisible control character - it's supposed to display as a blank string. What output do you expect? Commented Feb 5, 2010 at 19:39
  • What is character 145? is it alphabetic or some sort of graphic? Commented Feb 5, 2010 at 19:40
  • 2
    You're surrounding #2 with two apostrophes. Maybe test2 prints '' which is easily confused with “ Commented Feb 5, 2010 at 19:40

3 Answers 3

Reset to default
11

Character 147 is U+0093 SET TRANSMIT STATE. Like all the Unicode characters in the range 0-255, it is the same as the ISO-8859-1 character of the same number. ISO-8859-1 assigns 147 to this invisible control code.

What you are thinking of is not ‘ASCII’ or even ‘ISO-8859-1’, but Windows code page 1252. This is a non-standard encoding that is like 8859-1, but assigns the characters 128-159 to various typographical extensions such as smart quotes instead of the largely-useless control codes. In code page 1252, character 147 is “, aka U+201C LEFT DOUBLE QUOTATION MARK.

If you want to convert Windows code pages (often misleadingly known as ‘ANSI’) to Unicode characters you will need to specify the code page you want, for example:

System.Text.Encoding.getEncoding(1252).GetChars(new byte[] { 147 })

System.Text.Encoding.Default will give you the default encoding on your server. For a server in the Western European locale, that'll be 1252. Elsewhere, it won't be. It's generally not a good idea to have a dependency on the locale's default code page in a server application.

In any case, you should not be getting bytes like 147 representing a “ in the input to a web application. That will only happen if your page itself is in code page 1252 encoding (and just to confuse and mislead even more, when you say your page is in ISO-8859-1 format, browsers will silently use code page 1252 instead). Your page may also be in 1252 if you've failed to specify any encoding for it (the browser guesses; other locales will guess different code pages so it'll all be a big mess).

Make sure you use UTF-8 for all encodings in your web app, and mark your pages as such. Today, all web apps should be using UTF-8.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

@bobince - Great information, thank you very much. I don't suppose you would have any links to documentation about this kind of stuff? I'm just trying to learn as much as possible about this issue before putting a fix into place.
The Spolsky article usually gets wheeled out at this point! (joelonsoftware.com/articles/Unicode.html)... I have my reservations about some of the material in this, but I suppose it's a reasonable enough primer.
@bobince - Is there any chance that a user copy and pasting from a word processor would send the values in to the web interface? This is a pretty rare problem, but each user I have interviewed said they were copying and pasting from their word processor on their mac.
oh, and the page is tagged UTF-8
A web browser DOM's whole content model, including input.value, is natively Unicode based. A “ character pasted into an input field will always submit encoded according the page's declared charset, so as byte 0x93 if the page is encoded in cp1252, or as bytes 0xE2, 0x80, 0x9C in UTF-8. Whilst it is technically possible to submit a real character 147 from a UTF-8-encoded page (as sequence 0xC2 0x93), it is very unlikely anyone would input a character 147.
|
3

.NET uses unicode (UCS-2) which is the same as ASCII only for values below 128.

ASCII doesn't define values above 127.

I think you may be thinking of ANSI, which defines values above 127 as (mostly) language characters needed for most European languages. or OEM (the original IBM pc character set) which defines characters > 127 as (mostly) symbols.

The difference in how the characters above 127 are interpreted is called a code page, or an encoding. (hence System.Text.Encoding). So you could probably get test 3 working if you used a different encoding, perhaps System.Text.Encoding.Default.

Edit: Ok, now that we know that the encoding you want is ANSI, it's clearer what is happening.

The rule for character conversions is to replace characters that can't be represented in encoding as some other character - usually a box. But for ASCII, there is no box character, so it uses a ? instead. This explains test 3.

test1 and 2 are both using Convert.ToChar with an integer constant. Which will interpret the input as a UNICODE character, not an ANSI character, so no conversion is being applied. Unicode character 147 is a non-printing character.

Improve this answer

Comments

0

I get question marks for all 3 of those in a console app (.NET 3.5SP1). They should all be equivalent, as far as I know. John Knoeller is correct regarding ASCII vs ANSI.

Have you tried using one of the Encoding classes' GetBytes() on the original string and iterating through, removing (by copying "good" bytes to another buffer) the values you don't want?

e.g. (using Linq):

byte[] original = System.Text.Encoding.ASCII.GetBytes(badString);
byte[] clean = (from b in original where b < 145 || b > 148 select b).ToArray<byte>();
string cleanString = System.Text.Encoding.ASCII.GetString(clean);

ASCII is probably the wrong one to use here, to be honest; if the original text is Unicode it could conceivably do bad things (if you get passed UTF-16 for example).

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.