How to use ReadAllText when file encoding unknown

Question

Im reading a file with ReadAllText

    String[] values = File.ReadAllText(@"c:\\c\\file.txt").Split(';');

    int i = 0;

    foreach (String s in values)
    {
        System.Console.WriteLine("output: {0} {1} ", i, s);
        i++;
    }

If I try to read some files I get sometimes the the wrong character back (for ÖÜÄÀ...). The output is like '?', its because there is some trouble with the encoding:

output: 0 TEST
output: 1 A??O?

One solution would be to set the encoding in ReadAllText, lets say something like ReadAllText(@"c:\\c\\file.txt", Encoding.UTF8) that could fix the problem. But what if I would still get '?' as output? What if I dont know the encoding of the file? And what if every single file got a different encoding? What would be the best way to do it with c#? Thank you

You need to know what the encoding is. And there is no 100% reliable way to find out based purely on the contents of the file. — David Heffernan
– David Heffernan, Commented May 25, 2012 at 11:20
Please refer to this post stackoverflow.com/questions/2239968/… — Dhaval
– Dhaval, Commented May 25, 2012 at 11:25
You can use File.ReadAllText(Path, Encoding.Default) and the framework detects automatically if it is an ANSI or UTF8 file. — Elmue
– Elmue, Commented Jun 4, 2024 at 2:18

Romil Kumar Jain · Accepted Answer · 2012-05-25 11:18:49Z

10

The only way to reliably do this is to look for byte order marks at the start of the text file. (This blob more generally represents the endianness of character encoding used, but also the encoding - e.g. UTF8, UTF16, UTF32). Unfortunately, this method only works for Unicode-based encodings, and nothing before that (for which much less reliable methods must be used).

The StreamReader type supports detecting these marks to determine the encoding - you simply need to pass a flag to the parameter as such:

new System.IO.StreamReader("path", true)

You can then check the value of stremReader.CurrentEncoding to determine the encoding used by the file. Note however that if no byte encoding marks exist, then CurrentEncoding will default to Encoding.Default.

Refer codeproject solution to detect encoding

answered May 25, 2012 at 11:18

Romil Kumar Jain

20.8k9 gold badges67 silver badges95 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

MarkJ Over a year ago

If no byte encoding marks exist, then CurrentEncoding will use Encoding.UTF8 not Encoding.Default. "The detectEncodingFromByteOrderMarks parameter detects the encoding by looking at the first three bytes of the stream. It automatically recognizes UTF-8, little-endian Unicode, and big-endian Unicode text if the file starts with the appropriate byte order marks. Otherwise, the UTF8Encoding is used." from the docs

Tieme · Accepted Answer · 2016-01-30 13:38:12Z

You have to check file encoding first. try this

System.Text.Encoding enc = null; 
System.IO.FileStream file = new System.IO.FileStream(filePath, 
    FileMode.Open, FileAccess.Read, FileShare.Read); 
if (file.CanSeek) 
{ 
    byte[] bom = new byte[4]; // Get the byte-order mark, if there is one 
    file.Read(bom, 0, 4); 
    if ((bom[0] == 0xef && bom[1] == 0xbb && bom[2] == 0xbf) || // utf-8 
        (bom[0] == 0xff && bom[1] == 0xfe) || // ucs-2le, ucs-4le, and ucs-16le 
        (bom[0] == 0xfe && bom[1] == 0xff) || // utf-16 and ucs-2 
        (bom[0] == 0 && bom[1] == 0 && bom[2] == 0xfe && bom[3] == 0xff)) // ucs-4 
    { 
        enc = System.Text.Encoding.Unicode; 
    } 
    else 
    { 
        enc = System.Text.Encoding.ASCII; 
    } 

    // Now reposition the file cursor back to the start of the file 
    file.Seek(0, System.IO.SeekOrigin.Begin); 
} 
else 
{ 
    // The file cannot be randomly accessed, so you need to decide what to set the default to 
    // based on the data provided. If you're expecting data from a lot of older applications, 
    // default your encoding to Encoding.ASCII. If you're expecting data from a lot of newer 
    // applications, default your encoding to Encoding.Unicode. Also, since binary files are 
    // single byte-based, so you will want to use Encoding.ASCII, even though you'll probably 
    // never need to use the encoding then since the Encoding classes are really meant to get 
    // strings from the byte array that is the file. 

    enc = System.Text.Encoding.ASCII; 
}

Thanks! Note that the FileStream is open after this code and should be closed if this code is used in some GetEncoding method.

Aislan Gelatti Rocha · Accepted Answer · 2021-11-22 01:09:41Z

0

In my case, I was creating some simple json file and was getting same error. The problem was creating the file using Visual Studio (2019 at the moment).

I am sure you can find some configuration in VS options to deal with this issue. However, the quickiest way I've found was to create the same file and content using Notepad++. You can set the encoding in Notepad++ by visiting Encoding top menu. And I believe you may also find similar config in other text editors.

answered Nov 22, 2021 at 1:09

Aislan Gelatti Rocha

712 bronze badges

Collectives™ on Stack Overflow

How to use ReadAllText when file encoding unknown

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related