How to workaround failures with Unicode filepaths?

Windows-specific: use short paths (short filenames, 8.3 file names)

This method should work with all parts of the Mathematica system.

Limitations

• short path can be found only for existing file/directory

• short paths exist only if generation of them isn't disabled by administrator (by default it is enabled)

• may not work for network paths

• this method isn't reliable for localized versions of Windows because on them the short paths allowed to contain non-ASCII characters (example).

Solutions

There is an undocumented function FileInformation which allows to obtain the short path (works at least starting from Mathematica 11.3, thanks to GenericAccountName):

Export["Тест.txt", ""]
FileInformation["Тест.txt", "AbsoluteShortFileName"]

"Тест.txt"

"D:\\DOCUME~1\\47BB~1.TXT"

Also starting from version 12.0.0 there is an under-documented (but "official") way to obtain the short path:

Information[File["Тест.txt"], "Properties"] // Shallow

{"ObjectType", "FileName", "DirectoryName", "AbsoluteFileName", "AbsoluteShortFileName", "ShortFileName", "FileBaseName", "FileExtension", "CreationDate", "LastAccessDate", <<33>>}

Information[File["Тест.txt"], "AbsoluteShortFileName"]

"D:\\DOCUME~1\\47BB~1.TXT"

Here is a function which expands a file/directory name and then obtains the corresponding short path using cmd.exe by this method (tested on Windows 7 x64 but should work at least starting from Windows XP):

shortPath[longPath_String] /; FileExistsQ[longPath] := 
 Import["!cmd /c for %A in (\"" <> ExpandFileName[longPath] <> "\") do @echo %~sA", "Text"]

Another solution via NETLink` can be found in this great answer by WReach (see also this answer).

Usage

Create a PDF file with Unicode filepath and find the short path to it:

Export["Тест.pdf", Plot[Sin[x], {x, 0, Pi}]]
shortPath["Тест.pdf"]

"Тест.pdf"
"D:\\DOCUME~1\\FF12~1.PDF"

Import this file:

Import[shortPath["Тест.pdf"]]

Import as "String" first, and then apply ImportString or ...

Currently it is the most elegant workaround, and besides that it is expected to be OS-independent.

Limitations

• This method is intended exclusively for import-related tasks.

• If the path defined via $TemporaryDirectory contains non-ASCII characters, ImportString will fail because under the hood it creates a temp file in $TemporaryDirectory and then passes its path to the appropriate importer, hence we return back to the situation with non-ASCII file path (checked with Mathematica 8.0.4 and 11.1.0 on Windows 7 x64). A workaround is to Import from a stream with BinaryFormat -> True (see below).

Solution

It is as simple as described in the head of this answer. You Import your file with Unicode path as "String" (what works!), and then do with the obtained string what you had to do with the original file. For example pass it to ImportString if your purpose was to import the file:

ImportString[Import[<path to PDF file>, "String"], "PDF"]

... or directly include it into HTTPRequest instead of symbolic link to the local file (see the corresponding example in the question):

HTTPRequest["http://stackoverflow.com/upload/image", <|
  "Body" -> {"image" -> <|"Content" -> Import["Тест.png", "String"], 
                          "MIMEType" -> "image/png"|>}|>]

With Mathematica 10 or higher instead of Import one can use ReadString what gives identical result and also is a bit faster:

ImportString[ReadString[<path to PDF file>], "PDF"]

A low-level alternative is to OpenRead the file with option BinaryFormat -> True and then Read it as a whole with the RecordSeparators -> {} option (don't forget to Close it afterwards!):

# &[Read[#, Record, RecordSeparators -> {}], Close[#]] &@
 OpenRead[fileName, BinaryFormat -> True]

... or directly Import from the stream:

# &[Import[#, "PDF"], Close[#]] &@OpenRead[<path to PDF file>, BinaryFormat -> True]

Usage

Import a PDF file with Unicode filepath:

Export["Тест.pdf", Plot[Sin[x], {x, 0, Pi}]]

ImportString[Import["Тест.pdf", "String"], "PDF"]

"Тест.pdf"

Upload an image with Unicode filepath to imgur.com:

Export["Тест.png", Plot[Sin[x], {x, 0, Pi}]]

URLRead[HTTPRequest[
  "http://stackoverflow.com/upload/image", <|
   "Body" -> {"image" -> <|"Content" -> Import["Тест.png", "String"], 
       "MIMEType" -> "image/png"|>}|>]]

"Тест.png"

Works on Linux: set the environment variable LANG to en_US.UTF-8

In particular, with Mathematica installed in ~/opt/Mathematica/11.2.0, running it in the following way appears to make Import and Export work without any special tricks:

LANG=en_US.UTF-8 ~/opt/Mathematica/11.2.0/Executables/Mathematica

When it's run as in the example above, the following commands work as intended:

Export["/tmp/тест.txt", "Hello"]
Import@%

"/tmp/тест.txt"

"Hello"