According to the documentation, PowerShell Core 6.0 adds support with this escape sequence:

PS> "`u{0048}"
H

see https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_special_characters?view=powershell-6#unicode-character-ux

Maybe this isn't the PowerShell way, but this is what I do. I find it to be cleaner.

[regex]::Unescape("\u0048") # Prints H
[regex]::Unescape("\u0048ello") # Prints Hello

For those of us still on 5.1 and wanting to use the higher-order Unicode charset (for which none of these answers work) I made this function so you can simply build strings like so:

'this is my favourite park ',0x1F3DE,'. It is pretty sweet ',0x1F60A | Unicode

#takes in a stream of strings and integers,
#where integers are unicode codepoints,
#and concatenates these into valid UTF16
Function Unicode {
    Begin {
        $output=[System.Text.StringBuilder]::new()
    }
    Process {
        $output.Append($(
            if ($_ -is [int]) { [char]::ConvertFromUtf32($_) }
            else { [string]$_ }
        )) | Out-Null
    }
    End { $output.ToString() }
}

Note that getting these to display in your console is a whole other problem, but if you're outputting to an Outlook email or a Gridview (below) it will just work (as utf16 is native for .NET interfaces).

This also means you can also output plain control (not necessarily unicode) characters pretty easily if you're more comfortable with decimal since you dont actually need to use the 0x (hex) syntax to make the integers. 'hello',160,'there' | Unicode would put a non-breaking space betwixt the two words, the same as if you did 0xA0 instead.

To make it work for characters outside the BMP you need to use Char.ConvertFromUtf32()

'this is my favourite park ' + [char]::ConvertFromUtf32(0x1F3DE) + 
'. It is pretty sweet ' + [char]::ConvertFromUtf32(0x1F60A)

In PowerShell 6.0 or newer you can also use `u{}

"this is my favourite park `u{1F3DE}. It is pretty sweet `u{1F60A}"

This special character was added in PowerShell 6.0.

The Unicode escape sequence (`u{x}) allows you to specify any Unicode character by the hexadecimal representation of its code point. This includes Unicode characters above the Basic Multilingual Plane (> 0xFFFF) which includes emoji characters such as the thumbs up (`u{1F44D}) character. The Unicode escape sequence requires at least one hexadecimal digit and supports up to six hexadecimal digits. The maximum hexadecimal value for the sequence is 10FFFF.

Unicode character (`u{x})

Another way using PowerShell.

$Heart = $([char]0x2665)
$Diamond = $([char]0x2666)
$Club = $([char]0x2663)
$Spade = $([char]0x2660)
Write-Host $Heart -BackgroundColor Yellow -ForegroundColor Magenta

Use the command help Write-Host -Full to read all about it.

Note that some characters like 🌎 might need a "double rune" to be printed:

PS> "C:\foo\bar\$([char]0xd83c)$([char]0xdf0e)something.txt"

Will print:

C:\foo\bar\🌎something.txt

You can find these "runes" here, in the "unicode escape" row: https://dencode.com/string