Transforming a UTF-8 string in a const context

I ended up writing a basic UTF-8 decoder for this purpose, as well as using a macro wrapper. The blog post linked in the question definitely helped, especially their later example of transcoding UTF-8 to UTF-16 at compile time

It seems that I've initially missed that the length calculation doesn't have to be implemented as a macro and could be a const function itself; this greatly simplified the implementation:

const fn decode_utf8_char(s: &str, start_pos: usize) -> (char, usize) {
    let s = s.as_bytes();
    let mut first_byte = s[start_pos];
    if first_byte.is_ascii() {
        (first_byte as char, 1usize)
    } else {
        // Use u32 for temporary values to carry out validation later
        let mut res = 0u32;
        let mut cont_byte_count = 0usize;
        while first_byte & 0x40 != 0 {
            cont_byte_count += 1;
            res = (res as u32) << 6 | (s[start_pos + cont_byte_count] & 0x3F) as u32;
            first_byte = first_byte << 1;
        }
        res = res | ((first_byte as u32 & 0x7F) << (cont_byte_count * 5));
        (
            char::from_u32(res).expect("Failed to decode a UTF-8 character"),
            cont_byte_count + 1,
        )
    }
}

const fn utf8_len(s: &str) -> usize {
    let mut length = 0usize;
    let mut input_pos = 0usize;
    while input_pos < s.len() {
        let (_, len) = decode_utf8_char(&s, input_pos);
        input_pos += len;
        length += 1;
    }
    length
}

const fn literal_to_custom_encoding_internal<const LEN: usize>(s: &str) -> [u8; LEN] {
    let mut output = [0u8; LEN];
    let mut input_pos = 0usize;
    let mut output_pos = 0usize;

    while input_pos < s.len() {
        let (char, len) = decode_utf8_char(&s, input_pos);
        output[output_pos] = char_to_custom_encoding(char);
        input_pos += len;
        output_pos += 1;
    }

    output
}

macro_rules! literal_to_custom_encoding {
    ($s:literal) => {{
        const STRING: [u8; utf8_len($s)] = literal_to_custom_encoding_internal::<{ utf8_len($s) }>($s);
        STRING
    }};
}

This can then be used as

literal_to_custom_encoding!("Hello, world!");

The proc-macro required should really be quite simple - you just have to parse a string literal from the macro input and convert it to a byte string literal for the macro output (using the quote and syn crates):

#[proc_macro]
fn custom_encoding(input: TokenStream) -> TokenStream {
    if let syn::Lit::Str(s) = syn::parse_macro_input!(input as syn::Lit) {
        let encoded = s.value().chars().map(char_to_custom_encoding).collect::<Vec<u8>>();

        use quote::ToTokens as _;
        syn::LitByteStr::new(&encoded, s.span()).into_token_stream().into()
    } else {
        quote::quote! {
            compile_error!("expected string literal")
        }.into()
    }
}

Playground (modified from example code to work in the playground)

Using non-const and alloc is fine here, as the macro itself runs at compile time, and requires a separate crate anyway. Your crate structure will need to be somewhat complicated, looking something like the following:

├── custom_encoding // contains char_to_custom_encoding, no_std
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
├── custom_encoding_macro // a proc-macro crate, contains custom_encoding! macro, depends on custom_encoding, uses std
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
├── Cargo.toml // depends on custom_encoding and custom_encoding_macro, no_std
└── src
    ├── main.rs
    └── lib.rs