Convert text to bytes from Bash shell?

Question

How can a text string be turned into UTF-8 encoded bytes using Bash and/or common Linux command line utilities? For example, in Python one would do:

"Six of one, ½ dozen of the other".encode('utf-8')
b'Six of one, \xc2\xbd dozen of the other'

Is there a way to do this in pure Bash:

STR="Six of one, ½ dozen of the other"
<utility_or_bash_command_here> --encoding='utf-8' $STR
'Six of one, \xc2\xbd dozen of the other'

bash doesn't have a clear "text string" vs. "bytes" distinction. When you use STR="Six of one, ½ dozen of the other", it's already basically a list of bytes (more accurately, a C string), maybe in UTF-8 encoding, maybe in something else. Try echo "$STR" | od -x, and you'll probably see "bdc2" in the results. So I'm not really clear what you're trying to accomplish here. — Gordon Davisson
– Gordon Davisson, Commented May 16, 2018 at 15:47

Machinexa · Accepted Answer · 2021-09-24 12:46:33Z

5

Python to the rescue!

alias encode='python3 -c "from sys import stdin; print(stdin.read().encode(\"utf-8\"))"'

root@kali-linux:~# echo "½ " | encode
b'\xc2\xbd \n'

Also, you can remove b'' with some sed/awk thingy if you want.

answered Sep 24, 2021 at 12:46

Machinexa

6291 gold badge11 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user1717828 Over a year ago

I appreciate the effort, and unvoted for trying, but IIRC I specifically couldn't use Python to solve this problem and had to rely on bash utilities, which I guess I didn't make clear in the question. If Python is available, another option is encode() { python3 -c "print('$1'.encode())" ; } and calling encode "½ "

Machinexa Over a year ago

I too didn't realize you specifically asked for bash utils/pure bash. I saw perl answer so gave python answer. My bad

choroba · Accepted Answer · 2018-05-16 15:25:44Z

4

Perl to the rescue!

echo "$STR" | perl -pe 's/([^x\0-\x7f])/"\\x" . sprintf "%x", ord $1/ge'

The /e modifier allows to include code into the replacement part of the s/// substitution, which in this case converts ord to hex via sprintf.

answered May 16, 2018 at 15:25

choroba

245k27 gold badges221 silver badges304 bronze badges

Comments

Louis Maddox · Accepted Answer · 2022-07-19 13:34:52Z

I adapted Machinexa's nice answer a little for my needs

encoding="utf-8" is the default so no need to pass
more concise to just import sys and use directly
Here I'm looking to make a unique set not a list or concatenated bytestring

alias encode='python3 -c "import sys; enc = sys.stdin.read().encode(); print(set(enc))"'

So then I can get a set without repetition:

printf "hell0\x0\nworld\n:-)\x0:-(\n" | \
  grep -a "[[:cntrl:]]" -o | \
  perl -pe 's/([^x\0-\x7f])/"\\x" . sprintf "%x", ord $1/ge' | \
  encode

⇣

{b'\x00'}

and then if you wanted to drop the Python byte repr b'' and the backslash:

alias encode='python3 -c "from sys import stdin; encoded = stdin.read().encode(\"utf-8\"); s = set(encoded.splitlines()[:-1]); print({repr(char)[3:-1] for char in s})"'

which for the previous command gives {'x00'} instead

Collectives™ on Stack Overflow

Convert text to bytes from Bash shell?

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related