PHP Unicode in JSON

Question

I'm sending a JSON POST body to my PHP web service that looks something like this:

{
    "foo": "☺"
}

When I echo out the body in the PHP, I see this:

{
    "foo":"\xe2\x98\xba"
}

I've also tried sending the \uXXXX equivalent:

{
    "foo": "\u263a"
}

This got further, in that the raw JSON string received had "foo":"\\u263a", but after json_decode the value turned to \xe2\x98\xba.

This is causing problems when I come to use the value in a JSON response. I get:

json_encode(): Invalid UTF-8 sequence in argument

At its simplest, this is what happens why I try to JSON encode the string:

> php -r 'echo json_encode("\x98\xba\xe2");'
PHP Warning:  json_encode(): Invalid UTF-8 sequence in argument in Command line code on line 1

My question is: how can I best get this smiley face from one end of my application to the other?

I'd appreciate any help you could offer.

echo "\xe2\x98\xba"; works; shows the correct unicode smilie. — Danny Beckett
– Danny Beckett, Commented Jun 3, 2013 at 11:49
@PleaseStand I sent it to error_log with error_log(file_get_contents('php://input')). — Ross McFarlane
– Ross McFarlane, Commented Jun 3, 2013 at 11:49
@rossmcf Could you post the lines that are throwing the error, please? — Mifeet
– Mifeet, Commented Jun 3, 2013 at 12:01
You need to understand the relationship between Unicode characters and their UTF-8 encodings. All that you are showing is correct, but context matters. — Jon
– Jon, Commented Jun 3, 2013 at 12:01

PleaseStand · Accepted Answer · 2013-06-03 12:36:24Z

3

PHP's json_decode() function behaves correctly given your input case, returning the sequence of UTF-8 bytes (E2 98 BA) that represent the character.

However, Apache HTTPD applies the \x escaping (in function ap_escape_logitem()) before writing the line to the error log (as you did for testing purposes using error_log()). As noted in file server/gen_test_char.c, "all [...] 8-bit chars with the high bit set" are escaped.

edited Jun 3, 2013 at 12:36

answered Jun 3, 2013 at 12:25

PleaseStand

32.2k7 gold badges72 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2020-06-20 09:12:55Z

2

I believe this is the correct behavior of json_encode. If you use the following:

<script>
    alert(
     <?php
       $a = "☺";
       echo json_encode($a);
     ?>
    );
</script>

The HTML output will be alert("\u263a"); and the alert will show ☺ since "\u263a" is a correct representation of the string in JavaScript.

Usage of JSON_UNESCAPED_UNICODE constant as the second parameter of json_encode in PHP is also an option, but available only for PHP 5.4.0 or newer.

In what scenario do you intend to use the value?

Edit:

php -r 'echo json_encode("\x98\xba\xe2");'

PHP Warning: json_encode(): Invalid UTF-8 sequence in argument in Command line code on line 1

The problem is you use a wrong sequence of characters. It should be

echo json_encode("\xe2\x98\xba"); // this works for me

instead of

echo json_encode("\x98\xba\xe2");

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Jun 3, 2013 at 11:49

Mifeet

13.8k6 gold badges62 silver badges110 bronze badges

5 Comments

Ross McFarlane Over a year ago

I think you're on to something here. The value needs to be returned as JSON, and that's where I'm having trouble.

Mifeet Over a year ago

@rossmcf So you want to send a string with that character as JSON reponse from PHP, right? And what is the trouble? If the JSON response is processed by JavaScript, it should behave correctly even if the result is \u263a instead of ☺.

Ross McFarlane Over a year ago

The trouble is that json_encode won't encode '\x98\xba\xe2', at least in my version of PHP.

Mifeet Over a year ago

@rossmcf I think the problem is somewhere in the code you're not showing us. My guess is there is an inappropriate double conversion.

Ross McFarlane Over a year ago

I've just added some more detail to the question. It seems to be a problem with json_encode when dealing with the internal representation of the string, i.e. '\x98\xba\xe2'.

Arun Killu · Accepted Answer · 2013-06-03 12:05:38Z

1

I think when you encode you have to use json_encode({ foo": "☺"}, JSON_UNESCAPED_UNICODE)

Basically json_encode function works only for UTF-8 encoding so before you encode check the encoding of string,like this .

 mb_check_encoding("your string", 'UTF-8') ;

if it returns false then you can convert to utf-8 using

utf8_encode("your string");

edited Jun 3, 2013 at 12:05

answered Jun 3, 2013 at 11:51

Arun Killu

14.3k5 gold badges38 silver badges66 bronze badges

1 Comment

Ross McFarlane Over a year ago

Thanks Arun. When I tried your suggestion, json_encode outputted: "\u0098\u00ba\u00e2", which is three other characters altogether.

Collectives™ on Stack Overflow

PHP Unicode in JSON

3 Answers 3

Comments

5 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

5 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related