JSON encoding/decoding with unicode in rails

Question

I ~~upgraded~~ downgraded to rails 2.3.17 due to the security bugs, but now I can't decode json strings that I have saved down to a DB if they have unicode in them :(. Is there a way to process the string such that it decodes properly?

e = ActiveSupport::JSON.encode({'a' => "Hello Unicode \u2019"})
ActiveSupport::JSON.decode(e)

gives me

RangeError: 8217 out of char range
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:314:in `unquote'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:251:in `strtok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:215:in `tok'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:178:in `lex'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:46:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/backends/okjson.rb:612:in `decode'
from /app/vendor/bundle/ruby/1.9.1/gems/activesupport-2.3.17/lib/active_support/json/decoding.rb:14:in `decode'
from (irb):30
from /usr/local/bin/irb:12:in `<main>'

I can't change the first line since it's coming from the DB like that.

This used to work.

Try not using ActiveSupport instead of the basic Jason library. — Linuxios
– Linuxios, Commented Feb 12, 2013 at 1:29

roo · Accepted Answer · 2013-02-15 03:19:19Z

3

You can change the backend JSON provider in ActiveSupport.

Add ActiveSupport::JSON.backend = "JSONGem" into an application initialiser (I added it to application.rb). This fixed the unicode parsing issues I had after I upgraded activesupport to 3.0.20.

See the vulnerability notice which caused this update - It mentions that this workaround should apply to 2.3.16 as well.

From rails console:

> ActiveSupport::VERSION::STRING
 => "3.0.20" 
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
RangeError: 8217 out of char range
> ActiveSupport::JSON.backend = "JSONGem" 
> ActiveSupport::JSON.decode('{"test":"string\u2019"}')
 => {"test"=>"string’"}

edited Feb 15, 2013 at 3:19

answered Feb 15, 2013 at 3:14

roo

7,2569 gold badges43 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

tadman · Accepted Answer · 2013-02-12 01:30:28Z

0

The JSON gem will handle this correctly.

As a note, the gem is much more strict than the other JSON parsers out there. For example:

{ 'test' : 'value' }

This is not valid JSON even though it looks okay.

For whatever reason the non-UTF-8 savvy JSON parser shipped as part of the 2.3.16 patch which is really sloppy on the part of the maintainer.

answered Feb 12, 2013 at 1:30

tadman

212k23 gold badges237 silver badges266 bronze badges

2 Comments

Brian Over a year ago

Doesn't work because the values in the DB were encoded with ActiveSupport::JSON.encode, and I doubt the gem will be able to decode all values. I temporarily fixed it by upgrading from 2.3.17 to 2.3.15. I think it's just an active support bug.

tadman Over a year ago

Yikes. You could always spin through all your records, decode with the broken parser, encode with the working one, and save them in-place.

Brian · Accepted Answer · 2013-02-12 01:46:30Z

-1

Switch to 2.3.15 which should be fine because that's when the fixes landed.
Curse the developer who started this project in rails
Begin work on porting to python post haste

answered Feb 12, 2013 at 1:46

Brian

2,6494 gold badges26 silver badges27 bronze badges

Collectives™ on Stack Overflow

JSON encoding/decoding with unicode in rails

3 Answers 3

Comments

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related