14

I have a form which lets users input text snippets. So how can figure out the language of the entered text?

Specifically these languages for now:

Arabic: هذه هي بعض النصوص العربية

Chinese: 这是一些阿拉伯文字

Japanese: これは、いくつかのアラビア語のテキストです

[Edit] The detection has work on text which is retrieved via an API too (no browser involved)

Improve this question
2

5 Answers 5

Reset to default
8

You can figure out whether the characters are from the Arabic, Chinese, or Japanese sections of the Unicode map.

If you look at the list on Wikipedia, you'll see that each of those languages has many sections of the map. But you're not doing translation, so you don't need to worry about every last glyph.

For example, your Chinese text begins (in hex) 0x8FD9 0x662F 0x4E00 - and those are all in the "CJK Unified Ideographs" section, which is Chinese. Here are a few ranges to get you started:

Arabic (0600–06FF)

Japanese

  • Hiragana (3040–309F)
  • Katakana (30A0–30FF)
  • Kanbun (3190–319F)

Chinese

  • CJK Unified Ideographs (4E00–9FFF)

(I got the hex for your Chinese by using a Chinese to Unicode Converter.)

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Does this method distinguish Persian and Arabic too?
This doesn't detect language, it detects characters. So unless Persian contains characters that are not used in Arabic, this will not do that.
1. "CJK Unified" C is for Chinese, and J is for Japanese. That means these characters may be in both Chinese or Japanese. 2. CJK characters covers more unicode points than what described here.
@tsh True, I'd forgotten about the overlap between Japanese and Chinese.
2

You could use the Google Ajax API for detecting the language of a snippet of text.

Improve this answer

Comments

1

Presumably guessing the user's language is to display responses in the proper language. What about examining the browser's settings for preferred languages? Obtain that from the HTTP header Accept-Language. See section 14.4 here.

Improve this answer

1 Comment

This sounds like a good solution, but I forgot to mention the technique has to work on text retrieved via API too.
0

I'm exploring the same thing, for server-side. Thus far I have found https://code.google.com/p/language-detection/. Hope this helps someone.

Improve this answer

Comments

0

You could use https://detectlanguage.com/ which is a webservice build around CLD2.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.