Tokenizing strings using regular expression in Javascript

Question

Suppose I've a long string containing newlines and tabs as:

var x = "This is a long string.\n\t This is another one on next line.";

So how can we split this string into tokens, using regular expression?

I don't want to use .split(' ') because I want to learn Javascript's Regex.

A more complicated string could be this:

var y = "This @is a #long $string. Alright, lets split this.";

Now I want to extract only the valid words out of this string, without special characters, and punctuation, i.e I want these:

var xwords = ["This", "is", "a", "long", "string", "This", "is", "another", "one", "on", "next", "line"];

var ywords = ["This", "is", "a", "long", "string", "Alright", "lets", "split", "this"];

What do you want to split it on? You said s.split(' ') but also you mentioned newlines and tabs. You seem to be looking for a regex tutorial, which isn't really Stack Overflow's focus. — nnnnnn
– nnnnnn, Commented Dec 9, 2011 at 6:37
@nnnnnn: I'm reading this doc from MDN. But at the same time, I doing some experiment. And this is my first attempt to split sentence into words. — Sarfaraz Nawaz
– Sarfaraz Nawaz, Commented Dec 9, 2011 at 6:39

Alexander Yezutov · Accepted Answer · 2011-12-09 07:16:29Z

9

Here is a jsfiddle example of what you asked: http://jsfiddle.net/ayezutov/BjXw5/1/

Basically, the code is very simple:

var y = "This @is a #long $string. Alright, lets split this.";
var regex = /[^\s]+/g; // This is "multiple not space characters, which should be searched not once in string"

var match = y.match(regex);
for (var i = 0; i<match.length; i++)
{
    document.write(match[i]);
    document.write('<br>');
}

UPDATE: Basically you can expand the list of separator characters: http://jsfiddle.net/ayezutov/BjXw5/2/

var regex = /[^\s\.,!?]+/g;

UPDATE 2: Only letters all the time: http://jsfiddle.net/ayezutov/BjXw5/3/

var regex = /\w+/g;

edited Dec 9, 2011 at 7:16

answered Dec 9, 2011 at 6:49

Alexander Yezutov

3,2242 gold badges24 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Sarfaraz Nawaz Over a year ago

Your both examples give wrong result. The result is containing the special characters.

Alexander Yezutov Over a year ago

Hey, i thought this was your intention :) if you wish only letters in output: jsfiddle.net/ayezutov/BjXw5/3. var regex = /\w+/g;

Sarfaraz Nawaz Over a year ago

+1. That is good. It seems this can be written in many different ways.

Alexander Yezutov Over a year ago

Yes, you are right. Basically, for english, \w is a more elegant form of [a-zA-Z0-9], but \w would work with other languages as well.

Nhu Trinh Over a year ago

look like \S = ^\s

Prince John Wesley · Accepted Answer · 2011-12-09 06:34:26Z

2

Use \s+ to tokenize the string.

answered Dec 9, 2011 at 6:34

Prince John Wesley

63.9k12 gold badges90 silver badges95 bronze badges

4 Comments

Sarfaraz Nawaz Over a year ago

That doesn't seem to work. I did var re = /\s+/; var words = re.exec(x); What am I doing wrong?

Kai Over a year ago

@Nawaz var words = x.split(/\s+/);

Kai Over a year ago

@Nawaz Also try var words = y.split(/[^A-Za-z0-9]+/); to strip out punctuation, too.

Sarfaraz Nawaz Over a year ago

@Kai: Of that helped for the first string. But it doesn't work with the second string y.

kennebec · Accepted Answer · 2011-12-09 07:00:40Z

2

exec can loop through the matches to remove non-word (\W) characters.

var A= [], str= "This @is a #long $string. Alright, let's split this.",
rx=/\W*([a-zA-Z][a-zA-Z']*)(\W+|$)/g, words;

while((words= rx.exec(str))!= null){
    A.push(words[1]);
}
A.join(', ')

/*  returned value: (String)
This, is, a, long, string, Alright, let's, split, this
*/

answered Dec 9, 2011 at 7:00

kennebec

105k32 gold badges109 silver badges127 bronze badges

Comments

Mar Cnu · Accepted Answer · 2017-11-30 02:47:55Z

Here is a solution using regex groups to tokenise the text using different types of tokens.

You can test the code here https://jsfiddle.net/u3mvca6q/5/

/*
Basic Regex explanation:
/                   Regex start
(\w+)               First group, words     \w means ASCII letter with \w     + means 1 or more letters
|                   or
(,|!)               Second group, punctuation
|                   or
(\s)                Third group, white spaces
/                   Regex end
g                   "global", enables looping over the string to capture one element at a time

Regex result:
result[0] : default group : any match
result[1] : group1 : words
result[2] : group2 : punctuation , !
result[3] : group3 : whitespace
*/
var basicRegex = /(\w+)|(,|!)|(\s)/g;

/*
Advanced Regex explanation:
[a-zA-Z\u0080-\u00FF] instead of \w     Supports some Unicode letters instead of ASCII letters only. Find Unicode ranges here https://apps.timwhitlock.info/js/regex

(\.\.\.|\.|,|!|\?)                      Identify ellipsis (...) and points as separate entities

You can improve it by adding ranges for special punctuation and so on
*/
var advancedRegex = /([a-zA-Z\u0080-\u00FF]+)|(\.\.\.|\.|,|!|\?)|(\s)/g;

var basicString = "Hello, this is a random message!";
var advancedString = "Et en français ? Avec des caractères spéciaux ... With one point at the end.";

console.log("------------------");
var result = null;
do {
    result = basicRegex.exec(basicString)
    console.log(result);
} while(result != null)

console.log("------------------");
var result = null;
do {
    result = advancedRegex.exec(advancedString)
    console.log(result);
} while(result != null)

/*
Output:
Array [ "Hello",        "Hello",        undefined,  undefined ]
Array [ ",",            undefined,      ",",        undefined ]
Array [ " ",            undefined,      undefined,  " "       ]
Array [ "this",         "this",         undefined,  undefined ]
Array [ " ",            undefined,      undefined,  " "       ]
Array [ "is",           "is",           undefined,  undefined ]
Array [ " ",            undefined,      undefined,  " "       ]
Array [ "a",            "a",            undefined,  undefined ]
Array [ " ",            undefined,      undefined,  " "       ]
Array [ "random",       "random",       undefined,  undefined ]
Array [ " ",            undefined,      undefined,  " "       ]
Array [ "message",      "message",      undefined,  undefined ]
Array [ "!",            undefined,      "!",        undefined ]
null
*/

Kai · Accepted Answer · 2011-12-09 06:49:01Z

1

var words = y.split(/[^A-Za-z0-9]+/);

answered Dec 9, 2011 at 6:49

Kai

9,4285 gold badges31 silver badges28 bronze badges

Comments

awdz9nld · Accepted Answer · 2012-06-24 09:57:08Z

0

In order to extract word-only characters, we use the \w symbol. Whether or not this will match Unicode characters or not is implementation-dependent, and you can use this reference to see what the case is for your language/library.

Please see Alexander Yezutov's answer (update 2) on how to apply this into an expression.

answered Jun 24, 2012 at 9:57

awdz9nld

1,75117 silver badges26 bronze badges

Collectives™ on Stack Overflow

Tokenizing strings using regular expression in Javascript

6 Answers 6

5 Comments

4 Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

5 Comments

4 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related