How do I tokenize a string with quoted strings in javascript?

Question

Tokens are separated by 1 or more spaces. "A quoted string" is a single token. Anything else not beginning with a quote is a token. I tried and failed with:

var tokenre = /"[^"]*"|[^"]\S+|\s\s*/g;

For instance I want this input

[4,4]  "This is fun"
 2  2 +
 #

To tokenize as

['[4,4]', '  ', '"This is fun"', '\n ', '2', '  ', '2', ' ', '+', '\n ', '#']

This could be tested with the following code:

var result = null;
do {
    result = tokenre.exec (program);
    console.log (result);
} while (result != null);

While this is for Java, I think this question might be helpful to you: stackoverflow.com/questions/366202/… — user47589
– user47589, Commented Jul 17, 2018 at 17:58
Looks like you need .match(/"[^"]*"|\S+|\s+/g), please check if it is in line with your requirements. — Wiktor Stribiżew
– Wiktor Stribiżew, Commented Jul 17, 2018 at 17:59

Wiktor Stribiżew · Accepted Answer · 2018-07-17 21:03:45Z

3

It seems you want to tokenize a string into whitespace and non-whitespace char chunks, but also separate "..." like substrings between quotes into separate elements.

You may achieve it using

s.match(/"[^"]*"|\S+|\s+/g)

See the regex demo.

Details

"[^"]*" - a ", then any 0+ chars other than a quote, and then a " (NOTE: to match regular escape sequences, you need to replace it with "[^"\\]*(?:\\[\s\S][^"\\]*)*")
| - or
\S+ - 1+ non-whitespace chars
| - or
\s+ - 1+ whitespace chars.

JS demo:

var s = "[4,4]  \"This is fun\"\n2  2 +\n#";
console.log(s.match(/"[^"]*"|\S+|\s+/g));

answered Jul 17, 2018 at 21:03

Wiktor Stribiżew

631k41 gold badges502 silver badges633 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How do I tokenize a string with quoted strings in javascript?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related