I have a string test="hello how are you all doing, I hope that it's good! and fine. Looking forward to see you.
I am trying to parse the string into words and punctuation marks using javascript. I am able to separate words but then punctuation marks disappear using the regex
var result= test.match(/\b(\w|')+\b/g);
So my expected output is
hello
how
are
you
all
doing
,
I
hope
that
it's
good
!
and
fine
.
Looking
forward
to
see
you
This first approach if you, and javascript's definition of "word" match. A more customizable approach is below.
Try test.split(/\s*\b\s*/). It splits on word boundaries (\b) and eats whitespace.
"hello how are you all doing, I hope that it's good! and fine. Looking forward to see you."
.split(/\s*\b\s*/);
// Returns:
["hello",
"how",
"are",
"you",
"all",
"doing",
",",
"I",
"hope",
"that",
"it",
"'",
"s",
"good",
"!",
"and",
"fine",
".",
"Looking",
"forward",
"to",
"see",
"you",
"."]
var test = "This is. A test?"; // Test string.
// First consider splitting on word boundaries (\b).
test.split(/\b/); //=> ["This"," ","is",". ","A"," ","test","?"]
// This almost works but there is some unwanted whitespace.
// So we change the split regex to gobble the whitespace using \s*
test.split(/\s*\b\s*/) //=> ["This","is",".","A","test","?"]
// Now the whitespace is included in the separator
// and not included in the result.
If you want words like "isn`t" and "one-thousand" to be treated as a single word while javascript regex considers them to be two you will need to create your own definition of a word.
test.match(/[\w-']+|[^\w\s]+/g) //=> ["This","is",".","A","test","?"]
This matches the actual words an punctuation characters separately using an alternation. The first half of the regex [\w-']+ matches whatever you consider to be a word, and the second half [^\w\s]+ matches whatever you consider punctuation. In this example I just used whatever isn't a word or whitespace. I also but a + on the end so that multi-character punctuation (such as ?! which is properly written ‽) is treated as a single character, if you don't want that remove the +.
/[^\w\s]/ where you are saying anything that is not a word and not a space. It's not perfect but you can add more exceptions to it. Or just list everything you consider punctuation /[!@#$%^&*(),.;:'"/?\\]/.
Use this:
[,.!?;:]|\b[a-z']+\b
See the matches in the demo.
For instance, in JavaScript:
resultArray = yourString.match(/[,.!?;:]|\b[a-z']+\b/ig);
Explanation
[,.!?;:] matches one character from inside the brackets|)\b match a word boundary[a-z']+ one or more letters or apostrophes\b word boundary