I need a tokenizer that given a string with arbitrary white-space among words will create an array of words without empty sub-strings.
For example, given a string:
" I dont know what you mean by glory Alice said."
I use:
str2.split(" ")
This also returns empty sub-strings:
["", "I", "dont", "know", "what", "you", "mean", "by", "glory", "", "Alice", "said."]
How to filter out empty strings from an array?
You probably don't even need to filter, just split using this Regular Expression:
" I dont know what you mean by glory Alice said.".split(/\b\s+/)
\b in regex?
str.match(/\S+/g)
returns a list of non-space sequences ["I", "dont", "know", "what", "you", "mean", "by", "glory", "Alice", "said."] (note that this includes the dot in "said.")
str.match(/\w+/g)
returns a list of all words: ["I", "dont", "know", "what", "you", "mean", "by", "glory", "Alice", "said"]
docs on match()
/S+/ matches against groups of characters that are not whitespace, whereas /w+/ matches groups of characters that are alphanumeric+underscore. That's why the period (.) character matches in one but not the other.You should trim the string before using split.
var str = " I dont know what you mean by glory Alice said."
var trimmed = str.replace(/^\s+|\s+$/g, '');
trimmed = str.split(" ")
I recommend .match:
str.match(/\b\w+\b/g);
This matches words between word boundaries, so all spaces are not matched and thus not included in the resulting array.
[^ ] instead of \w.see the filter method
http://www.hunlock.com/blogs/Mastering_Javascript_Arrays#quickIDX13
i think empty sub-string happen because there are multiple white-spaces you can use a replace() in a for loop to replace multiple white-spaces with a single white-space then split() to split the program using a single white space like this:
// getting full program from div
var program = document.getElementById("ans").textContent;
//removing multiple spaces
var res = program.replace(" ", " ");
for (i = 0; i <= program.length; i++) {
var res = res.replace(" ", " ");
}
// spliting each word using space as saperator
var result = res.split(" ");That is all that we need:
str.trim().split(' ')
