Why do I have empty tokens in Java String.split(regex)?

Question

I'm new to regular expression, I'm trying to use it to parse tokens separated by "(", ")" and blank space. This is my attempt:

String str = "(test (_bit1 _bit2 |bit3::&92;test#4|))";
String[] tokens = str.split("[\\s*[()]]"); 
for(int i = 0; i < tokens.length; i++)
  System.out.println(i + " : " + tokens[i]);

I expect the following output:

0 : test
1 : _bit1
2 : _bit2
3 : |bit3::&92;test#4|

However, there are two empty tokens appear in the actual output:

0 :
1 : test
2 :
3 : _bit1
4 : _bit2
5 : |bit3::&92;test#4|

I don't understand why I have two empty tokens in position 0 and 2. Could anyone give me a hint? Thank you.

===== Update ====

There was an answer of Alan Moore who deleted it. But I like the answer, so I copy it here for my own reference.

Your regex, [\s*[()]], matches one whitespace character (\s) or one of the characters *, (, or ). The delimiter at the beginning of the string (() is why you get the empty first token. There's no way around that; you just have to check for an empty first token and ignore it.

The second empty token is between the first space and the ( that follows it. That one's on you, because you used * (zero or more) instead of + (one or more). But fixing it isn't that simple. You want to split on spaces, parens, or both, but you have to make sure there's at least one character, whichever it is. This might do it:

\s*[()]+\s*|\s+

But you probably should allow for spaces between parens, too:

\s*(?:[()]+\s*)+|\s+

As a Java string literal, that would be:

\s*(?:[()]+\s*)+|\s+

Pablo Lozano · Accepted Answer · 2014-04-10 15:23:14Z

3

Your regexp is wrong, try this:

~~String[] tokens = str.split("[\s(\)]+");~~

String[] tokens = str.split("[\\s()]+"); //At least one character

UPDATE: I've noticed your code actually removes parentheses, so it seems you don't have to escape them between brackets ... not sure why, anyone can answer that?

NEW UPDATE: Thanks @AlanMoore for the explanation, as I understand parentheses within [] aren't needed to be escaped.

edited Apr 10, 2014 at 15:23

answered Apr 10, 2014 at 13:43

Pablo Lozano

10.4k2 gold badges43 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alan Moore Over a year ago

Parentheses never need to be escaped when they're inside a character class. The inner set of square brackets isn't really significant. Java (and only Java) treats it as a "nested" character class, which is unioned with the other stuff, so it's just like [\s*()]. The solution was to get rid of the * inside the brackets and add a + outside them, like you did. I misread your answer at first; if you edit it again, I can remove my downvote.

Sabuj Hassan · Accepted Answer · 2014-04-10 13:47:50Z

2

My suggestion will be, first remove the splitting characters from the both ends(to avoid empty string), and then do the splitting.

String[] tokens = str.replaceAll("^[\\s()]+|[\\s()]+$", "").split("[\\s()]+"); 
                           -- replace leading or trailing--

Also, I have placed your splitting characters(white space, ( )) inside character class []

answered Apr 10, 2014 at 13:47

Sabuj Hassan

39.7k14 gold badges83 silver badges88 bronze badges

Comments

GEMISIS · Accepted Answer · 2014-04-10 13:54:41Z

1

The problem that you are running into is that it is creating an empty string still between delimiters and then returning it once it hits a delimiter.

You can see what I'm talking about by adding an extra ( like this:

String str = "(test (_bit1 (_bit2 |bit3::&92;test#4|))";

The output will then become:

0 : 
1 : test
2 : 
3 : _bit1
4 : 
5 : _bit2
6 : |bit3::&92;test#4|

I would recommend the following code:

String str = "(test (_bit1 (_bit2 |bit3::&92;test#4|))";
String[] tokensArray = str.split("[\\s[()]*]");
ArrayList<String> tokens = new ArrayList<>();
for (String token : tokensArray) {
    if (!token.isEmpty()) {
        tokens.add(token);
    }
}
for (int i = 0; i < tokens.size(); i++)
    System.out.println(i + " : " + tokens.get(i));

What this does is remove any empty tokens from the array, since those are considered "improper" tokens.

answered Apr 10, 2014 at 13:54

GEMISIS

4443 silver badges12 bronze badges

1 Comment

Alan Moore Over a year ago

If you're willing to write that much code, you might as well use Matcher#find() in a loop to match the tokens directly.

piet.t · Accepted Answer · 2014-04-11 07:29:59Z

0

Index 0 is the token before the first (. Index 2 is the token between the space and the second ( in your input string.

I don't think you can avoid the first one, but you can suppress the second by using

str.split("[\\s()]+");

edited Apr 11, 2014 at 7:29

answered Apr 10, 2014 at 13:43

piet.t

11.9k21 gold badges45 silver badges56 bronze badges

2 Comments

sean Over a year ago

Thank you, but it doesn't work. This regex breaks the string into characters.

Alan Moore Over a year ago

You want to use +, not *. Your regex will match wherever it's applied because it doesn't have to consume any characters.

Collectives™ on Stack Overflow

Why do I have empty tokens in Java String.split(regex)?

4 Answers 4

1 Comment

Comments

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related