You can define a grammar using ANTLR and automatically generate a Ruby parser for those type of strings. ANTLR is a parser generator and it allows you to define a grammar for a language (such as a the boolean language that you described).
After parsing, you can specify what actions need to be taken to build the desired data structure (in your case a tree data structure that captures the structure of the query).
This is not particularly a Ruby problem as ANTLR can also generate parsers for other languages. In your case it would produce a Ruby parser that you can integrate into your application to parse the strings and to produce the data structure that you need.