To search for a substring outside quotes, you can do something like this:
Pattern pat = Pattern.compile("^(?:[^']|'[^']*')*?a\\.b\\.c");
The first part will skip over:
every character that isn't a quote mark ([^']), or
every sequence of non-quote-mark characters enclosed in quotes ('[^']*').
Once those are skipped, then if it sees the pattern you want, it will know that it isn't inside quote marks.
This will handle a simple case. If things start getting more complicated, e.g. you want to allow \' to quote a quote mark in your input string the way C or Java does in a string literal, the regex starts getting more complicated, and you can quickly reach a point whether either your regex is unreadable or regexes aren't suitable solutions.
EDIT: fixed to put "reluctant" qualifier after second *, so that the first a.b.c will be found.
EDIT 2: If you want to replace the substring you find, it gets trickier. The above pattern matches the entire beginning of the string up through a.b.c, and I couldn't get a look-behind to work so that the match would be only the a.b.c part. I think you'll need to put the beginning of the string in a group, and then use $1 in the replacement string to copy the beginning:
Pattern pat = Pattern.compile("^((?:[^']|'[^']*')*?)a\\.b\\.c");
Matcher m = pat.matcher(source);
if (m.find()) {
result = m.replaceFirst("$1replacement");
}
I'm not sure replaceAll works with this, so if you want to replace all of them, you may need to loop.