Can we consider the owner as a group that only has exactly 1 member? Does the owner have any special abilities apart from the given permissions?
Not really. The owner owns the file and can change its metadata. For example, the commands chown, chgrp, chmod only work for the owner of a file/directory (or for root).
For non-ACL permissions, the owner, group, and other are exclusive groups - an account cannot be in more than one of them.
With this information you should be able to see that restricting the access permissions of a directory owner makes little sense, because as the owner they can assign any permission they choose. Therefore in your example you cannot have user3 as the owner.
Me, I'd probably own the directory as root. I'd have user2user1 and user3user2 in a group as you suggest. And user1user3 I'd leave out entirely so they they were considered as "other".
If you had yet other accounts on the system that should not be able to access these directories like user3, I'd create another group containing all the permitted users and use a directory with that group permission to host the target directory:
holding_dir: owner:root=rwx, group:big_group(users 1,2,3):rx, other:-
|
+-- target1: owner:root=rwx, group:small_group(users 2,3):rwx, other:rx
+-- target2: owner:root=rwx, group:small_group(users 2,3):rwx, other:wx