Office Add-ins XML manifest doesn't provide the ability to set specific size of the button (Large vs. Small). As the ribbon gets more crowded, the add-in commands will adjust (collapse) in an orderly way. In all cases, the add-in commands for an add-in will be grouped together. More on the topic you may find at: How do add-in commands appear?
EDIT:
After your comment I have verified myself how buttons behave when resizing the window and get less space for ribbon controls. The following are the stages:
- Lot of space for ribbon controls

- Resizing down and ribbon gets "crowed"

- Resizing even more to get kick for collapsing the ribbon groups

As you may observe this logic is built in into application and match to description from the documentation I have mentioned above. Your question is "why three items get small buttons when the group has more than three elements?" (see point #1). Well, over here I can only observe and speculate that it is internal implementation for desktop apps. Documentation has luck of such information and may be someone from Microsoft Office team able to clarify why this work like that.
Conclusion: There is no (documented) way to specify the size of the ribbon control in the manifest file.
Work around: You may group your controls in the groups by 1, 2 or 3 controls to be displayed as Large item. Be prepared user may see the smaller version of your controls anyway, which means at least provide meaningful Supertip.
Anything else: If this is something you desperately need to be included as the new feature to the Office.js you may add your scenario into https://officespdev.uservoice.com/