Couple of possible solution:
Dirty one
party.attendees.joins(:invitations).select('parties.*', 'COUNT(DISTINCT invitations.id) as invitation_count').group('parties.id').order('invitation_count')
Pros: might work
Cons: This is ugly, does not explain itself and you will end up with an extremely hard to query relation. Rather avoid.
Slightly cleaner
Sort it on the application level:
party.attendees.includes(:invitations).sort_by {|att| att.inivtaions.to_a.size }
Pros: It is more clear what you're doing
Cons: Returns non-querable array instead of relation. Still need explanation why you're calling to_a (to avoid N+1 problem).
The cleanest
Add an extra column invitations_count to attendees table, and add counter_cache: true to the belongs_to :attendee in your Invitation model. Profit!
party.attendees.order(:invitations_count)
Pros: Nice and self-explanatory. No need to load invitations from the database. Easily querable association.
Cons: You need to update all the records' counter_cache before you start using it:
Attendees.includes(:invitations).each { |att| att.invitation_count = att.invitation.to_a.size }
After this, rails will take care of holding the column in sync with a number of associated with attendee inivtations.