You're probably looking for:
In [773]: [list(x) for x in set(map(tuple, list1)).intersection(map(tuple, list2))]
Out[773]: [['trees', 'bushes'], ['cup', 'mug']]
You could also use the & operator (the intersection operator for sets):
In [778]: [list(x) for x in set(map(tuple, list1)) & set(map(tuple, list2))]
Out[778]: [['trees', 'bushes'], ['cup', 'mug']]
The reason your code does not work is because you're not converting each individual list element into something that a set can hash. For example, tuples are hashable but lists are not, and so your method gives:
In [774]: set(list1)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-774-a09f7eaac729> in <module>()
----> 1 set(list1)
TypeError: unhashable type: 'list'
However, converting each element to a tuple allows them to be hashed:
In [775]: set(map(tuple, list1))
Out[775]: {('cup', 'mug'), ('happy', 'joy', 'smile'), ('trees', 'bushes')}
The reason for this is that tuples are immutable containers.