Let's make this concrete. Your suggested alternative to condition variables is for the "waiter" to do this:
loop:
lock mutex
check predicate
if (predicate is false)
unlock mutex
sleep a bit // (is this what you had in mind?)
goto loop
And for the "signaler" to do this:
lock mutex
make predicate true
unlock mutex
Here, "predicate" might be "the queue is not empty", for instance.
There are two problems with this approach. The first is the one you identified: The constant polling is inefficient. If you imagine hundreds or thousands of threads across the whole system trying to operate this way, it would bring the system to its knees. Or your "sleep a bit" would have to be so long that the sleeps themselves would add up to annoying delays.
The second problem is more subtle. There is no guarantee that when a thread unlocks a mutex and then locks it again, another thread waiting on that mutex will be allowed to run. (This property of a mutex is called "fairness"; a mutex that provides it is said to be "fair". POSIX does not require mutexes to be fair.) No matter how long you sleep in the "waiter", there is no guarantee that the "signaler" will ever get past its lock mutex call.
Condition variables solve both of these problems.