The immediate error here is that Animal is defining two data constructors, which have nothing to do with Cat: The expression Cat is of type Animal, while the expression BigCat is of type Cat.
To create nested data types in simple fashion, you'd need to make the Cat type an argument to the relevant constructor:
data Cat = BigCat | SmallCat
data Animal = Cat Cat | Dog
You can then do something like this:
bigger (Cat SmallCat) (Cat BigCat) = False
bigger (Cat BigCat) (Cat SmallCat) = True
bigger Dog (Cat _) = True
bigger (Cat _) Dog = False
This becomes exceeding clumsy if extended beyond a trivial example, however, the redundancy in the Cat type is painful, and the two different uses of the identifier Cat is needlessly confusing. A slight improvement is to avoid conflating size with species, and instead do something like this:
data Size = Big | Small
data Species = Cat | Dog
data Animal = Animal Species Size
An advantage here is that you can more easily extend either types without having to add as much boilerplate nonsense as would otherwise be required.
However, both of these are very silly as anything other than toy examples and in actual use there's very likely to be a much better approach that would be preferable. If the types really are simple enumerations more meaningful than cats and dogs, then deriving Ord, Enum, &c. is preferable to special-purpose stuff. If the intent is a more open-ended way of modelling entities with various properties, it's worth thinking about other designs more tailored to the actual problem.
BigCat :: CatandSmallCat :: Cat, butCat :: Animal. The constructorCathas no relation to the typeCat.