You want to use TypeFamilies for this to create an associated type:
{-# LANGUAGE TypeFamilies, TypeSynonymInstances, FlexibleInstances #-}
class Vector v where
-- Declares a family of types called Item, parametrized on the
-- instance v of Vector, and the kind of Item v must be *,
-- meaning that it must be a type, not a type constructor
-- (e.g. Maybe Int :: * vs Maybe :: * -> *)
type family Item v :: *
dot :: v -> v -> Item v
...
instance (Num a) => Vector (V3 a) where
type Item (V3 a) = a
dot (a, b, c) (d, e, f) = a*d + b*e + c*f
...
Then you can do
> dot (1, 2, 3) ((4, 5, 6) :: V3 Int)
32
Although I would recommend against using a type synonym instance, you'd be better off using a data type:
data V3 a = V3 a a a deriving (Eq, Show)
instance Functor V3 where
fmap f (V3 a b c) = V3 (f a) (f b) (f c)
instance (Num a) => Vector (V3 a) where
type Item (V3 a) = a
(V3 a b c) `vplus` (V3 d e f) = V3 (a + d) (b + e) (c + f)
vnegate v = fmap negate v
dot (V3 a b c) (V3 d e f) = a*d + b*e + c*f
This helps out the type checker a lot, in particular it means you wouldn't need the explicit type signature above. It also means your inferred types won't be (a, a, a) but V3 a (like when you see [Char] instead of String), which is easier to follow. It's not crucial, but helpful.
In case you're wondering, this is how GHC.Exts.IsList (for use with the new OverloadedLists extension) does it:
class IsList l where
type family GHC.Exts.Item l :: *
fromList :: [GHC.Exts.Item l] -> l
fromListN :: Int -> [GHC.Exts.Item l] -> l
toList :: l -> [GHC.Exts.Item l]