This is how I would write it (I have put everything in a self-contained gist in case you want to play with it). The State Int monad handles the threading of the counter, the Maybe alternative picks the right value. I think that the algorithm is more readable this way.
We start with a bunch of imports and your definition of Tree:
{-# LANGUAGE GADTs #-}
module NthElement where
import Data.Functor
import Control.Applicative
import Control.Monad
import Control.Monad.Trans.State
data Tree a where
Nil :: Tree a
Node :: Ord a => Tree a -> a -> Tree a -> Tree a
Then we arrive at the main definition: getNthElement runs the stateful computation and if we got Nothing back, it puts the number of visited nodes (original number minus the final Int state), otherwise it gives back the value it found.
getNthElement :: Int -> Tree a -> Either Int a
getNthElement k t = maybe (Left $ k - s) Right mv where
(mv , s) = go t `runState` k
go :: Tree a -> State Int (Maybe a)
The stateful computation itself is a direct translation of your high-level description of the algorithm:
If we find a Nil, there is no value to return (so we fail)
go Nil = return Nothing
If we find a Node then the nth value we are looking for is either in the right subtree, or in the middle if the counter is 0 after exploring the right subtree, or somewhere in the left subtree:
go (Node l v r) = do
rv <- go r
k <- get
() <- put $ k - 1
lv <- go l
return $ rv <|> (v <$ guard (k == 0)) <|> lv
With this definition it's not immediately clear that we are never doing work that is not needed (we do write lv <- go l when it might be the case that the value was already found!). However lazyness is on our side and to convince ourselves that it is indeed ok, we can define a left-infinite tree and observe that getNthElement always returns a value:
leftInfTree :: Tree Int
leftInfTree = foldr (\ k ih -> Node ih k Nil) Nil [1..]
Test in ghci:
*NthElement> getNthElement 10 leftInfTree
Right 11