I realize that this is an old question, but I think I have some insight on this. I recently developed some embedded prototypes for a project I was working on and needed a way for these device to communicate at high speeds to my PC. USB was quite a headache, and frankly, so is writing a custom tcpip stack although I have at least done that before. In addition, the embedded device had no way to get an ip address from the user other than hardcoding it into the c/assembly. So I thought what if I could connect over ethernet and just avoid all the IP layer nonsense. I decided to add magnetics to the board and bitbang 10Mbps Ethernet/MAC frames across the channel. On the PC side, I used libPCap to read out the raw ethernet frames and process them in my application. Worked like a charm for creating a high speed connection between my embedded project, and my pc. Well except there is a catch. It worked when connected to my pc over a crossover cable or through a dummy switch, but if you tried to connect it to the rest of the LAN through a traditional router, the router would discard the packet. Why? I never found out for sure, but my best guess is that the EtherType field in an Ethernet Frame which is used to denote the protocol used in the upper layer did not match the one of the known types (i.e. IP, ARP, etc).
The bottom line is while you can make a custom link layer protocol and implement it application side with a library like libPCap, its not guaranteed to work over all routers since some routers will drop your packets if they don't recognize the protocol you are using. It will definitely not work over the internet since the IP layer is what decides routing on the internet.
foobar://for a project I am working on. Any dot com would procedural generate a 3D model on my project.