Designing class for handling multiple communication protocols handling

What is the best way to create a generic API for the said functionality, and if possible involve some design pattern?

The Strategy Pattern looks suitable in this scenario.

First, define an interface for all your distinct communication startegies, Communication:

class Communication {
public:
   virtual ~CommunicationStrategy() = default;
   virtual bool sendData(uint8_t* buffer, const size_t& bufferSize) = 0;
   virtual void receiveData(uint8_t* dataBuffer, size_t& bufferSize) = 0;
};

Then, your concrete implementations – i.e., strategies – should derive from this interface:

class EthernetCommunication: public Communication {
public:
   // ...
   bool sendData(uint8_t*, const size_t&) override;
   void receiveData(uint8_t*, size_t&) override;
};

class SerialCommunication: public Communication {
public:
   // ...
   bool sendData(uint8_t*, const size_t&) override;
   void receiveData(uint8_t*, size_t&) override;
};

class CarrierPigeon: public Communication {
public:
   // ...
   bool sendData(uint8_t*, const size_t&) override;
   void receiveData(uint8_t*, size_t&) override;
};

The client code will work with a (pointer to) Communication – i.e., the interface – rather than directly with a particular implementation like EthernetCommunication, SerialCommunication, or CarrierPigeon. Thus, the code follows the "program to an interface, not to an implementation" advice. For example, you may have a factory function like:

std::unique_ptr<Communication> CreateCommunication();

This factory function returns one of the strategies above. Which strategy to return can be determined at run time.

std::unique_ptr<Communication> com = CreateCommunication();
// send data regardless of a particular communication strategy
com->sendData(buffer, bufferSize);

This way, the code above isn't coupled to any particular implementation, but only to the interface Communication which is common to all the different possible communication strategies.

If the different communication strategies don't need per-instance data, just having two callbacks instead of an object will do:

using data_sender_t = bool (*)(uint8_t*, const size_t&);
using data_receiver_t = void (*)(uint8_t*, size_t&);

// set these function pointers to the strategy to use
data_sender_t data_sender;
data_receiver_t data_receiver;