12

In a multi-thread code, if there are several threads trying to send data to a tcp socket at the same time, what will happen? will their data be mixed or will different threads will end up sending data one by one?

2
  • How are you writing to the socket? Are you using write(2) or some C++ abstraction? Commented Oct 30, 2011 at 0:38
  • thanks, please see below for my comments, it's send/sendto for packets ~ 100 bytes Commented Oct 30, 2011 at 0:47

2 Answers 2

11

It depends upon which primitives you're using to submit data to the socket.

If you're using write(2), send(2), sendto(2), or sendmsg(2) and the size of your message is small enough to fit entirely within the kernel buffers for the socket, then that entire write will be sent as a block without other data being interspersed.

If you're using fwrite(3) (or any other higher-level buffered IO abstraction), then there is a chance that your data will be sent without any other data being interspersed, but I would not rely upon this behavior.

I can't speak to sendfile(2) behavior. I'd like to think that the sendfile(2) operation "writes" the entire contents of the file to the socket before any other write(2) requests on the socket, but the documentation I've read doesn't say a word about it, so you better not make the assumption that it is in any sense "atomic".

The safest mechanism is for only a single thread to ever be submitting data to a socket.

Sign up to request clarification or add additional context in comments.

7 Comments

i'm using send/sendto, each time a thread is trying to send some data, the data size is short, about <200 bytes. currently I'm using mutex_lock to sync different threads before calling send, I'm trying to find out if I can remove the mutex_lock to improve performance.
In that case, I expect removing the mutex is probably safe. But be sure to check the send(2) return for -1, errno == EMSGSIZE, in case you're sending data faster than the network can deliver it.
The POSIX docs don't say a lot about thread-safety of these calls, other than the behavior of block/non-blocking IO. There is a posting of a bug in the Linux kernel for multi-threaded calls to write(2) with a file, but I don't know if that's been fixed. But definitely, the syscalls provide a much higher guarantee (implied or not) of atomicity than anything above them.
Correction to my earlier comment: POSIX requires all but a specified set of syscalls to be thread-safe: pubs.opengroup.org/onlinepubs/007908799/xsh/threads.html
PIPE_BUF is only relevant if you write to a pipe. If you're writing to a socket, there's really no guarantee. Use a mutex.
|
0

Indeed as the previous answer says: "The safest mechanism is for only a single thread to ever be submitting data to a socket."

But if you want to have multiple threads call read/write, you must guarantee thread safety yourself. I've written a wrapper around send and recv below that does this for you.

Send() and Recv() are thread safe in the sense that they wont cause a crash, but there is no guarantee that that data wont be "intermixed" with data sent from other threads. Since we most likely don't want that, we must block until all the thread's data is confirmed sent or received, or errored out. Such a long blocking call can be problematic, so make sure you're doing this on threads that can handle long blocking operations.

For linux:

#include <sys/types.h>
#include <sys/socket.h>
#include <map>
#include <mutex>



/* blocks until the full amount of bytes requested are read
 * thread safe
 * throws exception on error */
void recv_bytes(int sock, char* buf, int len, int flags){

    static std::map<int, std::mutex> mtx;

    std::lock_guard<std::mutex> lock(mtx[sock]);

    int bytes_received = 0;

    while (bytes_received != len){

        int bytes = recv(sock, buf + bytes_received, len - bytes_received, flags);

        //error check
        if (bytes == -1){
            throw std::runtime_error("Network Exception");
        }

        bytes_received += bytes;

    } 
}



/* blocks until the full amount of bytes requested are sent
 * thread safe
 * throws exception on error */
void send_bytes(int sock, char* buf, int len, int flags){

    static std::map<int, std::mutex> mtx;

    std::lock_guard<std::mutex> lock(mtx[sock]);

    int bytes_sent = 0; 

    while (bytes_sent != len){

        int bytes_s0 = send(sock, buf, len, flags);

        if (bytes_sent == -1) {
            throw std::runtime_error("Network Exception");
        }

        bytes_sent += bytes_s0;

    }
}

5 Comments

Throwing an exception and using mutex.lock()? If you throw the exception, this code can never run again! Use a lock_guard please
Ah, indeed I missed an unlock. lock_guards are a great invention.
Calling a blocking I/O function while holding a mutex is likely to make your program unresponsive for an extended period under some circumstances. (e.g. if the remote client isn't being responsive, the recv() call might block for a long time, during which the calling thread would of course be blocked, as would any other threads that tried to acquire the mutex)
It will only make the program unresponsive if you don't have a dedicated UI thread.
I understand the hesitation to make these calls block longer than they already do, but OP is asking about multithreaded sockets, and its presumable that he has a dedicated UI thread already.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.