rdma-core rsocket server receiver connect fork a new sub process, send return 13 permission denied - fork

Question

Now I am testing rdma-core with fork.
In server I use rselect and raccept to receive rsocket connection from client.
Server can rrecv the msg from client successfully.
But server cannot rsend to client. It return -1 and errno 13 permission denied.
THe same code without rsocket, it will send OK.
For rdma-core/librdmacm rsocket, how support fork?
int main(int argc,char *argv[])
{
int ret;
int server_sockfd, client_sockfd;
int server_len, client_len;
struct sockaddr_in server_address;
struct sockaddr_in client_address;
memset(&server_address,0,sizeof(server_address));
memset(&client_address,0,sizeof(client_address));
int result;
fd_set readfds, testfds;
int maxfd;
int on=1;
server_sockfd = rsocket(AF_INET, SOCK_STREAM, 0);
setsockopt(server_sockfd, SOL_SOCKET, SO_BROADCAST, &on, sizeof(on));
server_address.sin_family = AF_INET;
server_address.sin_port = htons(8888);
server_address.sin_addr.s_addr = htonl(INADDR_ANY);
ret = rbind(server_sockfd,(struct sockaddr *)&server_address,sizeof(server_address));
printf("rbind retunr %d\n", ret);
ret = rlisten(server_sockfd,5);
printf("rlisten retunr %d\n", ret);
//set fd_set
FD_ZERO(&readfds);
FD_SET(server_sockfd,&readfds);
maxfd = server_sockfd + 1;
while(1)
{
char ch;
int fd,i;
int nread;
testfds = readfds;
result = rselect(FD_SETSIZE, &testfds, NULL, NULL, NULL);
printf("rselect retunr %d\n", result);
if(result < 1)
{
printf("server5\n");
exit(1);
}
for( fd = 1; fd < maxfd; fd++)
{
if(FD_ISSET(fd,&testfds))
{
if(fd == server_sockfd)
{
memset(&client_address,0,sizeof(client_address));
client_len = sizeof(client_address);
client_sockfd = raccept(server_sockfd,(struct sockaddr *)&client_address,&client_len);
if(fork()==0)
{
setsockopt(client_sockfd, SOL_SOCKET, SO_BROADCAST, &on, sizeof(on));
rclose(server_sockfd);
ret = rrecv(client_sockfd,&ch,1,0);
printf("recv from client : %c, ret %d\n",ch, ret);
sleep(2);
ch++;
ret = rsend(client_sockfd,&ch,1,0);
printf("rsend to client : %c, ret %d, erro %d\n",ch, ret, errno);
}
}
}
}
}
}

Answer 1

As you can see from the rsocket man page:
The preload library can be used by setting LD_PRELOAD when running. Note that not all applications will work with rsockets. Support is limited based on the socket options used by the application. Support for fork() is limited, but available. To use rsockets with the preload library for applications that call fork, users must set the environment variable RDMAV_FORK_SAFE=1 on both the client and server side of the connection. In general, fork is supportable for server applications that accept a connection, then fork off a process to handle the new connection.
As far as I can see, to use rsockets with fork, you have to:
Use sockets API in both your client and your servers (rather than using rsocket calls directly).
Run both the client and your server with these environment variables: RDMAV_FORK_SAFE=1 LD_PRELOAD=/usr/lib/x86_64-linux-gnu/rsocket/librspreload.so (or change the path according to where librspreload.so is installed on your system).
Without using LD_PRELOAD it seems like fork is not supported, meaning that a child process can't use rsockets that the parent has created.