Setting Receive Packet Steering (RPS) for 32 cores - linux-kernel

Question

I am not sure that I used the correct commands for setting the RPS for a 32 cores machine. This is what I used: echo 1f > /sys/class/net/eth0/queues/rx-0/rps_cpus
Should it be "echo 1f..." or "echo f..." or anything else?

Answer 1

Pinterest engineers just say:
echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus
But there are another their talk:
echo ffff > /sys/class/net/eth0/queues/rx-0/rps_cpus
Can't see the actual difference from kernel point of view but you can try both

Answer 2

Reproducing from the Performance Tuning Guide for Red Hat Enterprise Linux 7:
The rps_cpus files use comma-delimited CPU bitmaps. Therefore, to
allow a CPU to handle interrupts for the receive queue on an
interface, set the value of their positions in the bitmap to 1. For
example, to handle interrupts with CPUs 0, 1, 2, and 3, set the value
of rps_cpus to 00001111 (1+2+4+8), or f (the hexadecimal value for
15).
So for quad-core cpus...
echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus
and for cpus with >4 cores...
echo ff+ > /sys/class/net/eth0/queues/rx-0/rps_cpus
where ff+ is a regex for appending extra f(s) for each set of additional 4 cores

Answer 3

You might want to try using binary calculator and nproc:
echo "obase=16;2^$(nproc)-1" | bc > /sys/class/net/eth0/queues/rx-0/rps_cpus
That formula will configure RPS for any number of cores.

Answer 4

I have seen also <flags>,<flags>,<flags> notation. My machine refuses to accept that.
Other than this I also see that after reboot the values got reset to 0. I bypass it by defining enabling and starting the service that corrects the value to ff (that is what I wanted it to be).

Answer 5

I used this
# Input
NIC_NAME="em1";
if [[ "$1" != "" ]]; then
NIC_NAME="$1";
fi
# Tuna Install
#echo "`date '+%Y-%m-%d %H:%M:%S'` - Install numactl" ;
#yum install -y tuna ;
# Tuna Config
echo "`date '+%Y-%m-%d %H:%M:%S'` - Tuna Info: " ;
tuna --irqs=${NIC_NAME}-\* --cpus=0-3 --move --spread ;
tuna --irqs=${NIC_NAME}-txrx-\* --cpus=0-3 --move --spread ;
tuna --irqs=${NIC_NAME}-tx-\* --cpus=0-3 --move --spread ;
tuna --irqs=${NIC_NAME}-rx-\* --cpus=0-3 --move --spread ;
tuna --irqs=${NIC_NAME}-\* --show_irqs ;
# Install
#echo "`date '+%Y-%m-%d %H:%M:%S'` - Install numactl" ;
#yum install -y numactl ;
#echo "";
# Numa Info
echo "`date '+%Y-%m-%d %H:%M:%S'` - Numa Info: " ;
numactl --hardware | grep "^node . cpus" ;
echo "";
# Numa Mask
echo "`date '+%Y-%m-%d %H:%M:%S'` - Numa Mask: " ;
NUMA_COUNT=$(numactl --hardware | grep "^node . cpus\:" | wc -l);
NUMA_MASK=("");
i=0;
while [ $i -lt ${NUMA_COUNT} ]; do
NUMA_MASK[$i]=$(numactl --hardware | grep "^node $i cpus\:" | awk -F':' '{ print $2}' | sed 's/ /\+2\^/g' | awk '{print "obase=16;0" $0}' | bc | rev | sed -e 's/\([0-F][0-F][0-F][0-F][0-F][0-F][0-F][0-F]\)/\1,/g' | rev);
echo " NUMA_MASK[$i]: ${NUMA_MASK[$i]}";
let i++;
done
echo "";
# Set RPS/XPS
echo "`date '+%Y-%m-%d %H:%M:%S'` - Set RPS/XPS: " ;
QUEUE_RX_COUNT=0;
QUEUE_TX_COUNT=0;
i=0;
while [ $i -lt 16 ]; do
let NUMA_ID=(i % NUMA_COUNT);
QUEUE_RX_RPS="/sys/class/net/${NIC_NAME}/queues/rx-$i/rps_cpus";
QUEUE_TX_XPS="/sys/class/net/${NIC_NAME}/queues/tx-$i/xps_cpus";
if [[ -e "${QUEUE_RX_RPS}" && "${NUMA_MASK[$NUMA_ID]}" != "" ]]; then
echo ${NUMA_MASK[$NUMA_ID]} > "${QUEUE_RX_RPS}" ;
NUMA_MASK_ID=$(cat "${QUEUE_RX_RPS}");
echo " ${QUEUE_RX_RPS}: $NUMA_MASK_ID" ;
let QUEUE_RX_COUNT++;
fi;
if [[ -e "${QUEUE_TX_XPS}" && "${NUMA_MASK[$NUMA_ID]}" != "" ]]; then
echo ${NUMA_MASK[$NUMA_ID]} > "${QUEUE_TX_XPS}" ;
NUMA_MASK_ID=$(cat "${QUEUE_TX_XPS}");
echo " ${QUEUE_TX_XPS}: $NUMA_MASK_ID" ;
let QUEUE_TX_COUNT++;
fi;
let i++;
done
echo "";
# Set Sock Rps Flow
echo "`date '+%Y-%m-%d %H:%M:%S'` - Set Sock Rps Flow: " ;
SOCK_RX_FLOW_ENTRIES="/proc/sys/net/core/rps_sock_flow_entries" ;
SOCK_RX_FLOW_COUNT=32768 ;
echo "${SOCK_RX_FLOW_COUNT}" > ${SOCK_RX_FLOW_ENTRIES} ;
SOCK_RX_FLOW_COUNT=$(cat ${SOCK_RX_FLOW_ENTRIES}) ;
echo " ${SOCK_RX_FLOW_ENTRIES}: ${SOCK_RX_FLOW_COUNT}" ;
echo "";
# Set Queue Rps Flow
echo "`date '+%Y-%m-%d %H:%M:%S'` - Set Queue Rps Flow: " ;
let SET_QUEUE_RX_FLOW_COUNT=(SOCK_RX_FLOW_COUNT / QUEUE_RX_COUNT);
i=0;
while [ $i -lt 16 ]; do
QUEUE_RX_FLOW="/sys/class/net/${NIC_NAME}/queues/rx-$i/rps_flow_cnt";
if [[ -e "${QUEUE_RX_FLOW}" && ${SET_QUEUE_RX_FLOW_COUNT} -gt 0 ]]; then
echo ${SET_QUEUE_RX_FLOW_COUNT} > "${QUEUE_RX_FLOW}" ;
GET_QUEUE_RX_FLOW_COUNT=$(cat "${QUEUE_RX_FLOW}");
echo " ${QUEUE_RX_FLOW}: ${GET_QUEUE_RX_FLOW_COUNT}" ;
fi;
let i++;
done
echo "";