I want to use conntrack to track per-connection bytes, packets etc. on an end-host with a single network interface, rather than connections through a router with multiple interfaces. That means I will track only connections which terminate on the host.
I have set up conntrack on multiple linux distributions, and the answer to conntrack -L is always the same: "0 flow entries have been shown".
Is there any way of tracking per-connection stats in this way using with conntrack or something else on a Linux end-host?
Conntrack will be the best way as it stores tuple for each connection. Moreover load the necessary conntrack module in /lib/modules/xxx/kernel/net/netfilter/yyy.ko and /lib/modules/kernel/xxx/net/ipv4/netfilter/yyyy.ko Though you want connection as you PC as end point Conntrack tool will work if the necessary conntrack module is installed.
Another approach will be write a kernel module and hook it at PRE-ROUTING as you want to catch as end-point and parse the skbs to fullfill you requirements.
Related
Most of the examples provided for sending SNMP traps are simple ones like the one below.
snmptrap -v 1 -c public host TRAP-TEST-MIB::demotraps localhost 6 17 '' \
SNMPv2-MIB::sysLocation.0 s "Just here"
Take any MIB file, they contain many complex object groups, for example, systemGroup contains sysLocation, sysName, etc.
Could someone help in bringing out examples to show the way how to send snmp traps which includes such OBJECT-GROUPS. Adding one more question here, Does SNMPTRAPD support internationalization?
It is really bad practice to define the SNMP notification (trap or inform) the way that it contains the entire OBJECT GROUP or even worse the entire SNMP table. The reason is that you don't really need all these variables anyway. The other reason is that the packet/PDU is limited by MTU size. So it is possible that you'll not be able to send the data within single UDP packet due to its size.
The proper scenario would be to have few varbinds and you could also initiate some polling cycle to find out what happens if you need more details when you receive such trap.
SNMPTRAPD and NET-SNMP library in general do not support internationalization (UNICODE). The library is limited to ASCII charset only.
There are commercial products on the market including NetDecision TrapVision and some other that fully support UTF-8 internationalization.
I have a list of IP addresses or Cisco router. Now i need to find out, which IP address is assigned to which interface (e.g. i have IP 192.168.1.1 and i need to learn that it is IP address of fa0/0 interface). Which MIB can i use to get the list of IP add with corresponding interfaces via SNMP?
Thanks
I believe the SNMP OID you need for this is 1.3.6.1.2.1.4.34, the object being ipAddressTable from the IP-MIB. See the following URL:
http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?local=en&translate=Translate&objectInput=1.3.6.1.2.1.4.34#oidContent
Cisco has very good SNMP info on their site. The SNMP Object Navigator, or the IOS MIB Locator are your friends. Tons of info there, and you can look at it from any angle you want. For example, provide the router's IOS image filename and see what MIBs it supports, etc.
As far as interfaces and IP address info, that's the most basic of stuff so you will be ok, no need to find any weird MIB for that.
My suggestion would be: make sure SNMP is enabled on the router and an SNMP community is set, jump on a Unix/Linux box and point snmpwalk to it and pull all available info. That, paired with the MIB file, is usually the best way to make sure which element you want. If on Windows, there are several free SNMP clients that can "snmpwalk" a device and do the equivalent.
Let me know how that goes!
You can try the OID 1.3.6.1.2.1.4.20 to get the list of IP add with corresponding interfaces via SNMP.
Actually, i had a list of IPs before, i needed just to match them with int names. I did it using 2 MIBs - 1.3.6.1.2.1.4.20.1.2.+IPaddress returns the index of interface and using this index i used 1.3.6.1.2.1.2.2.1.2.+IntfIndex (obtained in previous step). I did it for every IP address in list and works like charm.
There is also the ifxTable which has improved interface speed information for high speed ethernet ports:
http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?local=en&translate=true&objectInput=ifXTable
What are the techniques/algorithms used in WAN optimization? I am looking for a reference which can give a good theory supported with code examples, I have taken a look in Steelhead manual from Riverbed and I found the following main techniques used in:
SDR (Scalable Data Referencing): which breaks up TCP data into
unique data chunks, each chunk has a reference number, where when the
same byte sequence occurs in future transmission, the reference
number is only sent across the LAN instead of raw data chunks.
Connection pooling: The product creates pools of idle TCP
connection (for HTTP as example), where when a client tries to create
a new connection to a previously visited target, it uses one from its
pool, which, in turns, overcomes three-way TCP handshake.
The product reduces the number of round trips over WAN for common
actions (opening/editing remote shared files/folders), it supports
most of intended protocols: CIFS, MAPI, HTTP … etc.
Data compression.
Through my search I found 3 open source projects aim to do WAN optimization, these are:
TrafficSqueezer
WANProxy
OpenNOP
TrafficSqueezer seems to have more features but the comments in its page in sorceforge do not give a good sense about it. I tried to find a document within these projects with good info but I couldn't.
the techniques that can reduce the traffic amount most - are of course compression and data deduplication (both WAN optimisers built up the same data based on a algorithm on memory or HDD - as soon as there is again the same traffic pattern - the pattern is replaced with a pointer to the data and a length - therefore you can save up to 99% when you transfer the same file twice, but even different files have a lot of common data where deduplication can optimise a lot!).
(you will find a lot of sources on the web: e.g. http://www.computerweekly.com/feature/How-data-deduplication-works)
in your example this is technique called SDR.
Riverbed has also a lot of protocol support - which makes for e.g. CIFS, SMB and MAPI more delay aware (e.g. a lot of packages are buffered and sent once - so save roundtrips)
Also F5 does e.g. FTP and HTTP optimisations to get those more performant.
when there is a lot delay on the WAN link - of course you can also save time with connection pooling - so pre-established TCP sessions (you can save the time that would be needed for a tcp 3way handshake)
so at a glance:
-data deduplication
-connection pooling
-compression
-protocol optimisation
i am sure you can find a lot in the f5 doku (F5 WOM is the product), bluecoat does offer WAN optimisation as well and of course Riverbed. also silverpeak might be worth a try.
for the opensouce ones i only have experiences on traffic squeezer, but there hasn't been a comparable feature-set to commercial products this time.
Is there anything in the Ruby library, or as a Ruby gem, that would help me construct a UDP datagram? I've looked into sockets, but there doesn't seem to be a way to simply build one and not send it.
My use case is this:
I need to build a single UDP datagram, and then pass it off to another module which will be responsible for sending it out. Simply put, I just need to be able to specify the src/dst address and port, as well as the payload.
I suppose in the worse case I can build some kind of a struct and fill in respective bits by hand, but it feels like reinventing the wheel as well as a lot of work. The underlying stuff in the sockets API should have something similar that I can make use of, shouldn't it?
The so called socket API is inherently a very low-level C-language API. When you create a UDP socket the connection information is stored in the kernel. You're never "building UDP datagrams", you're writing data into a UDP socket file descriptor, and the physical data packets to be sent over the wire are then constructed inside the kernel.
So yes, if you want a data structure that you can pass around your application that contains a destination address and some data, then you need to create this structure yourself since it doesn't exist by itself anywhere else.
It's not that much work. Just two (or three) data elements. You could use a simple Struct to do the job, unless you need more elaborate functionality and then you just build a normal class.
EDIT
Looks like I misunderstood your question. See my comment below for resources. For example here's some code from the Racket library docs, which probably is closer to what you were looking for:
# tack on UDP
n.l4 = UDP.new
# randomize source port
n.l4.src_port = 1024 + rand(65535-1024)
# take destination port from the commandline
n.l4.dst_port = ARGV[2].to_i
# build a random amount of garbage for the payload
n.l4.payload = Misc.randstring(ARGV[3].to_i)
# fix 'er up (checksum, length) prior to sending
n.l4.fix!(n.l3.src_ip, n.l3.dst_ip)
I have written an application that communicates over TCP using a proprietary protocol. I have a test client that starts a thousand threads that each make requests from the client, and I noticed I can only get about 100 requests/second, even with fairly simple operations. These requests are all coming from the same client so that might be relevant.
I'm trying to understand how I can make things faster. I've read a bit about performance tuning in this area, but I'm trying to understand what I need to understand to performance tune network applications like this. Where should I start? What linux settings do I need to override, and how do I do it?
Help is greatly appreciated, thanks!
Have you considered using asynchronous methods to do the test instead of trying to spawn lots of threads. Each time one thread stops and other starts on the same cpu core, aka. context switching, there can be a very significant overhead. If you want a quick example of networking using asynchronous methods check out networkComms.net and look at how the NetworkComms.ConnectionListenModeUseSync property is used here. Obviously if you're running in linux you would have to use mono to run networkComms.net.
Play around with the Sysctls and Socket Options of the TCP stack: man tcp(7). E.g. you can change the send and receive buffer of tcp or switch NO_DELAY on. Actually to tune the TCP stack itself you should know how TCP works. Things like slow start, congestion control, congestion window etc. But this is related to the transmitting/receiving performance and the buffers maybe with your process handling.
You need to Understand the following Linux utility Command
uptime - Tell how long the system has been running.
uptime gives a one line display of the following information. The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
top provides an ongoing look at processor activity in real time. It displays a listing of the most CPU-intensive tasks on the system, and can provide an interactive interface for manipulating processes
The mpstat command writes to standard output activities for each available processor, processor 0 being the first one. Global average activities among all processors are also reported. The mpstat command can be used both on SMP and UP machines, but in the latter, only global average activities will be printed. If no activity has been selected, then the default report is the CPU utilization report.
iostat - The iostat command is used for monitoring system input/output device
loading by observing the time the devices are active in relation to
their average transfer rates.
vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity. The first report produced gives averages since the last reboot
free - display information about free and used memory on the system
ping : ping uses the ICMP protocol's mandatory ECHO_REQUEST datagram to elicit an ICMP ECHO_RESPONSE from a host or gateway
Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval)
Then you need to know how TCP protocol work, Learn how to identify the Network Latency, where is the Problem, is the problem with ACK,SYCN,ACK SYC, DATA, RELEASE