• RPCOverTCPCapture

Capturing RPC over TCP traffic

This page provides a description of issues in capturing RPC over TCP traffic and tools that I use for it.

The NFS benchmarking project page is at NFSBenchmarking

I can be reached at <shehjart AT gelato DOT unsw DOT edu DOT au>

Issues

I was using nfsdump when I worked on NFS traffic capturing last time around (see TBBT). This time I noticed that Linux implementation can now use TCP as the underlying transport. The problems arise from the fact that TCP is stream-oriented and we use RPC over TCP for NFS. This leads to two problems:

1. It's possible that a single RPC message gets fragmented or segmented over two TCP segments. One of the problems this can cause is the splitting of RPC or NFS headers over two different segments.
2. It is also possible that the RPC headers start from somewhere in the middle of a segment and not right after the TCP header ends.

For a brief write-up on how RPC is encapsulated over TCP, see http://www.cs.helsinki.fi/linux/linux-kernel/2001-38/1106.html

For a brief description on the kind of problem faced, especially when capturing READ and WRITE calls for NFS, see http://www.ethereal.com/lists/ethereal-dev/200106/msg00197.html

Now that we know that, its easy to see why nfsdump cannot work here. nfsdump operates in a stateless manner, processing and interpreting each packet by itself, whereas what we need is a stateful capture tool that knows about the RPC encapsulation and maintains state about a previous segment that contained the RPC frame length field. We can then use this knowledge to perform segment re-assembly. At the same time, we can also use the length field knowledge to determine the bytes in the middle of a segment where one RPC frame ends and another begins. Fortunately for us, this is already implemented in Wireshark and to top it all, in its command line tool tshark. See http://mirror.ethereal.com/lists/ethereal-dev/200304/msg00238.html

Ok, so its called PDU Tracking..hmm.

NOTE: There have been reports of a bug in wireshark TCP re-assembly which prevents it from correctly re-assembling application layer PDUs like RPC payloads, if the Record Marker or the Fragment Header is spread into two TCP segments.

Using tshark for stateful RPC over TCP capture

To use tshark to capture only RPC or NFS packets, we need to know a few things.

Display filter Vs. Capture filter

Wireshark allows us to capture as well as dissect packets in its UI. It uses two types of filters for this purpose. The Display filter allows us to filter the packets that we want to dissect within the UI, regardless of what was actually captured. Capture filter is the filter used for filtering out packets while capturing the traffic from the interfaces. Capture filters are the same as the filter syntax of pcap. In other words:

• Display filters: used for filtering while viewing
• Capture filters: used for filtering while capturing

The relevant difference is that Capture filter syntax does not allow us to capture RPC/NFS traffic, mainly because it requires filtering of traffic above the transport layer. This is one of the major reasons why we cant just use tcpdump for capturing NFS traffic because tcpdump only supports Capture filters or pcap filters. Read about pcap capture filter syntax here: http://www.tcpdump.org/tcpdump_man.html

Wireshark also provides Display filters, for eg, we can type 'rpc' into the Filter text box in its UI to view only the RPC traffic. The best part about tshark is that it allows use of Wireshark's display filters for capturing traffic from the wire. So, if we need just the RPC traffic from the interface all we do is specify 'rpc' as the filter on the tshark command line..neat.

Preference settings for TCP and RPC protocols

Besides the knowledge of filters we need to set configuration options in the wireshark's preferences file. This is generally found as $HOMEDIR/.wireshark/preferences file. Before tshark dumps the traffic, it opens this file to read configuration options. The relevant ones are:

# Whether the RPC dissector should reassemble messages spanning multiple TCP segments.
# To use this option, you must also enable "Allow subdissectors to reassemble TCP streams" in the TCP protocol settings.
# TRUE or FALSE (case-insensitive).
rpc.desegment_rpc_over_tcp: TRUE

#Reassemble fragmented RPC over TCP messages
rpc.defragment_rpc_over_tcp:  TRUE

#Dont allow sub-dissector to re-assemble TCP streams
tcp.desegment_tcp_streams: TRUE

Running tshark

I generally run tshark with the following options. Note that unlike tcpdump, tshark does not take 65536 as the snaplen when -s option is set to 0.

# -R <expr> use Display filter <expr> for capturing traffic
# -i <if> specify interface, same as tcpdump
# -s 65536, be explicit about the size to capture, NOTE: 0 means zero, not 65536, as it is for tcpdump
# -w <path_to_file>, path to dump file, same as tcpdump

$ tshark -i eth0 -s 65536 -w dumpfile

The tshark manual is here: http://www.wireshark.org/docs/man-pages/tshark.html