Contents
The NFS benchmarking project page is here: NFSBenchmarking
I can be reached at <shehjart AT gelato DOT NO SPAM unsw DOT edu GREEBLIES DOT au>
nfsdump
nfsdump is a NFS traffic capture tool. It was originally hacked from tcpdump by Van Jacobson and others and later modified by Daniel Ellard at Syrah@harvard.
It's available here: http://www.eecs.harvard.edu/sos/software/index.html
Some things that need to be kept in mind while using nfsdump.
- It only outputs the filtered NFS traffic to the standard output. The output is in human-readable format onto the stdout, much like tcpdump.
If used with -w <dumpfile> command line option of tcpdump, it just dumps all packets into the file, including non-NFS traffic. In this regard, it provides the same functionality as the tcpdump -w option. It doesnt provide binary packet dumping.
- Even though nfsdump package comes with a version of the libpcap, if the system has the debian package libpcap-dev installed, the packaged libpcap is not required.
nfsdump, as of now, does not recognize the NFS MOUNT protocol. It prints 'unknown' in the operation field, when it encounters this packet, though all other fields are correct. This is not critical for the file system hierarchy creation step required later in the NFS benchmarking project. Nor does this effect the trace player.
- Specifically for the NFS benchmarking project, the output from the stdout of nfsdump needs to be sent to the anonymize.pl script that comes with the TBBT package. The commands that I use are:
$ nfsdump -i <if> > <trace_file> $ ./anonymize.pl -m <map_base_prefix> < <trace_file> > <trace_file.anon>
- anonymize.pl is described below.
- When used without the -v, i.e. verbose command line parameter, it actually outputs the IP addresses in dotted decimal notation, whereas the default behaviour is to output the address in Hex format. Either output works with the anonymizer and the trace player.
Since, the Linux NFS has now moved to using RPC over TCP, nfsdump cannot be used anymore because its packet dumping facilities are stateless as compared to some stateful packet dissection that is required for extracting RPC messages over TCP stream. This is discussed in detail here: RPCOverTCPCapture
A new generation of tools developed as part of the NFSBenchmarking project use a binary trace format. Processing binary files is faster and the storage space required is much less. For converting nfsdump text-based traces into such a binary format, there's a script called nfsfump2anon which is part of the nfsreplay source package. See nfsreplayUsersGuide for more details.
anonymize.pl
This script is used for anonymizing the NFS traces, to ensure that privacy of the people using the test network and their data is maintained. It also comes packaged with the nfsdump source and resides in the scripts subdir of the nfsdump package.
The original trace is taken as input from the stdin.
The anonymized output is sent to the stdout.
It replaces the required parts of the traces with IDs. Some of which are plain counters.
It requires a command line argument that is a string. It is used as a base prefix for the *.db files it creates. This prefix can also be a directory name. Use the -m <basename> command line option to specify the base name.
It maintains a perl dictionary type called unmapped keywords. The keywords specified in the dictionay are not anonymized. In other words, if some of the fields in the traffic contain these keywords, then they are not replaced with arbit strings. This helps in recognizing access patterns of specific files. Though the dictionary needs to be hard coded in the perl source file. I might change this in the next few days. Probably make it read the dictionary from a runtime config file.
- Not doing this right now, since its not really needed for replaying the traces.
It also outputs a set of *.db files, which are readable by using the Berkeley DB library. It contains mappings of the strings that it replaced in the NFS traffic, in case we need to recover the original trace. The mappings it stores are:
String to GID
String to hosts, IP addresses.
String to filenames or path component.
String to UID.
The database files are not required for FS hierarchy creation.
nfsscan.pl
This script comes as part of the TBBT package. The original paper is here: http://www.eecs.harvard.edu/sos/papers/P149-zhu.pdf
The TBBT tarball is available here: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX <Licensing issues about the source code, still confirming the details, Shehjar>
Confirmed..cannot release the source.
It takes the anonymized traces from the anonymize.pl script as input from he stdin. The script generates all sorts of information. Most of it is required for direct benchmarking using the traces. But some part of the script produces information into files which are used by the rfs.pl script to create a replica filesystem hierarchy of the part of the FS that was accessed on the remote file server in the traces.
It needs a base name as an arg to the -o option so that it can output its results into files with that basename as the prefix.
rfs.pl
This script comes as part of the TBBT package. The rfs.pl script uses the data generated by the nfsscan.pl script to create a replica of the part of file system that was accessed in the traces. This script is present in the same directory as the nfsscan.pl and runs in the same directory to produce a FS hierarchy.
The output filesystem hierarchy is in the directory called RFSFS. This is the directory that needs to be moved to the NFS server which will be profiled. There it will also be needed to be added to the /etc/exports file.
It also produces files named fh-path-map*, of which only one, fh-path-map-play is needed by the TBBT trace player during the replay run. These files contain the mapping of filehandles appearing in the trace to the directories and files which they represent in the anonymised traces.
TBBT Trace player
Finally, the sfs3 executable is the actual trace replay tool. It performs two functions:
- Trace pairing: The trace player needs that in the trace dump provided to it as the input, the replies follow the corresponding requests immediately. This is called pairing the trace, as it involves taking the requests in the order they appear in the anonymised trace dump and for each requests parsing the trace dump to find the reply to it. This is done by using the RPC XID that is part of the trace dump, since each request/reply pair has a unique RPC XID. The pair trace function outputs these into a filename that ends with .pair suffix after the input trace file's name. The pair trace function is executed using the -pair_trace command line argument to the sfs3 executable.
- Trace replay: The trace replay is the main objective of the sfs3 executable. It is completely described in the TBBT paper.
TBBT Trace Player Output Format
A sample line is:
1137589200.673666 125.031c 30.0801 U C3 81a3d2ae 1 getattr fh 010000010009000102 00000074881600e61625c3 euid 18be6 egid 18be1 con = XXX len = XXX
Fields are:
- Timestamp in seconds.usec
- Source IP address (anonymised in our case).port
- Destination IP address.port
- Protocol (U for UDP, T for TCP)
- C (for Call) or R (for reply) followed by the version number.
- the XID
- the opcode
- The string version of hte opcode
- If a response, the string OK or response=XXX where XXX is the actiual response code.
After that the fields depend on the kind of call or response.
Quick Instructions
To start with you need an anonymised trace.
- Process the trace to get a sample file system.
$ nfsscan.pl trace.anonymised $ rfs.pl
- Copy RFSFS to the server.
tar cf - RFSFS | ssh root@server sh -c 'cd /mountpoint; tar xvf -;'
- Arrange for the directory to be exported with appropriate options on the server, e.g.,
echo "/mountpoint *" >> /etc/exports exportfs /mountpoint
- Generate the paired trace file.
sfs3 -pair_trace trace.anonymised
- Replay the trace
sfs3 server:/mountpoint trace.anonymised.pairs fh_map_file <scale> <warmup>where <scale> is a number between 1 and 10000 to say how fast to go, and <warmup> is a number between 0 and 100 to say how long to go for before starting measurements.
