( home )
 Last Modified:  24 May 2001
CPU OVERHEAD,
Interrupt Driven


 
Notes after initial post: a)  The below timing relies on getrusage and gettimeofday for percent user information; both of these utilities have time resolution down to only 10 microseconds on aztec.  The tests have been reperformed using more accurate methods:  fixed time and fixed work.  See  CPU Overhead II and  CPU Overhead III , respectively.  The second page has a short discussion comparing these two methods.
b)  The shape and orientation of the below graph is to be expected based on information in   "constant cost."   The magnitude of the curve has not been cross checked.
c) The below discussion mentions unexpected spikes to 100% user time; the spikes occured at the beginning or in the midst of performing timing runs.   Subsequently, some comments in the existing Myrinet Control Program were found that stated to recheck the value when setting a signal interrupt bit.  The comment stated that, depending on traffic levels, the bit may not be set every time code attempts to set it.  The MCP provides a "_gm_interrupt" function that ensures the bit is set.  The below SSL code changes were modified accordingly and the spiking never recurred during CPU Overhead II testing.  It is suspected that, prior to this change, the MCP would end up in an infinite loop waiting for an interrupt to be cleared by the host that was never "sent" in the first place.
PURPOSE: As the rate of interrupts to the CPU increases, one expects that a running process will be given less and less CPU time.  The CPU Overhead test provides a view into the relationship between  interrupt frequency and cost to a running process.
HISTORY: The code written to measure CPU usage relative to interrupt frequency was developed within SSL.  The code includes changes to commercial code:  Myrinet Control Program and GM driver.  The changes to commercial code evolved from similar changes written to explore  interrupt latency .  The code for monitoring CPU usage was written and runs independently.
UNIT MEASURE: This test measures the percent of a process's CPU time that is given to the user.  For purposes of this test, the user is the program which monitors CPU usage.  The percentage of process time allowed to the user is determined by  100 * (process user time) /  [(process user time) + (process system time)]. 
USE: The modified Myrinet Control Program  and GM driver  are installed using the ./GM_INSTALL script in the binary directory; see changes flagged with "sslchange."  During installation, terminal output indicates when the driver is installed and an "SSL" interrupt flag is included in the terminal output.  The flag indicates that the series of interrupts has commenced.  The timing program is then executed by the tester and completes in about a minute and prints out the results.  A typical table of results are shown here. (37 RTC Units Plus - see below).  Note that the table indicates the percent of CPU time that went to the process (i.e., see the second to last column "PROC/CLOCK").  For test results validation, this value should be at or near 100% or else it may be the case that interrupts are being handled (i.e., not measured) outside of the timing process.  To achieve these results, the results below were obtained with the host disconnected from the departments network and without other actively running user processes.
INTERNALS: During installation of the MCP, the MCP sends an initial interrupt.  The driver responds by storing the interrupt parameters in shared memory.  The MCP picks up the interrupt parameters which consist of interrupt period and duration values.  The duration value is simply how long the MCP will continue sending  interrupts.  The period value is the number of MCP RTC units that the MCP will wait before sending each interrupt.  The duration and frequency of the interrupts are passed this way so that those variables may be changed in the driver without having to then recompile the MCP; the test machine compiled the MCP in about thirtyfive minutes while the driver compiled in less than a minute.

Once the MCP confirms that the first interrupt was received, the interrupt stream begins as follows.  The MCP runs through the specified period of RTC units and then sends an interrupt to the host.  Once the MCP has confirmation that the interrupt was received, the MCP waits another period of RTC units before sending the next interrupt and the cycle repeats.

The timing program first makes a trial timing run and adjusts an iteration parameter to achieve the desired timing length.  Subsequent timing iterations are accomplished without making system calls; this way, if there are no interrupts then the time used by the process is pure user time.  The timing program assumes that the interrupt stream from the Myrinet card is of long enough duration for the timing program to complete.  The effect of interrupts is indicated by decreased user time and increased system time for the process.  User time and system time for the timing process are obtained from the getrusage library function.  The test uses this data and clock data to make the measurement as well as to monitor what percentage of all CPU time is given to the process.   See figures below.

PLATFORMS: While the test is extensible x86 platforms running Linux and having a Myrinet card, the setup for the test is relatively time consuming if the card was not previously in use on the host.  Setup involves installing a cross-compiler in order to compile the code for the Myrinet Control Program; installation takes into account specific host parameters such as the size of various types.
SAMPLE DATA: See the figues below.  Each plotted point is based on the average of fifteen timing runs with each run consisting of four seconds.  Two Myrinet network cards were used during tests.  The first figure contains data from both Myrinet cards.  The second and third figures depict the same data on a per card basis.  This data was gathered on the same host machine.  Card and host information is appended below the figures. 

Not that the abscissa units are in card RTC Units Plus.  This reflects that the MCP waited the specified RTC units before sending an interrupt and then waited for confirmation before starting the cycle again.  In other words, the "Plus" is the interrupt latency period as seen by the Myrinet card.  So, the zero point on the abscissa represents an interrupt frequency period equivalent to actual latency period during the test.

POSSIBLE FUTURE
MODIFICATIONS:
If the number of interrupts handled by the kernel were measured during a test period then the average cost in time to the user for an interrupt could be calculated from the test results.
NOTES: During timing runs, the user percent would sometimes unreasonably spike to 100%; these spikes are not shown below.  This typically occured when either RTC Units Plus was set at the extreme low end (i.e., < 50) or the test had just been previously run several times.  If RTC Units Plus was not at the extreme low end then powering down the host for a period of time and reperforming the test would resolve the problem.  Occasionally, the spikes would occur in the middle of a timing run, as though the hardware and/or software reached a saturation point over time. 


 
 


 
 

CARD AND HOST INFORMATION:   The tests were run on an FIC VA-503+ motherboard, 500 MHz AMD-K6(tm)-2 processor, and Red Hat 6.2 with Linux Kernel 2.2.14-5.0.  Both Myrinet cards had the following information on them:
 
ON CHIPS
ON BOARD
ON STICKER
LANai 7.2 9947
PCIDMA 1.0 9947
Myrinet M2L-PCI64/2-3.0
1999 Myricom Inc.
357957
M2L-PCI64A-2
40681
A-3935  3952

The two cards are distinguishable by a colon-separated number on the sticker:  00:60:dd:7f:cd:23 on the "first" card and 00:60:dd:7f:cd:21 on the "second" card.



Bill Lawry