[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: CPU scaling

Howard Chu wrote: 
Has anyone got a dual or quad socket Intel Xeon based server for testing? I've
been testing on two AMD systems, one quad socket dual core and one dual socket
quad core. There are a lot of different ways to tune these systems...

Thanks to Matt Ezell at the University of Tennessee for loaning me some of his servers over the holidays. He gave me access to a Dell 2950 with a pair of Intel Xeon 5345s (2.2GHz quad core processors) for testing. The test results are available here

http://connexitor.com/blog/pivot/entry.php?id=191#body

(My apologies to the researchers running their simulations on the machines we used for load generators. Java can be pretty unforgiving as a resource hog...)

slapd currently uses a single listener thread and a pool of some number of
worker threads. I've found that performance improves significantly when the
listener thread is pinned to a single core, and no other threads are allowed
to run there. I've also found that performance improves somewhat when all
worker threads are pinned to specific cores, instead of being free to run on
any of the remaining cores. This has made testing a bit more complicated than
I expected.

The Intel system behaved differently (of course). In general, the system delivered best performance when the listener thread was free to be assigned to any core of the active set. However, when testing with 6 or 7 cores, performance was extremely erratic, and only stabilized when the listener thread was pinned to the first CPU socket.

I originally was just pinning the entire process to a set number of cores
(first 1, then 2, incrementing up to 8) to see how performance changed with
additional cores. But due to the motherboard layout and the fact that the I/O
bridges are directly attached to particular sockets, it makes a big difference
exactly which cores you use.

I custom built a kernel (2.6.24-rc3 with Jeff Garzik's patched device drivers) with Intel IO/AT DMA support, which seemed to allow the ethernet driver a lot more leeway. Instead of the ethernet driver consuming a single CPU core, its cycles were evenly distributed across all 8 cores. A pity there's no such DMA engine on the AMD systems. (Without this DMA engine, the ethernet driver also consumed a single core on the Intel system.)

Another item I noticed is that while we scale perfectly linearly from 1 core
to 2 cores in a socket (with a dual-core processor), as we start spreading
across multiple sockets the scaling tapers off drastically. That makes sense
given the constraints of the Hypertransport connections between the sockets.

Note that on the 4P AMD system, two of the processors have unused Hypertransport links. It would definitely improve things if the two CPUs were connected by that extra link; IIRC some other 4P AMD systems do that.

On the quad-core system we scale pretty linearly from 1 to 4 cores (in one
socket) but again the improvement tapers off drastically when the 2nd socket
is added in.

Well. The AMD quad-core Opteron certainly scales much better than the other two systems.

I don't have any Xeon systems to test on at the moment, but I'm curious to see
how they do given that all CPUs should have equal access to the northbridge.
(Of course, given that both memory and I/O traffic go over the bus, I'm not
expecting any miracles...)

As expected, the Xeon's multithreaded/multicore performance is hampered by its FSB architecture.

The quad-core system I'm using is a Supermicro AS-2021M-UR+B; it's based on an
Nvidia MCP55 chipset. The gigabit ethernet is integrated in this chipset.
Using back-null we can drive this machine to over 54,000
authentications/second, at which point 100% of a core is consumed by interrupt
processing in the ethernet driver. The driver doesn't support interrupt
coalescing, unfortunately. (By the way, that represents somewhere between
324,000pps and 432,000pps. While there's only 5 LDAP packets per transaction,
some of the client machines choose to send separate TCP ACKs, while others
don't, which makes the packet count somewhere between 5-8 packets per
transaction. I hadn't taken those ACKs into account when I discussed these
figures before. At these packet sizes (80-140 bytes), I think the network
would be 100% saturated at around 900,000pps.)

For these interfaces and interrupt rates, with such small packets, 50% utilization really isn't that bad.

Interestingly, while 2 cores can get over 13,000 auths/second, and 4 cores can
get around 25,000 auths/second (using back-hdb), with all 8 cores it's only
peaking at 29,000 auths/second. This tells me it's better to run two separate
slapds in a mirrormode configuration on this box (4 cores per process) than to
run a single process across all of the cores. Then I'd expect to hit 50,000
auths/second total, pretty close to the limits of the ethernet device/driver.

50K/sec is definitely out of reach given how many CPU cycles the ethernet driver consumes. But still the quad-core Opteron delivers amazing performance, especially given the fact that its core clock speed is so much slower than the other systems'. Just goes to show, in a database-oriented workload, memory bandwidth rules.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/