RE: Memory and CPU usage

[Tony Earnshaw <tonni@billy.demon.nl>]

søn, 2003-02-16 kl. 19:47 skrev Howard Chu:

> I see a lot of memory problems with back-bdb when a particular configuration
> is set to allow too many threads. back-bdb uses a lot of memory in each
> thread, much more than back-ldbm.

In my particular configuration I seem to be running (at rest) 7 threads,
each of which (though this is shared memory) is running 4,312 KB shared
memory.

> In general though, you shouldn't be
> configuring a large number of threads; a single CPU can only answer so many
> requests before it gets bogged down in I/O waits anyway.

O.k.

> All the threads in
> the world won't improve your concurrency once your I/O bandwidth (disk and
> network) is saturated. And using too many threads will push you to the
> saturation point sooner, because the increased memory usage will trigger
> excessive paging/swapping (thrashing).

Stupid question (RTFM), but how would I limit the number of threads?
Troubleshooting is obvious, no problem with finding out if the number of
threads is exceeding any limit set.

> Some may be good, but more is not always better.

Threads often have a habit of multiplying if they are not limited to any
specific value.

> (And I still believe that a
> good async I/O API obviates the need for threads. So whether the situation is
> "good" to begin with is still debatable...) Threads don't come for free, and
> today they're not even "cheap".

What's with "today?" RAM expensive? Or processor level 2 cache?

> As a sysadmin you need to understand the
> limits of the machines you're working with.

Absolutely.

> A particular CPU has only so much
> on-chip Level 1 cache, some amount of Level 2 cache, page tables, VM
> lookaside tables, etc...

O.k.

> When you spawn more execution paths than your CPU's
> hardware has cache/table space for, you destroy the CPU performance. When you
> spawn so many threads that all of the free RAM in your address space is
> consumed, slapd aborts because it cannot allocate any more dynamic memory for
> its normal operation. This is a strong message to you, the administrator,
> that your hardware is inadequate to support your configuration.

As far as T.'s "(?=undefined)" goes, where are we at here? Where does
the "(?=undefined)" come from? From your explanation, it could be easy
to deduce why his configuration didn't work. Did the "(?=undefined)"
have anything to do with it? It doesn't seem so, to me.

Best,

Tony

-- 

Tony Earnshaw

When you rob a person of his illusions,
you are robbing him of his happiness


e-post:		tonni@billy.demon.nl
www:		http://www.billy.demon.nl