Hello Guys, how are you doing ?
In the last weeks, my openldaps servers were suffering with many come down. I have a peak about the 400 connections per peer ( We have 2 ldap servers ).
Although I has added more memory (4 GB), the problem decreased, but not resolved.
So, How Could I troubleshooting it ?
My enviroment:
- Two Red Hat Enterprise Linux Server release 6.6 (Santiago)
- Two processors and 12 GB of RAM
- Both environments are virtualized.
***************************************
My slapd.conf:
moduleload back_ldap
moduleload translucent
moduleload dynlist
moduleload back_monitor
backend mdb
backend ldap
allow bind_v2
allow bind_anon_dn
database mdb
directory /usr/local/openldap/var/openldap-translucent
suffix "*****"
rootdn "*************"
rootpw ****************************
maxsize 4294967296
sizelimit 100000
overlay translucent
uri "ldap://localhost:1389/"
translucent_bind_local on
translucent_pwmod_local on
translucent_local ******************************
idassert-bind bindmethod=none
overlay dynlist
dynlist-attrset groupOfURLs memberURL member:uniqueMember
**************************
My slapd-syncrepl.conf:
allow bind_v2
allow bind_anon_dn
moduleload syncprov
moduleload dynlist
# Definicoes da base primaria
database mdb
suffix "**"
rootdn "******"
directory /usr/local/openldap/var/openldap-syncrepl
rootpw ******
sizelimit 100000
maxsize 8589934592
overlay dynlist
dynlist-attrset groupOfURLs memberURL member:uniqueMember
loglevel sync stats
idletimeout 0
# ACLs
include /usr/local/openldap/etc/openldap/schema/sicredi.acl
overlay syncprov
# Começo do Consumidor
index entryUUID eq
# syncrepl directives
syncrepl rid=0
provider=ldap://MASTER_Production:389
bindmethod=simple
binddn="************"
credentials=****
searchbase="****"
logbase="cn=accesslog"
logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
type=refreshAndPersist
retry="60 +"
syncdata=accesslog
# Refer updates to the master
updateref ldap://MASTER_Production:389
***** My openldap version: openldap-ltb-2.4.39-1.el6.x86_64
And What I saw in the logs:
Aug 19 09:51:43 ds1openldap2h kernel: slapd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Aug 19 09:51:43 ds1openldap2h kernel: slapd cpuset=/ mems_allowed=0
Aug 19 09:51:43 ds1openldap2h kernel: Pid: 4233, comm: slapd Not tainted 2.6.32-504.23.4.el6.x86_64 #1
Aug 19 09:51:43 ds1openldap2h kernel: Call Trace:
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff810d4241>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81127500>] ? dump_header+0x90/0x1b0
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8122ee7c>] ? security_real_capable_noaudit+0x3c/0x70
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81127982>] ? oom_kill_process+0x82/0x2a0
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff811278c1>] ? select_bad_process+0xe1/0x120
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81127dc0>] ? out_of_memory+0x220/0x3c0
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff811346ff>] ? __alloc_pages_nodemask+0x89f/0x8d0
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8116c9aa>] ? alloc_pages_current+0xaa/0x110
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff811248f7>] ? __page_cache_alloc+0x87/0x90
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff811242de>] ? find_get_page+0x1e/0xa0
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81125897>] ? filemap_fault+0x1a7/0x500
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8114ed04>] ? __do_fault+0x54/0x530
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8114f2d7>] ? handle_pte_fault+0xf7/0xb00
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8114ff79>] ? handle_mm_fault+0x299/0x3d0
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8104d096>] ? __do_page_fault+0x146/0x500
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff81529a1e>] ? thread_return+0x4e/0x7d0
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8153001e>] ? do_page_fault+0x3e/0xa0
Aug 19 09:51:44 ds1openldap2h kernel: [<ffffffff8152d3d5>] ? page_fault+0x25/0x30
Aug 19 09:51:44 ds1openldap2h kernel: Mem-Info:
Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA per-cpu:
Aug 19 09:51:44 ds1openldap2h kernel: CPU 0: hi: 0, btch: 1 usd: 0
Aug 19 09:51:44 ds1openldap2h kernel: CPU 1: hi: 0, btch: 1 usd: 0
Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA32 per-cpu:
Aug 19 09:51:44 ds1openldap2h kernel: CPU 0: hi: 186, btch: 31 usd: 0
Aug 19 09:51:44 ds1openldap2h kernel: CPU 1: hi: 186, btch: 31 usd: 0
Aug 19 09:51:44 ds1openldap2h kernel: Node 0 Normal per-cpu:
Aug 19 09:51:44 ds1openldap2h kernel: CPU 0: hi: 186, btch: 31 usd: 10
Aug 19 09:51:44 ds1openldap2h kernel: CPU 1: hi: 186, btch: 31 usd: 0
Aug 19 09:51:44 ds1openldap2h kernel: active_anon:2571953 inactive_anon:399686 isolated_anon:0
Aug 19 09:51:44 ds1openldap2h kernel: active_file:305 inactive_file:604 isolated_file:0
Aug 19 09:51:44 ds1openldap2h kernel: unevictable:0 dirty:1 writeback:0 unstable:0
Aug 19 09:51:44 ds1openldap2h kernel: free:29811 slab_reclaimable:2307 slab_unreclaimable:7010
Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA free:15276kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14884kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 19 09:51:44 ds1openldap2h kernel: lowmem_reserve[]: 0 3000 12090 12090
Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA32 free:53124kB min:16748kB low:20932kB high:25120kB active_anon:2127764kB inactive_anon:559804kB active_file:0kB inactive_file:152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:128kB shmem:0kB slab_reclaimable:8kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:900kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug 19 09:51:44 ds1openldap2h kernel: lowmem_reserve[]: 0 0 9090 9090
Aug 19 09:51:44 ds1openldap2h kernel: Node 0 Normal free:51016kB min:50752kB low:63440kB high:76128kB active_anon:8160048kB inactive_anon:1038940kB active_file:1224kB inactive_file:2052kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:9308160kB mlocked:0kB dirty:4kB writeback:0kB mapped:1288kB shmem:0kB slab_reclaimable:9220kB slab_unreclaimable:28028kB kernel_stack:1392kB pagetables:44504kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug 19 09:51:44 ds1openldap2h kernel: lowmem_reserve[]: 0 0 0 0
Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA: 1*4kB 1*8kB 2*16kB 2*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 3*4096kB = 15276kB
Aug 19 09:51:44 ds1openldap2h kernel: Node 0 DMA32: 2599*4kB 1163*8kB 354*16kB 75*32kB 28*64kB 15*128kB 9*256kB 12*512kB 9*1024kB 0*2048kB 1*4096kB = 53236kB
Aug 19 09:51:44 ds1openldap2h kernel: Node 0 Normal: 846*4kB 640*8kB 413*16kB 236*32kB 119*64kB 48*128kB 28*256kB 10*512kB 2*1024kB 0*2048kB 0*4096kB = 50760kB
Aug 19 09:51:44 ds1openldap2h kernel: 5598 total pagecache pages
Aug 19 09:51:44 ds1openldap2h kernel: 4605 pages in swap cache
Aug 19 09:51:44 ds1openldap2h kernel: Swap cache stats: add 1173695, delete 1169090, find 121530/123860
Aug 19 09:51:44 ds1openldap2h kernel: Free swap = 0kB
Aug 19 09:51:44 ds1openldap2h kernel: Total swap = 2097148kB
Aug 19 09:51:44 ds1openldap2h kernel: 3145712 pages RAM
Aug 19 09:51:44 ds1openldap2h kernel: 97157 pages reserved
Aug 19 09:51:44 ds1openldap2h kernel: 797 pages shared
Aug 19 09:51:44 ds1openldap2h kernel: 3014707 pages non-shared
Aug 19 09:51:44 ds1openldap2h kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Aug 19 09:51:44 ds1openldap2h kernel: [ 441] 0 441 2795 0 0 -17 -1000 udevd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1251] 0 1251 47346 150 0 0 0 vmtoolsd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1290] 0 1290 23283 37 1 -17 -1000 auditd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1326] 65 1326 107809 573 1 0 0 nslcd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1340] 0 1340 62279 388 1 0 0 rsyslogd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1353] 0 1353 2707 46 0 0 0 irqbalance
Aug 19 09:51:44 ds1openldap2h kernel: [ 1369] 32 1369 4744 15 1 0 0 rpcbind
Aug 19 09:51:44 ds1openldap2h kernel: [ 1380] 81 1380 5881 36 1 0 0 dbus-daemon
Aug 19 09:51:44 ds1openldap2h kernel: [ 1413] 0 1413 1020 0 1 0 0 acpid
Aug 19 09:51:44 ds1openldap2h kernel: [ 1423] 68 1423 10041 151 0 0 0 hald
Aug 19 09:51:44 ds1openldap2h kernel: [ 1424] 0 1424 5100 2 1 0 0 hald-runner
Aug 19 09:51:44 ds1openldap2h kernel: [ 1456] 0 1456 5630 2 1 0 0 hald-addon-inpu
Aug 19 09:51:44 ds1openldap2h kernel: [ 1470] 68 1470 4502 2 1 0 0 hald-addon-acpi
Aug 19 09:51:44 ds1openldap2h kernel: [ 1510] 55 1510 2488944 50131 1 0 0 slapd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1572] 28 1572 241540 244 1 0 0 nscd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1597] 0 1597 16081 20 1 -17 -1000 sshd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1606] 38 1606 6566 72 1 0 0 ntpd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1629] 0 1629 28188 2 0 0 0 abrtd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1637] 0 1637 28131 23 1 0 0 abrt-dump-oops
Aug 19 09:51:44 ds1openldap2h kernel: [ 1651] 0 1651 51804 2959 1 0 0 osad
Aug 19 09:51:44 ds1openldap2h kernel: [ 1661] 0 1661 28742 18 1 0 0 crond
Aug 19 09:51:44 ds1openldap2h kernel: [ 1771] 497 1771 257947 665 1 0 0 icinga2
Aug 19 09:51:44 ds1openldap2h kernel: [ 1795] 0 1795 25232 26 1 0 0 rhnsd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1804] 0 1804 27085 23 0 0 0 rhsmcertd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1818] 0 1818 1016 1 0 0 0 mingetty
Aug 19 09:51:44 ds1openldap2h kernel: [ 1820] 0 1820 1016 1 0 0 0 mingetty
Aug 19 09:51:44 ds1openldap2h kernel: [ 1822] 0 1822 1016 1 0 0 0 mingetty
Aug 19 09:51:44 ds1openldap2h kernel: [ 1824] 0 1824 1016 1 0 0 0 mingetty
Aug 19 09:51:44 ds1openldap2h kernel: [ 1826] 0 1826 1016 1 0 0 0 mingetty
Aug 19 09:51:44 ds1openldap2h kernel: [ 1828] 0 1828 2794 0 1 -17 -1000 udevd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1829] 0 1829 2794 0 1 -17 -1000 udevd
Aug 19 09:51:44 ds1openldap2h kernel: [ 1830] 0 1830 1016 1 0 0 0 mingetty
Aug 19 09:51:44 ds1openldap2h kernel: [21760] 55 21760 4578590 2911874 1 0 0 slapd
Aug 19 09:51:44 ds1openldap2h kernel: [16962] 0 16962 3105 113 1 0 0 nmon_x86_64_rhe
Aug 19 09:51:44 ds1openldap2h kernel: [20314] 0 20314 3059 113 0 0 0 nmon_x86_64_rhe
Aug 19 09:51:44 ds1openldap2h kernel: Out of memory: Kill process 21760 (slapd) score 957 or sacrifice child
Aug 19 09:51:44 ds1openldap2h kernel: Killed process 21760, UID 55, (slapd) total-vm:18314360kB, anon-rss:11646816kB, file-rss:680kB
Thanks for your attention.
--
Atenciosamente,
Édnei Rodrigues