Full_Name: Howard Chu Version: 2.5 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (80.233.42.124) Submitted by: hyc IDLs used for index slots are hardcoded to a max of 65536 elements in the DB and twice that in memory. Sites with larger DBs tend to need larger IDLs; if they're running with sufficient memory it ought to be possible to configure larger IDLs.
in master
changed notes changed state Open to Test moved from Incoming to Software Enhancements
--On Friday, February 15, 2019 2:37 PM +0000 hyc@openldap.org wrote: > Full_Name: Howard Chu > Version: 2.5 > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (80.233.42.124) > Submitted by: hyc > > > IDLs used for index slots are hardcoded to a max of 65536 elements in the > DB and twice that in memory. Sites with larger DBs tend to need larger > IDLs; if they're running with sufficient memory it ought to be possible > to configure larger IDLs. Per the documentation, this new setting appears to be a new type of "global" option that affects all MDB databases in the configuration. This new setting, per the documentation, must be configured in a "backend mdb" section. However, there appear to be some problems with the implmentation. Issue #1: "backend mdb" requires a "directory" directive Since this is a new "global" concept, "directory" should not be required (and in fact, seems problematc). Issue #2: If a "directory" statement is supplied (see above), slapd core dumps when attempting to convert this configuration to cn=config: The slapd.conf file I'm using to test is: include /usr/local/etc/openldap/schema/core.schema include /usr/local/etc/openldap/schema/cosine.schema include /usr/local/etc/openldap/schema/inetorgperson.schema modulepath /usr/local/lib64/openldap moduleload back_mdb backend mdb idlexp 32 database mdb directory /var/openldap-data suffix dc=example,dc=com index default eq index objectClass (The directory line location was moved to the "backend" section after initial testing due to issue #1). --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
--On Monday, May 06, 2019 9:28 PM +0000 quanah@symas.com wrote: To further document the issues after further testing with OpenLDAP master as of 6/25/2019 a) If a value for idlexp is set in a slapd.conf file that is not allowed (< 16 or > 31), converting to cn=config using slaptest will incorrectly report that the target "slapd.d" directory doesn't exist, like: /usr/local/etc/openldap# /usr/local/sbin/slaptest -f slapd.conf -F /tmp/slapd.d slaptest: bad configuration directory! ls -ld /tmp/slapd.d drwxr-xr-x 2 root root 40 Jun 25 17:59 /tmp/slapd.d b) If the idlexp value is corrected to be within the allowed range, the converted cn=config database loses the backend configuration and the idlexp setting: /usr/local/sbin/slaptest -f slapd.conf -F /tmp/slapd.d config file testing succeeded cd /tmp/slapd.d/ find . -type f | xargs grep -i idlexp ./cn=config/cn=schema.ldif:olcAttributeTypes: ( OLcfgBkAt:12.1 NAME 'olcBkMdbIdlExp' DESC 'Power of 2 u ./cn=config/cn=schema.ldif: onfiguration' SUP olcBackendConfig STRUCTURAL MAY olcBkMdbIdlExp ) No olcBackend... file exists, etc. Suggested remedies: a) The man page documentation be updated to note the limitations on valid ranges for the idlexp setting (16<=x<=31), at least for 64-bit systems. b) slaptest provides a valid error if the idlexp setting is not within the valid range (as opposed to complaining the target directory does not exist, when in fact it does). c) That conversion from slapd.conf to cn=config be fixed so that it works. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
--On Tuesday, June 25, 2019 7:07 PM +0000 quanah@symas.com wrote: > c) That conversion from slapd.conf to cn=config be fixed so that it works. I'm not entirely sure that the fix that went in for this in ec411582d663667d6b638162db51dfa70f5263d3 is correct. Specifically, unlike everything else, there is no associated weight. If in the future other backends (such as SQL, LDAP, etc) support server-wide settings via an olcBackend configuration, the weight would need to exist. root@anvil4:/tmp/q/slapd.d/cn=config# ls -l total 64 -rw------- 1 root root 448 Jun 27 12:19 'cn=module{0}.ldif' drwxr-x--- 2 root root 4096 Jun 27 12:19 'cn=schema' -rw------- 1 root root 39646 Jun 27 12:19 'cn=schema.ldif' -rw------- 1 root root 435 Jun 27 12:19 'olcBackend=mdb.ldif' -rw------- 1 root root 596 Jun 27 12:19 'olcDatabase={-1}frontend.ldif' -rw------- 1 root root 584 Jun 27 12:19 'olcDatabase={0}config.ldif' -rw------- 1 root root 859 Jun 27 12:19 'olcDatabase={1}mdb.ldif' I.e., I was rather expecting: olcBackend={1}mdb.ldif or similar. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
Quanah Gibson-Mount wrote: > --On Tuesday, June 25, 2019 7:07 PM +0000 quanah@symas.com wrote: > >> c) That conversion from slapd.conf to cn=config be fixed so that it works. > > > I'm not entirely sure that the fix that went in for this in ec411582d663667d6b638162db51dfa70f5263d3 is correct. Specifically, unlike everything else, there is > no associated weight. If in the future other backends (such as SQL, LDAP, etc) support server-wide settings via an olcBackend configuration, the weight would > need to exist. > > root@anvil4:/tmp/q/slapd.d/cn=config# ls -l > total 64 > -rw------- 1 root root 448 Jun 27 12:19 'cn=module{0}.ldif' > drwxr-x--- 2 root root 4096 Jun 27 12:19 'cn=schema' > -rw------- 1 root root 39646 Jun 27 12:19 'cn=schema.ldif' > -rw------- 1 root root 435 Jun 27 12:19 'olcBackend=mdb.ldif' > -rw------- 1 root root 596 Jun 27 12:19 'olcDatabase={-1}frontend.ldif' > -rw------- 1 root root 584 Jun 27 12:19 'olcDatabase={0}config.ldif' > -rw------- 1 root root 859 Jun 27 12:19 'olcDatabase={1}mdb.ldif' > > > I.e., I was rather expecting: > > olcBackend={1}mdb.ldif > > or similar. No, because order is irrelevant for these. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
--On Thursday, June 27, 2019 8:35 PM +0000 hyc@symas.com wrote: > No, because order is irrelevant for these. Cool, thanks! I'll continue on with deeper testing then. :) --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
--On Thursday, June 27, 2019 8:56 PM +0000 quanah@symas.com wrote: > --On Thursday, June 27, 2019 8:35 PM +0000 hyc@symas.com wrote: > >> No, because order is irrelevant for these. > > Cool, thanks! I'll continue on with deeper testing then. :) Given the current implementation of OpenLDAP, this feature is impossible to use w/o recompiling OpenLDAP when a change to the IDL size is made. This is because LDAP_PVT_THREAD_STACK_SIZE must be adjusted as well and that requires a recompile. The default size for LDAP_PVT_THREAD_STACK_SIZE is: ( 1 * 1024 * 1024 * sizeof(void *) ) which works for an IDL size of 16 (2^16) which is 65536. If you change the IDL size, say to 22, then the new IDL size is: 4,194,304. We then use this difference to find the offset we need to adjust LDAP_PVT_THREAD_STACK_SIZE by: 4194304/65536 = 64 So it needs to be 64 time larger: ( 64 * 1024 * 1024 * sizeof(void *) ) Generally, this feature is simply unusuable (currently) as a tunable given the requirement for recompiling OpenLDAP to use it. --QUanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
Needs more work as per last comment
Pull out documentation for feature for 2.5 since it is not currently workable. Fully fix in 2.6 so this can be configurable.
MR for 2.5 specific doc cleanup: https://git.openldap.org/openldap/openldap/-/merge_requests/230
Commits: • a40f6bff by Quanah Gibson-Mount at 2021-02-18T18:47:40+00:00 ITS#8977 - Remove documentation for idlexp The idlexp feature depends on additional work that is not yet done. Remove documentation for the feature
(In reply to Quanah Gibson-Mount from comment #8) > --On Thursday, June 27, 2019 8:56 PM +0000 quanah@symas.com wrote: > > > --On Thursday, June 27, 2019 8:35 PM +0000 hyc@symas.com wrote: > > > >> No, because order is irrelevant for these. > > > > Cool, thanks! I'll continue on with deeper testing then. :) > > Given the current implementation of OpenLDAP, this feature is impossible to > use w/o recompiling OpenLDAP when a change to the IDL size is made. This > is because LDAP_PVT_THREAD_STACK_SIZE must be adjusted as well and that > requires a recompile. > > The default size for LDAP_PVT_THREAD_STACK_SIZE is: > > ( 1 * 1024 * 1024 * sizeof(void *) ) > > which works for an IDL size of 16 (2^16) which is 65536. > > If you change the IDL size, say to 22, then the new IDL size is: 4,194,304. > We then use this difference to find the offset we need to adjust > LDAP_PVT_THREAD_STACK_SIZE by: > > 4194304/65536 = 64 > > So it needs to be 64 time larger: > > ( 64 * 1024 * 1024 * sizeof(void *) ) > > > Generally, this feature is simply unusuable (currently) as a tunable given > the requirement for recompiling OpenLDAP to use it. I believe the above explanation is incorrect, so want to update it. 1024*1024^2 = 1,048,576 This appears to work with an idlexp of 20, which is also 1,048,576 So really what is required is that the LDAP_PVT_THREAD_STACK_SIZE_VALUE total needs to match the 2^idlexp value. So, 2^22 is 4,194,304, so in this case, the LDAP_PVT_THREAD_STACK_SIZE_VALUE would need to be recompiled as ( 4 * 1024 * 1024 * sizeof(void *) )
(In reply to Quanah Gibson-Mount from comment #8) > --On Thursday, June 27, 2019 8:56 PM +0000 quanah@symas.com wrote: > > > --On Thursday, June 27, 2019 8:35 PM +0000 hyc@symas.com wrote: > > > >> No, because order is irrelevant for these. > > > > Cool, thanks! I'll continue on with deeper testing then. :) > > Given the current implementation of OpenLDAP, this feature is impossible to > use w/o recompiling OpenLDAP when a change to the IDL size is made. This > is because LDAP_PVT_THREAD_STACK_SIZE must be adjusted as well and that > requires a recompile. This is actually not correct. The stack size was only required because IDLs were of constant size and some were allocated on the stack. But with this patch, since IDL sizes are variable they must all be dynamically allocated using ch_malloc, and so has no impact on the stack any more.
tested with 2.5.2, and indeed this works now. Will revert my prior commit.
Commits: • fc0cb887 by Quanah Gibson-Mount at 2021-03-02T19:56:51+00:00 Revert "ITS#8977 - Remove documentation for idlexp" This reverts commit a40f6bff894d02603d74eef35bd1a7ae1993537e.