Created attachment 972 [details] Stack trace of segfault I have an openldap 2.6.2 server "ldap1" with the following DIT: dc=example,dc=com (back-mdb) ou=users ou=local cn=admin cn=sync ... ou=remote (back-ldap -> ldaps://dc1.example.com) ... Local user entries are created under subtree "ou=local,ou=users,dc=example,dc=com", and the subtree "ou=remote,ou=users,dc=example,dc=com" is a proxy to an Active Directory server "dc1.example.com" (subtree "ou=users,dc=ad,dc=example,dc=com"). The concrete configuration is as follows: ---------------- dn: olcDatabase={2}ldap,cn=config objectClass: olcDatabaseConfig objectClass: olcLDAPConfig olcDatabase: {2}ldap olcSuffix: ou=remote,ou=users,dc=example,dc=com olcSubordinate: TRUE olcRootDN: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth olcDbURI: ldaps://dc1.example.com olcDbIDAssertBind: bindmethod=simple binddn="cn=aduser,ou=users,dc=ad,dc=example,dc=com" credentials=secret tls_reqcert=demand mode=none olcDbIDAssertAuthzFrom: {0}dn.exact:gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth olcDbIDAssertAuthzFrom: {1}dn.exact:cn=admin,ou=local,ou=users,dc=example,dc=com dn: olcOverlay={0}rwm,olcDatabase={2}ldap,cn=config objectClass: olcOverlayConfig objectClass: olcRwmConfig olcOverlay: {0}rwm olcRwmRewrite: {0}rwm-suffixmassage "ou=users,dc=ad,dc=example,dc=com" olcRwmMap: {0}objectclass inetOrgPerson organizationalPerson olcRwmMap: {1}objectclass posixAccount user olcRwmMap: {2}attribute uid sAMAccountName olcRwmMap: {3}attribute homeDirectory unixHomeDirectory olcRwmMap: {4}attribute ou * olcRwmMap: {5}attribute cn * olcRwmMap: {6}attribute sn * olcRwmMap: {7}attribute givenName * olcRwmMap: {8}attribute mail * olcRwmMap: {9}attribute uidNumber * olcRwmMap: {10}attribute gidNumber * olcRwmMap: {11}attribute * dn: olcDatabase={3}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {3}mdb olcDbDirectory: /var/lib/ldap olcSuffix: dc=example,dc=com olcRootDN: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth olcAccess: {0}to * by dn.exact="cn=admin,ou=local,ou=users,dc=example,dc=com" write by dn.exact="cn=sync,ou=local,ou=users,dc=example,dc=com" write by * break olcAccess: {1}to attrs=userPassword by anonymous auth by self write by * none olcAccess: {2}to * by * read olcDbIndex: objectClass eq,pres olcDbIndex: ou,cn,mail,surname,givenname eq,pres,sub ---------------- So far, so good. A subtree search on "ou=users,dc=example,dc=com" returns both local and remote users. But when I create the second server "ldap2" with similar configuration and configure refreshAndPersist replication, I run into a problem. (1) When I configure on "ldap1" server, ---------------- dn: olcOverlay={0}syncprov,olcDatabase={3}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: {0}syncprov ---------------- and on "ldap2" server, ---------------- dn: olcDatabase={3}mdb,cn=config changeType: modify replace: olcSyncrepl olcSyncrepl: {0}rid=301 provider="ldap://ldap1/" bindmethod=simple binddn="cn=sync,ou=local,ou=users,dc=example,dc=com" credentials=secret searchbase="dc=example,dc=com" type=refreshAndPersist retry="5 12 60 +" timeout=1 ---------------- the initial refresh stage fails. (a) Whith the above configuration, the refresh failes with "(48) Inappropriate authentication", because the bind DN "cn=sync,ou=local,ou=users,dc=example,dc=com" does not have access to the subordinate database. (b) When I add "cn=sync,ou=local,ou=users,dc=example,dc=com" to the ID assertion list on "ldap1" server, ---------------- dn: olcDatabase={2}ldap,cn=config changeType: modify add: olcDbIDAssertAuthzFrom olcDbIDAssertAuthzFrom: {2}dn.exact:cn=sync,ou=local,ou=users,dc=example,dc=com ---------------- the refresh fails with "(12) Critical extension is unavailable", because Active Directory does not support Sync Request Control. (c) Even if the remote server supports Sync Request Control, the refresh fails with the message "server sent multiple refreshDone messages? Ending session". The refreshDone messages are sent twice, one for the sperior databese and the other for the subordinate database. (d) If I delete olcSubordinate attribute and restart slapd on "ldap1" server, ---------------- dn: olcDatabase={2}ldap,cn=config changeType: modify delete: olcSubordinate ---------------- then the refresh stage completes successfully. Once the persistent session is established, I can add olcSubordinate attribute again. ---------------- dn: olcDatabase={2}ldap,cn=config changeType: modify add: olcSubordinate olcSubordinate: TRUE ---------------- When I modify entries in the subordinate database on "ldap1" server, no change notification is sent to "ldap2" server. This is the desired behavior, but if I restart slapd on "ldap1" server, the refresh starts failing again. (2) When I configure the glue overlay explicitly before the syncprov overlay, as described in "man slapd-config", ---------------- dn: olcOverlay={0}glue,olcDatabase={3}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcConfig olcOverlay: {0}glue dn: olcOverlay={1}syncprov,olcDatabase={3}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: {1}syncprov ---------------- the refresh stage completes successfully without attempting to search the subordinate database. This is fine because I do not need to synchronize the subordinate database between "ldap1" and "ldap2" servers. However, when I modify an entry in the subordinate database on "ldap1" server, slapd crashes by segmentation fault. See the attached file for stack trace. After some research, I found that the cause of the crash is as follows: In syncprov_matchops(), it attempts to get the modified entry with DN = op->o_req_ndn. But since op->o_req_ndn has been rewritten in the rmw overlay, glue_back_select() incorrectly selects the mdb backend, which should be the ldap backend. At this point, op->o_bd->be_private holds a value of type ldapinfo_t, but mdb_entry_get() tries to interpret it as type struct mdb_info, causing a segfault. In summary, the problem is: When I configure refreshAndPersist synchronization for a database with a subordinate ldap backend using DN rewriting, (1) The subordinate database cannot be excluded from both refresh and persistent stage of the synchronization: When the glue overlay is not explicitly configured: - In the refresh stage, the subordinate database is included in the search. - In the persist stage, the subordinate database is excluded from the synchronization. When the glue overlay is explicitly configured before the syncprov overlay: - In the refresh stage, the subordinate database is excluded from the search. - In the persist stage, the subordinate database is included in the synchronization. This seems to be inconsistent. (2) If the subordinate database is included in the refresh stage, the refresh fails for one of the following reasons: - the syncrepl user is not allowed to access the subordinate database - the remote server does not support Sync Request Control - multiple refreshDone messages are returned The refresh stage completes successfully if olcSubordinate attribute is deleted from the subordinate database. olcSubordinate attribute can be added again once the persistent session is established, but the refresh stage starts failing again if slapd is restarted. (3) If the subordinate database is included in the persist stage, modifying entries in the subordinate database causes slapd to crash.
Can you provide a self contained way of reproducing this issue? Thanks
Created attachment 976 [details] LDIF files to reproduce the issue I was able to reproduce the issue by creating 3 servers on AWS. (See the attached archive LDIFs.zip) Remote Server (instead of Active Directory) OS: AlmaLinux 9.2 LDAP: openldap-servers-2.6.2-2.el9 from EPEL repository hostname: dc1 config database: dc1-config.ldif mdb database: dc1-mdb.ldif Sync Provider OS: AlmaLinux 9.2 LDAP: openldap-servers-2.6.2-2.el9 from EPEL repository config database: ldap1-config.ldif mdb database: ldap1-mdb.ldif Sync Consumer OS: AlmaLinux 9.2 LDAP: openldap-servers-2.6.2-2.el9 from EPEL repository config database: ldap2-config.ldif (1) In this configuration, the contents on the "ldap1" server are replicated to the "ldap2" server, but the refresh stage fails with the error "(48) Inappropriate authentication." (2) When I add "cn=sync,ou=local,ou=users,dc=example,dc=com" to the ID assertion list on the "ldap1" server, ---------------- dn: olcDatabase={2}ldap,cn=config changeType: modify add: olcDbIDAssertAuthzFrom olcDbIDAssertAuthzFrom: dn.exact:cn=sync,ou=local,ou=users,dc=example,dc=com ---------------- the refresh fails with the error "server sent multiple refreshDone messages? Ending session." If the "dc1" server is an Active Directory, then the error is "(12) Critical extension is unavailable," because the Active Directory does not support the ManageDsaIT control (not the Sync Request Control that I mistakenly mentioned in the previous comment). (3) When I configure the glue overlay explicitly before the syncprov overlay on the "ldap1" server, ---------------- dn: olcOverlay={0}glue,olcDatabase={3}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcConfig olcOverlay: {0}glue dn: olcOverlay={1}syncprov,olcDatabase={3}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: {1}syncprov ---------------- the refresh stage completes successfully without attempting to search the subordinate database. However, when I modify an entry in the subordinate database on "ldap1" server, slapd crashes by segmentation fault.
Hi Hiroyuki, I have now been able to reproduce the crash locally, thanks for the detailed description on how you got there! The issue comes from rwm being asked to rename entries from ou=remote,ou=users,dc=example,dc=com to ou=users,dc=ad,dc=example,dc=com which no longer fits the glued DB's suffix and that confuses back-glue passing its entries to the wrong backend to be released. We might need to wrap the entries returned by glue_entry_get_rw in a way that glue_entry_release_rw can route them back consistently. In the interim you might have to adjust your configuration so that rwm doesn't rename entries outside of their DB's suffix (or have dc=example,dc=com also be a back-ldap proxy to another server where it is stored - this should avoid the crash until this issue is fixed). Regards,
Hi Hiroyuki, I've prepared a patch that should fix the crash you're seeing here: https://git.openldap.org/openldap/openldap/-/merge_requests/649 Can you test it helps in your environment?
Dear Ondřej, Thank you for providing a patch so quickly. I have applied the patch to my openldap 2.6.2 environment, but unfortunately I still get a segfault: -------- [ec2-user@ldap1 ~]$ sudo ldapmodify -Y EXTERNAL -H ldapi:/// <<END dn: cn=Remote User,ou=remote,ou=users,dc=example,dc=com changeType: modify replace: uidNumber uidNumber: 10001 END SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 modifying entry "cn=Remote User,ou=remote,ou=users,dc=example,dc=com" ldap_result: Can't contact LDAP server (-1) -------- The log output is: -------- Aug 25 18:45:35 ldap1 slapd[246667]: conn=1001 fd=16 ACCEPT from PATH=/var/run/ldapi (PATH=/var/run/ldapi) Aug 25 18:45:35 ldap1 slapd[246667]: conn=1001 op=0 BIND dn="" method=163 Aug 25 18:45:35 ldap1 slapd[246667]: conn=1001 op=0 BIND authcid="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" authzid="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" Aug 25 18:45:35 ldap1 slapd[246667]: conn=1001 op=0 BIND dn="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" mech=EXTERNAL bind_ssf=0 ssf=71 Aug 25 18:45:35 ldap1 slapd[246667]: conn=1001 op=0 RESULT tag=97 err=0 qtime=0.000037 etime=0.000111 text= Aug 25 18:45:35 ldap1 slapd[246667]: conn=1001 op=1 MOD dn="cn=Remote User,ou=remote,ou=users,dc=example,dc=com" Aug 25 18:45:35 ldap1 slapd[246667]: conn=1001 op=1 MOD attr=uidNumber Aug 25 18:45:35 ldap1 slapd[246667]: conn=1001 op=1 syncprov_matchops: not recording uuid for dn=cn=Remote User,ou=remote,ou=users,dc=example,dc=com on opc=0x7f0c9c0032e8 Aug 25 18:45:35 ldap1 systemd[1]: slapd.service: Main process exited, code=dumped, status=11/SEGV Aug 25 18:45:35 ldap1 systemd[1]: slapd.service: Failed with result 'core-dump'. -------- And the stack trace is: -------- Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000564e94f4636c in mdb_env_pick_meta (env=0x564e95aad5f0) at back-mdb/./../../../libraries/liblmdb/mdb.c:3944 3944 return metas[ metas[0]->mm_txnid < metas[1]->mm_txnid ]; [Current thread is 1 (Thread 0x7f0cad7fd640 (LWP 246672))] (gdb) bt #0 0x0000564e94f4636c in mdb_env_pick_meta (env=0x564e95aad5f0) at back-mdb/./../../../libraries/liblmdb/mdb.c:3944 #1 mdb_txn_renew0 (txn=txn@entry=0x7f0c9c11e1d0) at back-mdb/./../../../libraries/liblmdb/mdb.c:2688 #2 0x0000564e94f46990 in mdb_txn_begin (env=0x564e95aad5f0, parent=0x0, flags=131072, ret=0x7f0c9c003588) at back-mdb/./../../../libraries/liblmdb/mdb.c:2910 #3 0x0000564e94f6b428 in mdb_opinfo_get (op=op@entry=0x7f0c9c0023f0, mdb=mdb@entry=0x564e95aaed90, rdonly=rdonly@entry=1, moip=moip@entry=0x7f0cad7fa340) at back-mdb/id2entry.c:782 #4 0x0000564e94f72bfd in mdb_entry_get (op=0x7f0c9c0023f0, ndn=0x7f0c9c002438, oc=0x0, at=0x0, rw=0, ent=0x7f0cad7fa638) at back-mdb/id2entry.c:611 #5 0x0000564e94f355af in glue_entry_get_rw (op=0x7f0c9c0023f0, dn=0x7f0c9c002438, oc=0x0, ad=0x0, rw=0, e=0x7f0cad7fa638) at backglue.c:915 #6 0x0000564e94f30ad0 in overlay_entry_get_ov (op=op@entry=0x7f0c9c0023f0, dn=0x7f0c9c002438, oc=oc@entry=0x0, ad=ad@entry=0x0, rw=rw@entry=0, e=e@entry=0x7f0cad7fa638, on=0x564e95ab9620) at backover.c:364 (... the rest is the same as the attachment in my first comment) -------- The segfault occurs in glue_entry_get_rw after the entry in the ldap backend has been modified. At this point, glue_entry_get_rw is called with dn="cn=remote user,ou=users,dc=ad,dc=example,dc=com", and glue_back_select still incorrectly selects the mdb backend. Regards,
Yeah, you're right, there's another bug somewhere in how rwm deals with the response, not renaming the request DN to the original on the way back. Not sure if there's a good way to fix this that could be backported to 2.6 and below. syncprov should also really be able to keep itself to the backend it was configured on so rwm wouldn't be involved at all. I assume that's because of how it handles its "fake" response but I might be wrong here.
On Fri, Aug 25, 2023 at 10:51:43AM +0000, openldap-its@openldap.org wrote: > I have applied the patch to my openldap 2.6.2 environment, but > unfortunately I still get a segfault: Hi Hiroyuki, can you try reconfiguring your database to use back-relay pointing to back-ldap? This might work around most of the confusion that causes rwm+syncprov to interact so badly. Thanks,
The right way around would probably be to use back-meta, which has rewrite functionality built in so syncprov doesn't have to suffer from what rwm has to do to the operation structure when it's configured as an overlay. A brief outline (slapd.conf style because that's easier for intent): database meta suffix ou=remote,ou=users,dc=example,dc=com uri ldaps://dc1.example.com/ou=remote,ou=users,dc=example,dc=com suffixmassage "ou=remote,ou=users,dc=example,dc=com" "ou=users,dc=ad,dc=example,dc=com" idassert ...
Created attachment 978 [details] Patch file Hi Ondřej, I have created a patch, although it is not complete. Could you review it and let me know if there are any problems? (1) When the glue overlay is not explicitly configured, the refresh stage of persistent sync request now completes successfully. I made the following changes for this case: syncprov_search_cb: Set op->o_no_subordinate_glue=1 on completion of search to prevent the glue overlay from processing subordinates. syncprov_op_search: Returns SLAPD_NO_REPLY if a refreshDone intermediate message was successfully sent. glue_op_search: Do not send any response if the search resut for one backend is SLAPD_NO_REPLY. glue_sub_search: Once the search for one backend is complete, remove any callbacks added while processing that backend. With these changes, sync requests will only search the primary backend. This is sufficient for our use case as we do not need to replicate the subordinate database. (2) When the glue overlay is explicitly configured above the syncprov overlay, modifications on the subordinate backend will no longer cause a segfault. The changes for this case are: syncprov_matchops: Saves the request DN before the rwm overlay rewrites it, and use the saved DN to get the entry after the modification. rwm_response: Rewrite back the DN and attributes of the added entry op->ora_e. Although these changes avoid the segfault, the following issues remain: - Modifications on the subordinate backend will result in an error 'got empty or invalid syncUUID' on the consumer, because overlay_entry_get_ov() does not return the entryUUID attribute when the backend is back-ldap. - When the syncprov overlay is enabled on the remote databese, the refresh stage of persistent search never completes if the provider contextCSN is ahead of the consumer contextCSN. This is because an unnecessary Sync Request control is sent to the remote server. - If I use back-relay pointing to back-mdb instead of back-ldap, modifications on the subordinate database are replicated as expected, but the contextCSN is not updated properly. It seems that the syncprov overlay does not work well with the ldap or relay backend. This seems to be a separate issue from the glue and rwm overlay issue. I have confirmed that this patch passes all of the 'make test', but I am not sure if it does not adversely affect other operations... Regards,
Created attachment 979 [details] Modified patch file I made a small change to the patch to properly clean up the callbacks.
On Mon, Sep 11, 2023 at 10:25:27AM +0000, openldap-its@openldap.org wrote: > I made a small change to the patch to properly clean up the callbacks. Hi Hiroyuki, thanks for the patch, I've just started reviewing it. Could you adapt test053-syncprov-glue to test the kinds of issues you encountered or create a new test (possibly based on it) that does? I think the issues you pick up on cut rather deep and it is important that we don't accidentally introduce bugs in this area in the future. Thanks,
Working with your patch further, it seems test018 (and maybe others) are triggering memory issues: e.g. try compiling with `-DSLAP_NO_SL_MALLOC=1` and running `SLAPD_COMMON_WRAPPER=valgrind ./run test018` when in tests/. Looking at valgrind.* files under testrun will show that you might be freeing memory that's still being used later. Unfortunately I'm not sure what your testing configurations were when you saw the issues you're fixing so can't say for sure how the patch should be updated. In general when dealing with memory safety and OpenLDAP, it is useful to build with -DSLAP_NO_SL_MALLOC=1 to make sure the operation's memory pool doesn't hide use-after-free or other issues that might exist/be introduced. If you prefer, you can create a merge request on git.openldap.org and we can move over there. It does make it easier to reference specific parts of the code.
Hi Ondřej, Thank you for pointing out the memory issue. I identified the problem and fixed it. I would like to create a merge request for the fixed patch. I will also create a test for this issue as you suggested. Regards,
On Thu, Dec 14, 2023 at 03:10:38AM +0000, openldap-its@openldap.org wrote: > Thank you for pointing out the memory issue. I identified the problem and fixed > it. > I would like to create a merge request for the fixed patch. I will also create > a test for this issue as you suggested. Hi Hiroyuki, I've just noticed your Gitlab account, please confirm your email address there and ping me when that's done so I can approve the account request. You will be able to create merge requests then. Thanks,
Hi Ondřej, Thank you for approving my Gitlab account. I created a merge request. https://git.openldap.org/openldap/openldap/-/merge_requests/670 I would appreciate it if you could check it out. Thanks,
IPR notice: The attached patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Hiroyuki Homma <homma@allworks.co.jp>. I have not assigned rights and/or interest in this work to any party. I, Hiroyuki Homma, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.
head: • 3cf5db16 by Ondřej Kuzník at 2023-10-09T20:22:42+00:00 ITS#10080 Preserve original choice of backend for entry_release
head: • 9b9469e2 by Ondřej Kuzník at 2023-11-16T16:45:26+00:00 ITS#10080 Fix slapcat when gluing is on • 5cbb29cf by Hiroyuki Homma at 2024-01-11T04:45:08+00:00 ITS#10080 Fix refreshAndPersist synchronization problem with glue + rwm • 212fdc51 by Hiroyuki Homma at 2024-01-11T04:45:08+00:00 ITS#10080 Fix typo in test script and add checks for ADD and MODRDN.
RE26: • 94cc4f34 by Ondřej Kuzník at 2024-01-11T18:00:09+00:00 ITS#10080 Preserve original choice of backend for entry_release • db3d964e by Ondřej Kuzník at 2024-01-11T18:02:01+00:00 ITS#10080 Fix slapcat when gluing is on • 8074a30b by Hiroyuki Homma at 2024-01-11T18:02:07+00:00 ITS#10080 Fix refreshAndPersist synchronization problem with glue + rwm • 238c2907 by Hiroyuki Homma at 2024-01-11T18:02:11+00:00 ITS#10080 Fix typo in test script and add checks for ADD and MODRDN.