Full_Name: Gerry Calderhead Version: OpenLDAP 2.4.25 OS: Linux RHEL5.5 URL: http://uploading.com/files/529d29ce/0001-MemberOf-AutoGroup-together-can-leave-stale-memberof.patch/ Submission from: (NULL) (80.238.1.135) What we're trying to do ----------------------- We are using OpenLDAP + GOsa to manage users who have access to an extranet. There are a set of tools on the extranet, let's call then VCS, Bugs, Wiki. The extranet is accessibly by third-parties. We want to delegate some admin to staff in the third parties. The structure we want to use is: Multiple organisations Each organisation has a group per tool, e.g. VCS_access, Bugs_Access, Wiki_Access. Organisation admins can add people to their own groups. We then want to aggregate all people with access to those tools into some top level groups, e.g. all_VCS_access = org1.VCS_access + org2.VCS_access all_Bugs_access = org1.Bugs_access + org2.Bugs_access all_Wiki_access = org1.Wiki_access + org2.Wiki_access It's important to us that when people are added/removed to the organisational groups that the group 'member' attributes and the user's 'memberof' attributes are correct and consistent as these for the basis for our access management. Observed Problem ----------------- If you add a user to a monitored group they are consistently added to the target group. You can now search for them using a memberof filter. If you then remove the user from a monitored group they are consistently removed from the target group. However if you then search for members of the target group using a memberOf filter the removed users always show up. 100% reproducible in our environment. Analysis -------- When using bother AutoGroup + MemberOf the "memberof" attributes are not always deleted when users are removed from a group. The approach this overlay takes to updating the target members list, when a member of a monitored list is removed, is to basically clear the membership and re-populate based on a search. Generally the Autogroup Entry requiring a refresh will be flagged in autogroup_modify_entry function. Within this response function we will then spot these flagged refreshes and will perform the deletes and searches to re-polulate the groups. So far, so simple. However, this response function is not necessarily executing in response context of the original modification (say of a "member") but could be in response to a different modification, specifically the addition of a "memberOf" attribute. i.e. it behaves asynchronously - refreshing the group sometime after the initial change. This is a problem when you wish to use MemberOf and AutoGroup together. 1. Admin removes a "member" of a group 2. AutoGroup spots the removal from a trigger group and flags a refresh on the AutoGroup Entry for the target group. 3. MemberOf overlay's _modify function is called, spots the removal of the "member" and updates the "memberOf" attribute of the trigger group. 4. This response function is called in the context of "memberOf" update 5. We spot the flag on the AGE, and delete all "member"s from the target and initiate searches to re-populate. 6. The MemberOf overlay's _modify function is called in response to the "member" changes in the target group (5). 8. MemberOf's _modify function is guarded against recursion, and exits immediately without removing the "memberof" attributes from the users in the target group. 9. When the search completes all remaining "member"'s will be re-added to the target group. The users will have their "memberOf"'s updated but the chance to remove the dead "memberOf"'s is gone. These will now linger around in the DB. So, the simplest solution I can think of is that we do not make any changes in the context of a modification that may have originated from the MemberOf overlay and instead wait on the next opportunity to trigger the refreshes. In reality if the trigger for the refresh was a "member" change then the refresh will happen during the course of the corresponding _response's - just not on any which modify "memberof". Fix ---- Add the following lines at the start of autogroup_response() if ( op->o_tag == LDAP_REQ_MODIFY ) { Modifications *mz; for ( mz = op->orm_modlist; mz; mz = mz->sml_next ) { if ( mz->sml_mod.sm_desc == agi->agi_memberof_ad ) { Debug(LDAP_DEBUG_TRACE,"==> autogroup_response in context of 'memberof' modification, do nothing!\n", 0, 0, 0); return SLAP_CB_CONTINUE; } } } Full git-format-patch present at URL specified. sha1sum 0001-MemberOf-AutoGroup-together-can-leave-stale-memberof.patch 76c2cf268fff37c0cd744e7b3cfd0afa63d9e991 0001-MemberOf-AutoGroup-together-can-leave-stale-memberof.patch Public Domain Notice -------------------- I, Gerry Calderhead, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.
Note the "Analysis" section is a copy/paste of the comment from the code. When it says "this response function" it means the function: autogroup_response. G On 11 June 2011 21:24, <openldap-its@openldap.org> wrote: > > *** THIS IS AN AUTOMATICALLY GENERATED REPLY *** > > Thanks for your report to the OpenLDAP Issue Tracking System. Your > report has been assigned the tracking number ITS#6970. > > One of our support engineers will look at your report in due course. > Note that this may take some time because our support engineers > are volunteers. They only work on OpenLDAP when they have spare > time. > > If you need to provide additional information in regards to your > issue report, you may do so by replying to this message. Note that > any mail sent to openldap-its@openldap.org with (ITS#6970) > in the subject will automatically be attached to the issue report. > > mailto:openldap-its@openldap.org?subject=(ITS#6970) > > You may follow the progress of this report by loading the following > URL in a web browser: > http://www.OpenLDAP.org/its/index.cgi?findid=6970 > > Please remember to retain your issue tracking number (ITS#6970) > on any further messages you send to us regarding this report. If > you don't then you'll just waste our time and yours because we > won't be able to properly track the report. > > Please note that the Issue Tracking System is not intended to > be used to seek help in the proper use of OpenLDAP Software. > Such requests will be closed. > > OpenLDAP Software is user supported. > http://www.OpenLDAP.org/support/ > > -------------- > Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved. > >
In further testing in our environment we've found this patch to be ineffective. We're seeing lockups consistently when changing other attributes. I'll follow up with more details and some traces shortly (miight try with tip first though :0) G
Hi, i'm also experiencing slapd deadlocks/lockups when using memberOf & autogroup, w/ 2.4.31. Both modules/overlays seem to step on each other's toes when updating an entry. Not 100% sure if it's the same issue as the OP was experiencing though. My config is the following : - i have a ou=people branch with inetOrgPerson objects, each one having an organization attr. - i have a ou=groups branch with groupOfNames objects, each populated by autogroup by using a labeledURI doing a filter on the organization attr (among other conditions) of the inetOrgPerson objects. - and finally, i have memberOf overlay that updates the inetOrgPerson objects with the final groups membership. When modifiying the organization attribute of an inetOrgPerson (and thus triggering an update of groups & membership at the same time), slapd deadlocks. It happens both if the inetOrgPerson gets out of a group, or is now a new member of a group (only showing the tail of the debug log, but i can provide it all): 51bb233b oc_check_required entry (uid=person,ou=people,dc=myorg,dc=fr), objectClass "inetOrgPerson" 51bb233b oc_check_allowed type "cn" 51bb233b oc_check_allowed type "structuralObjectClass" 51bb233b oc_check_allowed type "entryUUID" 51bb233b oc_check_allowed type "creatorsName" 51bb233b oc_check_allowed type "createTimestamp" 51bb233b oc_check_allowed type "objectClass" 51bb233b oc_check_allowed type "uid" 51bb233b oc_check_allowed type "givenName" 51bb233b oc_check_allowed type "sn" 51bb233b oc_check_allowed type "userPassword" 51bb233b oc_check_allowed type "title" 51bb233b oc_check_allowed type "employeeType" 51bb233b oc_check_allowed type "roomNumber" 51bb233b oc_check_allowed type "mail" 51bb233b oc_check_allowed type "telephoneNumber" 51bb233b oc_check_allowed type "memberOf" 51bb233b oc_check_allowed type "o" 51bb233b oc_check_allowed type "entryCSN" 51bb233b oc_check_allowed type "modifyTimestamp" 51bb233b oc_check_allowed type "modifiersName" 51bb233b => entry_encode(0x00000006): 51bb233b <= entry_encode(0x00000006): At that point, there are three active threads in gdb: ^C Program received signal SIGINT, Interrupt. 0x00007ffff5c67e75 in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0 (gdb) info thread Id Target Id Frame 3 Thread 0x7ffff0f53700 (LWP 16492) 0x00007ffff5c6b2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 2 Thread 0x7ffff1754700 (LWP 16491) 0x00007ffff59b10d3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 * 1 Thread 0x7ffff7ebb700 (LWP 16490) 0x00007ffff5c67e75 in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0 thread 1 & 2 seem okay... (gdb) bt #0 0x00007ffff5c67e75 in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ffff7f0fca9 in slapd_daemon () at ../../../../servers/slapd/daemon.c:2930 #2 0x00007ffff7ef687c in main (argc=<optimized out>, argv=<optimized out>) at ../../../../servers/slapd/main.c:1012 (gdb) thread 2 [Switching to thread 2 (Thread 0x7ffff1754700 (LWP 16491))] #0 0x00007ffff59b10d3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007ffff59b10d3 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff7f0d721 in slapd_daemon_task (ptr=<optimized out>) at ../../../../servers/slapd/daemon.c:2540 #2 0x00007ffff5c66b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007ffff59b0a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () but thread 3 seems to lock with himself, maybe trying to modify the same entry via autogroup & memberof at the same time ? (gdb) thread 3 [Switching to thread 3 (Thread 0x7ffff0f53700 (LWP 16492))] #0 0x00007ffff5c6b2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 (gdb) bt #0 0x00007ffff5c6b2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ffff74fe9ab in __db_pthread_mutex_lock () from /usr/lib/x86_64-linux-gnu/libdb-5.1.so #2 0x00007ffff758d5ca in __lock_get_internal () from /usr/lib/x86_64-linux-gnu/libdb-5.1.so #3 0x00007ffff758e9b3 in __lock_vec () from /usr/lib/x86_64-linux-gnu/libdb-5.1.so #4 0x00007ffff758ef18 in __lock_vec_pp () from /usr/lib/x86_64-linux-gnu/libdb-5.1.so #5 0x00007ffff32234a0 in hdb_cache_entry_db_relock (bdb=bdb@entry=0x7ffff83226a0, txn=<optimized out>, ei=0x7ffff82db7b0, rw=rw@entry=1, tryOnly=tryOnly@entry=0, lock=lock@entry=0x7ffff0f4fc70) at cache.c:198 #6 0x00007ffff3223eb3 in hdb_cache_modify (bdb=bdb@entry=0x7ffff83226a0, e=e@entry=0x7ffff85045c8, newAttrs=0x7ffff8517f48, txn=<optimized out>, lock=lock@entry=0x7ffff0f4fc70) at cache.c:1231 #7 0x00007ffff321255a in hdb_modify (op=0x7ffff0f503b0, rs=0x7ffff0f501d0) at modify.c:711 #8 0x00007ffff7f801c6 in overlay_op_walk (op=op@entry=0x7ffff0f503b0, rs=0x7ffff0f501d0, which=op_modify, oi=0x7ffff8325c90, on=0x0) at ../../../../servers/slapd/backover.c:671 #9 0x00007ffff7f8031b in over_op_func (op=0x7ffff0f503b0, rs=<optimized out>, which=<optimized out>) at ../../../../servers/slapd/backover.c:723 #10 0x00007ffff29dabb9 in memberof_value_modify (op=op@entry=0x7ffff0f50d40, ndn=<optimized out>, ad=0x7ffff82d3420, old_dn=old_dn@entry=0x0, old_ndn=old_ndn@entry=0x0, new_dn=new_dn@entry=0x7ffff0f50d68, new_ndn=new_ndn@entry=0x7ffff0f50d78) at ../../../../../servers/slapd/overlays/memberof.c:411 #11 0x00007ffff29db548 in memberof_res_modify (op=0x7ffff0f50d40, rs=<optimized out>) at ../../../../../servers/slapd/overlays/memberof.c:1457 #12 0x00007ffff7f227d6 in slap_response_play (op=op@entry=0x7ffff0f50d40, rs=rs@entry=0x7ffff0f50cd0) at ../../../../servers/slapd/result.c:507 #13 0x00007ffff7f22d6e in send_ldap_response (op=op@entry=0x7ffff0f50d40, rs=rs@entry=0x7ffff0f50cd0) at ../../../../servers/slapd/result.c:582 #14 0x00007ffff7f23816 in slap_send_ldap_result (op=0x7ffff0f50d40, rs=0x7ffff0f50cd0) at ../../../../servers/slapd/result.c:860 #15 0x00007ffff3211e67 in hdb_modify (op=0x7ffff0f50d40, rs=0x7ffff0f50cd0) at modify.c:749 #16 0x00007ffff7f801c6 in overlay_op_walk (op=op@entry=0x7ffff0f50d40, rs=0x7ffff0f50cd0, which=op_modify, oi=0x7ffff8325c90, on=0x0) at ../../../../servers/slapd/backover.c:671 #17 0x00007ffff7f8031b in over_op_func (op=0x7ffff0f50d40, rs=<optimized out>, which=<optimized out>) at ../../../../servers/slapd/backover.c:723 #18 0x00007ffff2be449f in autogroup_add_member_to_group (op=op@entry=0x7ffff82d9ab0, dn=dn@entry=0x7ffff82d9ad8, ndn=ndn@entry=0x7ffff82d9ae8, age=age@entry=0x7ffff82f8ed0) at autogroup.c:146 #19 0x00007ffff2be6aa2 in autogroup_response (op=0x7ffff82d9ab0, rs=<optimized out>) at autogroup.c:1302 #20 0x00007ffff7f7f598 in over_back_response (op=0x7ffff82d9ab0, rs=0x7ffff0f52a50) at ../../../../servers/slapd/backover.c:237 #21 0x00007ffff7f227d6 in slap_response_play (op=op@entry=0x7ffff82d9ab0, rs=rs@entry=0x7ffff0f52a50) at ../../../../servers/slapd/result.c:507 #22 0x00007ffff7f22d6e in send_ldap_response (op=op@entry=0x7ffff82d9ab0, rs=rs@entry=0x7ffff0f52a50) at ../../../../servers/slapd/result.c:582 #23 0x00007ffff7f23816 in slap_send_ldap_result (op=0x7ffff82d9ab0, rs=0x7ffff0f52a50) at ../../../../servers/slapd/result.c:860 #24 0x00007ffff3211e67 in hdb_modify (op=0x7ffff82d9ab0, rs=0x7ffff0f52a50) at modify.c:749 #25 0x00007ffff7f801c6 in overlay_op_walk (op=op@entry=0x7ffff82d9ab0, rs=0x7ffff0f52a50, which=op_modify, oi=0x7ffff8325c90, on=0x0) at ../../../../servers/slapd/backover.c:671 #26 0x00007ffff7f8031b in over_op_func (op=0x7ffff82d9ab0, rs=<optimized out>, which=<optimized out>) at ../../../../servers/slapd/backover.c:723 #27 0x00007ffff7f2a569 in fe_op_modify (op=0x7ffff82d9ab0, rs=0x7ffff0f52a50) at ../../../../servers/slapd/modify.c:303 #28 0x00007ffff7f2c42d in do_modify (op=0x7ffff82d9ab0, rs=0x7ffff0f52a50) at ../../../../servers/slapd/modify.c:177 #29 0x00007ffff7f12961 in connection_operation (ctx=ctx@entry=0x7ffff0f52ba0, arg_v=arg_v@entry=0x7ffff82d9ab0) at ../../../../servers/slapd/connection.c:1150 #30 0x00007ffff7f12c84 in connection_read_thread (ctx=0x7ffff0f52ba0, argv=<optimized out>) at ../../../../servers/slapd/connection.c:1286 #31 0x00007ffff7a73ff3 in ?? () from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 #32 0x00007ffff5c66b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #33 0x00007ffff59b0a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #34 0x0000000000000000 in ?? () If i examine the three hdb_modify() calls in that trace, the one in #7 and the one in #24 try to work on the same entry (dummy value is the modified inetOrgPerson), and the one in #15 modifies the groupOfNames from which the inetOrgPerson is a new member. I suppose it shouldn't happen... -- Landry Breuil Mouton a 5 pattes du CRAIG
Hi, after being adviced to try with mdb backend instead, which doesnt use locking, slapd exploded as soon as i had enabled the memberOf & autoGroup overlays and one client tried to connect to the server: 51c1b560 => mdb_search 51c1b560 mdb_dn2entry("dc=myorg") 51c1b560 => mdb_dn2id("dc=myorg") 51c1b560 <= mdb_dn2id: got id=0x1 51c1b560 => mdb_entry_decode: 51c1b560 <= mdb_entry_decode Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffb1fd3700 (LWP 15475)] autogroup_response (op=0x7ffff82d52b0, rs=0x7fffb1fd2a50) at autogroup.c:927 927 autogroup_def_t *agd = agi->agi_def; (gdb) bt #0 autogroup_response (op=0x7ffff82d52b0, rs=0x7fffb1fd2a50) at autogroup.c:927 #1 0x00007ffff7f7f598 in over_back_response (op=0x7ffff82d52b0, rs=0x7fffb1fd2a50) at ../../../../servers/slapd/backover.c:237 #2 0x00007ffff7f227d6 in slap_response_play (op=op@entry=0x7ffff82d52b0, rs=rs@entry=0x7fffb1fd2a50) at ../../../../servers/slapd/result.c:507 #3 0x00007ffff7f25261 in slap_send_search_entry (op=0x7ffff82d52b0, rs=0x7fffb1fd2a50) at ../../../../servers/slapd/result.c:1008 #4 0x00007ffff3213600 in mdb_search (op=0x7ffff82d52b0, rs=<optimized out>) at ../../../../../servers/slapd/back-mdb/search.c:852 #5 0x00007ffff7f801c6 in overlay_op_walk (op=op@entry=0x7ffff82d52b0, rs=0x7fffb1fd2a50, which=op_search, oi=0x7ffff82e6e50, on=0x0) at ../../../../servers/slapd/backover.c:671 #6 0x00007ffff7f8031b in over_op_func (op=0x7ffff82d52b0, rs=<optimized out>, which=<optimized out>) at ../../../../servers/slapd/backover.c:723 #7 0x00007ffff7f15641 in fe_op_search (op=0x7ffff82d52b0, rs=0x7fffb1fd2a50) at ../../../../servers/slapd/search.c:402 #8 0x00007ffff7f14f06 in do_search (op=0x7ffff82d52b0, rs=0x7fffb1fd2a50) at ../../../../servers/slapd/search.c:247 #9 0x00007ffff7f12961 in connection_operation (ctx=ctx@entry=0x7fffb1fd2ba0, arg_v=arg_v@entry=0x7ffff82d52b0) at ../../../../servers/slapd/connection.c:1150 #10 0x00007ffff7f12c84 in connection_read_thread (ctx=0x7fffb1fd2ba0, argv=<optimized out>) at ../../../../servers/slapd/connection.c:1286 #11 0x00007ffff7a73ff3 in ?? () from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 #12 0x00007ffff5c66b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #13 0x00007ffff59b0a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #14 0x0000000000000000 in ?? () (gdb) p agi $1 = (autogroup_info_t *) 0x0 So mdb doesnt seem to be an option here... -- Landry Breuil Mouton a 5 pattes du CRAIG
breuil@craig.fr wrote: > after being adviced to try with mdb backend instead, which doesnt use > locking, slapd exploded as soon as i had enabled the memberOf & > autoGroup overlays and one client tried to connect to the server: The last release you mentioned in this ITS was 2.4.31. Is that the version you're still using? You should not use back-mdb with this release. Use at least 2.4.35 or better current RE24 from git repo. http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=tree;h=refs/heads/OPENLDAP_REL_ENG_2_4;hb=refs/heads/OPENLDAP_REL_ENG_2_4 Ciao, Michael.
Hi, The last comments on this ITS mentioned crashes or deadlocks when using autogroup and memberof together. I tried this today with git master on both mdb and hdb backends. Adding a new entry that triggers an autogroup member to be added: I don't get a crash or deadlock, but the new entry doesn't have a memberOf attribute. It looks like the internal Modify adding the member happens before the Add completes, so memberof tries to add memberOf to an entry that doesn't exist yet. Changing autogroup to trigger on an alteration to an existing entry, for example addition of a specific attribute or auxiliary objectClass: with mdb this works and the memberOf attribute is added; with hdb I get the same deadlock Landry Breuil already reported. Looks like a recursive deadlock on the same thread. I think those problems are both different from the one this ITS is about, though. Maybe they should be filed separately. I also confirmed the problem originally reported on current git master, and the analysis sounds right, but I haven't had time to think through the proposed solution fully. I'm not totally sure whether it's autogroup or memberof that actually needs fixing. Maybe both. cheers, Ryan
As I understand it, Gerry's use case is: we have some groups, we want an autogroup containing the union of their members, and we want memberOf to accurately reflect both the manual and automatic group memberships. I think that's a reasonable thing to want; especially since the memberOf part is impossible with dynlist. Config for these tests: dyngroup schema with groupOfURLs modified to allow 'member' as per autogroup README, plus: dn: olcOverlay={1}memberof,olcDatabase={1}mdb,cn=config objectClass: olcMemberOf olcMemberOfGroupOC: groupOfURLs dn: olcOverlay={2}autogroup,olcDatabase={1}mdb,cn=config objectClass: olcAutomaticGroups olcAGattrSet: groupOfURLs memberURL member On Wed, Jun 4, 2014 at 12:24 PM, Ryan Tandy <ryan@nardis.ca> wrote: > Adding a new entry that triggers an autogroup member to be added: I > don't get a crash or deadlock, but the new entry doesn't have a > memberOf attribute. It looks like the internal Modify adding the > member happens before the Add completes, so memberof tries to add > memberOf to an entry that doesn't exist yet. slapadd: dn: cn=autogroup,dc=example,dc=com objectClass: groupOfURLs memberURL: ldap:///dc=example,dc=com??one?(objectClass=account) and then ldapadd: dn: uid=user,dc=example,dc=com objectClass: account uid: user The new entry doesn't gain a memberOf attr as expected. > Changing autogroup to trigger on an alteration to an existing entry, > for example addition of a specific attribute or auxiliary objectClass: > with mdb this works and the memberOf attribute is added; with hdb I > get the same deadlock Landry Breuil already reported. Looks like a > recursive deadlock on the same thread. slapadd: dn: cn=autogroup,dc=example,dc=com objectClass: groupOfURLs memberURL: ldap:///dc=example,dc=com??one?(objectClass=extensibleObject) dn: uid=user,dc=example,dc=com objectClass: account and then ldapmodify: dn: uid=user,dc=example,dc=com add: objectClass objectClass: extensibleObject under mdb, works as expected. under hdb, deadlock. > I think those problems are both different from the one this ITS is > about, though. Maybe they should be filed separately. > > I also confirmed the problem originally reported on current git master I came up with two ways to do this. I don't know which one Gerry used. You can use the same config as above and have autogroup expand attr values. The initial LDIF is: dn: cn=autogroup,dc=example,dc=com objectClass: groupOfURLs memberURL: ldap:///cn=staticgroup,dc=example,dc=com?member?base?(objectClass=*) dn: cn=staticgroup,dc=example,dc=com objectClass: groupOfNames member: cn=dummy dn: uid=user,dc=example,dc=com objectClass: account (You can't re-use groupOfURLs for the static group because autogroup blocks manual changes to its members.) Then ldapmodify: dn: cn=staticgroup,dc=example,dc=com add: member member: uid=user,dc=example,dc=com dn: cn=staticgroup,dc=example,dc=com delete: member member: uid=user,dc=example,dc=com Alternatively you can make autogroup search by memberOf. For that you need olcMemberOfGroupOC: groupOfNames (instead of GroupOfURLs). Then you do the same slapadd and ldapmodify, except with memberURL: ldap:///dc=example,dc=com??one?(memberOf=cn=staticgroup,dc=example,dc=com) AFAICT both approaches work (personally I prefer the first...), but memberOf is only applied to a single OC. If you want it for both, you might try the same thing with a second memberof instance, like this: dn: olcOverlay={1}memberof,olcDatabase={1}mdb,cn=config objectClass: olcMemberOf olcMemberOfGroupOC: groupOfURLs dn: olcOverlay={2}memberof,olcDatabase={1}mdb,cn=config objectClass: olcMemberOf olcMemberOfGroupOC: groupOfNames dn: olcOverlay={3}autogroup,olcDatabase={1}mdb,cn=config objectClass: olcAutomaticGroups olcAGattrSet: groupOfURLs memberURL member Then you get the problem Gerry reported: memberOf is added twice but only deleted once, the second value remains. Same result for both memberURL configurations. Minor nitpicks: if you give autogroup's olcAGattrSet exactly two arguments, slapadd crashes in ag_cfgen (autogroup.c:1761); and if you omit the filter from memberURL, i.e. memberURL: ldap:///cn=staticgroup,dc=example,dc=com?member?base? then slapd crashes in autogroup_modify_entry (autogroup.c:1381) while refreshing memberships. Ideally those would return an error instead of segfaulting. I'm also wondering, should autogroup/README perhaps recommend groupOfNames and labeledURI instead of groupOfURLs and member? hope this helps, Ryan
Back in June, Gerry replied to me privately. Copying his thoughts here, with his permission: On Sat, Jun 14, 2014 at 8:13 AM, Gerry Calderhead <gerry@everythingsucks.co.uk> wrote: > Hi, > > It's been a while since I looked at this - in fact since I originally > reported it. > In the end I concluded it was too much work for me to continue investigating > and re-implemented autogroup using an external perl script > and removed it from my build of openldap. > > My conclusion at the time, IIRC, was that the overlay architecture was a bit > broken and needed a re-think, specifically to avoid the need to be reentrant > - ultimately, by making the overlay CB's asynchronous. > > I think I had in mind something like: > create a thread local holding a notification queue > a _modify() call would add a notification to the queue for each overlay > The stack would unwind to some point (perhaps overlay is wrapped for > this?), and there the CBs would start to happen for overlays > As the overlays do their writes more notifications are added the queue > The cycle repeats until the overlays stop figging with stuff and the > queue empties > > The reason for the thread local was to avoid changing the overlay API > > Even with this approach there is still the possibility to get overlays > caught in an infinite loop - where overlay1 makes a change, > triggers overlay2 which makes a change and triggers overlay1 and so on...
changed notes changed state Open to Test moved from Incoming to Contrib
ryan@nardis.ca wrote: > As I understand it, Gerry's use case is: we have some groups, we want > an autogroup containing the union of their members, and we want > memberOf to accurately reflect both the manual and automatic group > memberships. I think that's a reasonable thing to want; especially > since the memberOf part is impossible with dynlist. > > Config for these tests: dyngroup schema with groupOfURLs modified to > allow 'member' as per autogroup README, plus: > > dn: olcOverlay={1}memberof,olcDatabase={1}mdb,cn=config > objectClass: olcMemberOf > olcMemberOfGroupOC: groupOfURLs > > dn: olcOverlay={2}autogroup,olcDatabase={1}mdb,cn=config > objectClass: olcAutomaticGroups > olcAGattrSet: groupOfURLs memberURL member > > On Wed, Jun 4, 2014 at 12:24 PM, Ryan Tandy <ryan@nardis.ca> wrote: >> Adding a new entry that triggers an autogroup member to be added: I >> don't get a crash or deadlock, but the new entry doesn't have a >> memberOf attribute. It looks like the internal Modify adding the >> member happens before the Add completes, so memberof tries to add >> memberOf to an entry that doesn't exist yet. > > slapadd: > > dn: cn=autogroup,dc=example,dc=com > objectClass: groupOfURLs > memberURL: ldap:///dc=example,dc=com??one?(objectClass=account) > > and then ldapadd: > > dn: uid=user,dc=example,dc=com > objectClass: account > uid: user > > The new entry doesn't gain a memberOf attr as expected. Running slapd with debug output clearly shows that memberOf tried to modify the target entry before mdb_add ever added the entry. The autogroup overlay is completely broken in its design. It is firing off side-effects before the main operation completes; doing so is stupid/wrong since the main operation may fail for legitimate reasons (e.g. access control checks). autogroup should be plugging into the overlay chain from the response side, so it only acts after the main operation has already succeeded. > Minor nitpicks: if you give autogroup's olcAGattrSet exactly two > arguments, slapadd crashes in ag_cfgen (autogroup.c:1761); and if you > omit the filter from memberURL, i.e. > > memberURL: ldap:///cn=staticgroup,dc=example,dc=com?member?base? > > then slapd crashes in autogroup_modify_entry (autogroup.c:1381) while > refreshing memberships. Ideally those would return an error instead of > segfaulting. > > I'm also wondering, should autogroup/README perhaps recommend > groupOfNames and labeledURI instead of groupOfURLs and member? Patches welcome. > > hope this helps, > Ryan > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
ryan@nardis.ca wrote: > Back in June, Gerry replied to me privately. Copying his thoughts > here, with his permission: > > On Sat, Jun 14, 2014 at 8:13 AM, Gerry Calderhead > <gerry@everythingsucks.co.uk> wrote: >> Hi, >> >> It's been a while since I looked at this - in fact since I originally >> reported it. >> In the end I concluded it was too much work for me to continue investigating >> and re-implemented autogroup using an external perl script >> and removed it from my build of openldap. >> >> My conclusion at the time, IIRC, was that the overlay architecture was a bit >> broken and needed a re-think, specifically to avoid the need to be reentrant >> - ultimately, by making the overlay CB's asynchronous. The autogroup overlay's design was broken, yes. The overlay mechanism overall is fine. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
ryan@nardis.ca wrote: > As I understand it, Gerry's use case is: we have some groups, we want > an autogroup containing the union of their members, and we want > memberOf to accurately reflect both the manual and automatic group > memberships. I think that's a reasonable thing to want; especially > since the memberOf part is impossible with dynlist. > > Config for these tests: dyngroup schema with groupOfURLs modified to > allow 'member' as per autogroup README, plus: > > dn: olcOverlay={1}memberof,olcDatabase={1}mdb,cn=config > objectClass: olcMemberOf > olcMemberOfGroupOC: groupOfURLs > > dn: olcOverlay={2}autogroup,olcDatabase={1}mdb,cn=config > objectClass: olcAutomaticGroups > olcAGattrSet: groupOfURLs memberURL member > > On Wed, Jun 4, 2014 at 12:24 PM, Ryan Tandy <ryan@nardis.ca> wrote: >> Adding a new entry that triggers an autogroup member to be added: I >> don't get a crash or deadlock, but the new entry doesn't have a >> memberOf attribute. It looks like the internal Modify adding the >> member happens before the Add completes, so memberof tries to add >> memberOf to an entry that doesn't exist yet. > > slapadd: > > dn: cn=autogroup,dc=example,dc=com > objectClass: groupOfURLs > memberURL: ldap:///dc=example,dc=com??one?(objectClass=account) > > and then ldapadd: > > dn: uid=user,dc=example,dc=com > objectClass: account > uid: user > > The new entry doesn't gain a memberOf attr as expected. Fixed in git master. >> Changing autogroup to trigger on an alteration to an existing entry, >> for example addition of a specific attribute or auxiliary objectClass: >> with mdb this works and the memberOf attribute is added; with hdb I >> get the same deadlock Landry Breuil already reported. Looks like a >> recursive deadlock on the same thread. > > slapadd: > > dn: cn=autogroup,dc=example,dc=com > objectClass: groupOfURLs > memberURL: ldap:///dc=example,dc=com??one?(objectClass=extensibleObject) > > dn: uid=user,dc=example,dc=com > objectClass: account > > and then ldapmodify: > > dn: uid=user,dc=example,dc=com > add: objectClass > objectClass: extensibleObject > > under mdb, works as expected. > under hdb, deadlock. Fixed in git master. >> I think those problems are both different from the one this ITS is >> about, though. Maybe they should be filed separately. >> >> I also confirmed the problem originally reported on current git master > > I came up with two ways to do this. I don't know which one Gerry used. > > You can use the same config as above and have autogroup expand attr > values. The initial LDIF is: > > dn: cn=autogroup,dc=example,dc=com > objectClass: groupOfURLs > memberURL: ldap:///cn=staticgroup,dc=example,dc=com?member?base?(objectClass=*) > > dn: cn=staticgroup,dc=example,dc=com > objectClass: groupOfNames > member: cn=dummy > > dn: uid=user,dc=example,dc=com > objectClass: account > > (You can't re-use groupOfURLs for the static group because autogroup > blocks manual changes to its members.) > > Then ldapmodify: > > dn: cn=staticgroup,dc=example,dc=com > add: member > member: uid=user,dc=example,dc=com > > dn: cn=staticgroup,dc=example,dc=com > delete: member > member: uid=user,dc=example,dc=com Fixed in git master. > > Alternatively you can make autogroup search by memberOf. For that you need > > olcMemberOfGroupOC: groupOfNames > > (instead of GroupOfURLs). Then you do the same slapadd and ldapmodify, > except with > > memberURL: ldap:///dc=example,dc=com??one?(memberOf=cn=staticgroup,dc=example,dc=com) Fixed in git master. > > AFAICT both approaches work (personally I prefer the first...), but > memberOf is only applied to a single OC. If you want it for both, you > might try the same thing with a second memberof instance, like this: > > dn: olcOverlay={1}memberof,olcDatabase={1}mdb,cn=config > objectClass: olcMemberOf > olcMemberOfGroupOC: groupOfURLs > > dn: olcOverlay={2}memberof,olcDatabase={1}mdb,cn=config > objectClass: olcMemberOf > olcMemberOfGroupOC: groupOfNames > > dn: olcOverlay={3}autogroup,olcDatabase={1}mdb,cn=config > objectClass: olcAutomaticGroups > olcAGattrSet: groupOfURLs memberURL member Not tested. > Then you get the problem Gerry reported: memberOf is added twice but > only deleted once, the second value remains. Same result for both > memberURL configurations. > > Minor nitpicks: if you give autogroup's olcAGattrSet exactly two > arguments, slapadd crashes in ag_cfgen (autogroup.c:1761); Fixed in git master. > and if you > omit the filter from memberURL, i.e. > > memberURL: ldap:///cn=staticgroup,dc=example,dc=com?member?base? > > then slapd crashes in autogroup_modify_entry (autogroup.c:1381) while > refreshing memberships. Ideally those would return an error instead of > segfaulting. Fixed in git master - the offending autogroup URLs are ignored. > > I'm also wondering, should autogroup/README perhaps recommend > groupOfNames and labeledURI instead of groupOfURLs and member? > > hope this helps, > Ryan > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
changed notes changed state Test to Release
fixed in master fixed in RE25 fixed in RE24
changed notes changed state Release to Closed