Full_Name: Emily Backes Version: 2.4.25 OS: any URL: Submission from: (NULL) (76.88.107.46) In recent OpenLDAPs (2.4.25 at least, but I haven't found exactly where it started), memberof interacts badly with accesslog. In a simple test case with a groupOfNames and two people, if you add a person to the group, memberOf should set their memberOf opeational attribute to point to the group. That works! But currently the accesslog db will only show the change for the memberof update and not the original group change. Digging deeper, I found: ==> hdb_add: reqStart=20110422103943.000001Z,cn=log oc_check_required entry (reqStart=20110422103943.000001Z,cn=log), objectClass "auditModify" oc_check_allowed type "objectClass" oc_check_allowed type "structuralObjectClass" oc_check_allowed type "reqStart" oc_check_allowed type "reqEnd" oc_check_allowed type "reqType" oc_check_allowed type "reqSession" oc_check_allowed type "reqAuthzID" oc_check_allowed type "reqDN" oc_check_allowed type "reqResult" oc_check_allowed type "reqMod" bdb_dn2entry("reqStart=20110422103943.000001Z,cn=log") send_ldap_result: conn=1000 op=1 p=3 send_ldap_result: err=68 matched="" text="" The changes are reaching accesslog, but don't make it into the logdb because their generated DNs based on reqStart match. reqStart is generated with a generalizedTime stamp where the microseconds are an incrementing count based on o_tincr, but this does not seem to be incremented, or incremented enough. It's not entirely clear why this is a problem now and not earlier. This may be related to ITS#6766.
changed notes changed state Open to Test moved from Incoming to Software Bugs
ebackes@symas.com wrote: > Full_Name: Emily Backes > Version: 2.4.25 > OS: any > URL: > Submission from: (NULL) (76.88.107.46) > > In recent OpenLDAPs (2.4.25 at least, but I haven't found exactly where > it started), memberof interacts badly with accesslog. See also: http://www.openldap.org/lists/openldap-technical/201104/msg00242.html > In a simple test case with a groupOfNames and two people, if you add a > person to the group, memberOf should set their memberOf opeational > attribute to point to the group. That works! But currently the > accesslog db will only show the change for the memberof update and not > the original group change. I can confirm that. > Digging deeper, I found: > [..] > The changes are reaching accesslog, but don't make it into the logdb > because their generated DNs based on reqStart match. Ah, that explains it. > reqStart is generated with a generalizedTime stamp where the > microseconds are an incrementing count based on o_tincr, but this does > not seem to be incremented, or incremented enough. > > It's not entirely clear why this is a problem now and not earlier. Maybe it was always a problem. Because I've started the thread above before installing 2.4.25: http://www.openldap.org/lists/openldap-technical/201103/msg00032.html I had 2.4.24 or 2.4.23 installed back then. > This may be related to ITS#6766. Seems similar and the group modification is the same like in cases where I observed the behaviour described in my postings. Ciao, Michael.
The openldap-techincal traffic Michael mentioned is almost certainly the same issue. In our case, yes this could have appeared as early as 2.4.23. What I'm seeing now is that formerly memberof functioned with accesslog in a delta-syncrepl type environment by having each stage of replication (master, hub, replica, etc) run the overlay to supply the changes. It seems like originally only the group change went through and the overlay relied on reqStart collisions to prevent its internal operations from reaching the replication log. So: the recent change that exposed this was something that changed the order these things are applied and sent to accesslog. Changing memberof again to reveal only the group changes should probably a separate ITS, or perhaps a discussion (openldap-devel) on how we want it to work. As-is, the fix in git master breaks compatibility with configurations from prior releases. Emily Backes Symas - The LDAP Guys ebackes@symas.com
ebackes@symas.com wrote: > The openldap-techincal traffic Michael mentioned is almost certainly the same issue. In our case, yes this could have appeared as early as 2.4.23. > > What I'm seeing now is that formerly memberof functioned with accesslog in > a delta-syncrepl type environment by having each stage of replication (master, hub, replica, etc) run the overlay to supply the changes. It seems like originally only the group change went through and the overlay relied on reqStart collisions to prevent its internal operations from reaching the replication log. > > So: the recent change that exposed this was something that changed the > order these things are applied and sent to accesslog. > > Changing memberof again to reveal only the group changes should probably a separate ITS, or perhaps a discussion (openldap-devel) on how we want it to work. As-is, the fix in git master breaks compatibility with configurations from prior releases. The breakage seems to go back to ITS#6329. The original design for memberOf was for the internal modifications to not be replicated. Instead, any replicas that wanted to maintain member information was expected to run an identical memberOf overlay configuration. The fact that memberOf didn't update the entryCSN on its internal modifications was intentional. This design constraint was broken by the fix to ITS#6329. The patch for #6329 and any ripple effect from it needs to be reverted. In general it's incorrect to replicate internal operations. The fanout from replicating every internal operation would be too large and the information content of what is being replicated is essentially nil. When the servers are configured identically they will maintain identical data by virtue of the overlays on each node performing the same internal operations in response to a given sequence of user operations. > Emily Backes > Symas - The LDAP Guys > ebackes@symas.com > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
I figured I would share a workaround that I'm currently using for this issue which may be of help to others. I've disabled the memberOf overlay in slapd, and use an external script to populate memberOf on the master server, which then replicates to the consumer servers. I currently run this every 5 minutes from cron as follows: memberof.pl --ldap Regards, -Kartik
Do you think this could be related to: http://www.openldap.org/its/index.cgi?findid=6864 I've been having similar issues with MemberOf and Accesslog overlays used together. In your fix, is the memberof overlay enabled on your consumer nodes? -Yuri On Wed, Jun 1, 2011 at 1:00 PM, <subbarao@computer.org> wrote: > This is a multi-part message in MIME format. > --------------050703040907090602090901 > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > Content-Transfer-Encoding: 7bit > > I figured I would share a workaround that I'm currently using for this > issue which may be of help to others. I've disabled the memberOf overlay > in slapd, and use an external script to populate memberOf on the master > server, which then replicates to the consumer servers. I currently run > this every 5 minutes from cron as follows: > > memberof.pl --ldap > > Regards, > > -Kartik > > --------------050703040907090602090901 > Content-Type: application/x-perl; > name="memberof.pl" > Content-Transfer-Encoding: 7bit > Content-Disposition: attachment; > filename="memberof.pl" > > #! /usr/bin/perl > > # Implements memberOf reverse mapping attributes -- workaround for when > # memberOf overlay isn't available > > use Net::LDAP; > use Net::LDAP::LDIF; > use Authen::SASL; > use Fcntl qw(LOCK_EX LOCK_NB); > use Getopt::Long; > > use strict; > > my $basedn = "dc=example,dc=com"; > > my @attrs = qw(member manager); > # Note -- this filter properly excludes dynamic groupOfURLs groups > my $attrfilter = '(|' . join("", map { "($_=*)" } @attrs) . ')'; > my %revattrs = (member => 'memberOf', manager => 'directReports'); > my %fwattrs = reverse %revattrs; > my $revattrfilter = '(|' . join("", map { "($_=*)" } values %revattrs) . > ')'; > my (%entries, %reventries); > > # Prevent multiple instances from running at the same time > open(LOCKFH, $0); flock(LOCKFH, LOCK_EX|LOCK_NB) or exit 1; > > my ($generate_ldif, $update_ldap); > GetOptions('ldif' => \$generate_ldif, 'ldap' => \$update_ldap); > > my $ldifout = Net::LDAP::LDIF->new('-', 'w'); > $ldifout->{change} = 1; > my $ldap = Net::LDAP->new('ldapi://') or die "ldapi: $@\n"; > my $sasl = Authen::SASL->new(mechanism => 'EXTERNAL'); > my $sasl_client = $sasl->client_new('ldap', 'localhost'); > $ldap->bind(undef, sasl => $sasl_client); > > # Build %entries and %reventries maps > my $mesg = $ldap->search(base => $basedn, > filter => $attrfilter, > attrs => \@attrs); > $mesg->code && die($mesg->error . "\n"); > foreach my $entry ($mesg->all_entries) {$entries{lc $entry->dn} = $entry } > > $mesg = $ldap->search(base => $basedn, > filter => $revattrfilter, > attrs => [values > %revattrs]); > $mesg->code && die($mesg->error . "\n"); > foreach my $entry ($mesg->all_entries) { $reventries{lc $entry->dn} = > $entry } > > # Go through and generate updates for the reverse mapping attributes > my ($dn, $entry); > while (($dn, $entry) = each %entries) { > foreach my $attr (@attrs) { > my $revattr = $revattrs{$attr}; > foreach my $val ($entry->get_value($attr)) { > $val = lc $val; > if (!$reventries{$val}) { > $reventries{$val} = Net::LDAP::Entry->new; > $reventries{$val}->dn($val); > $reventries{$val}->changetype('modify'); > } > $reventries{$val}->add($revattr => $entry->dn) > unless grep({ lc $_ eq $dn } > > $reventries{$val}->get_value($revattr)); > } > } > } > while (($dn, $entry) = each %reventries) { > foreach my $revattr (values %revattrs) { > foreach my $val ($entry->get_value($revattr)) { > $val = lc $val; > $reventries{$dn}->delete($revattr => $val) > if !exists($entries{$val}) > || !grep({ lc $_ eq $dn } > > $entries{$val}->get_value($fwattrs{$revattr})); > > } > } > if ($entry->changes) { > $ldifout->write_entry($entry) if $generate_ldif; > if ($update_ldap) { > my $modmesg = $entry->update($ldap); > $modmesg->code && die("LDAP: " .$modmesg->error . > "\n"); > } > } > } > > --------------050703040907090602090901-- > > >
On 06/01/2011 07:36 PM, Yuri Bank wrote: > Do you think this could be related to: > http://www.openldap.org/its/index.cgi?findid=6864 > > I've been having similar issues with MemberOf and Accesslog overlays > used together. I've occasionally experienced similar things that you describe but haven't captured the detailed data to be able to correlate that with this issue. > In your fix, is the memberof overlay enabled on your consumer nodes? No, it is disabled everywhere. -Kartik
changed notes changed state Test to Release
changed notes changed state Release to Closed
fixed in HEAD fixed in RE24