OpenLDAP
Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest

Viewing Software Bugs/6540
Full headers

From: ryans@aweber.com
Subject: test022-ppolicy is flawed, masks serious stability issue
Compose comment
Download message
State:
0 replies:
19 followups: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Major security issue: yes  no

Notes:

Notification:


Date: Wed, 28 Apr 2010 19:04:38 +0000
From: ryans@aweber.com
To: openldap-its@OpenLDAP.org
Subject: test022-ppolicy is flawed, masks serious stability issue
Full_Name: Ryan Steele
Version: 2.4.18
OS: Ubuntu Server
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (207.106.239.81)


When the chaining configuration for cn=config is added, as is done in
test022-ppolicy, the process of adding the module and overlay succeed, but
subsequent slapcat operations will fail with:

root@nebula:~# slapcat -n1
slapd-chain: first underlying database
"olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config"
cannot contain attribute "olcDbURI".
config error processing
olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config: 
slapcat: bad configuration file!

Additionally, if slapd is stopped after adding the configuration in
test022-ppolicy, the server will not start again, and on the foreground shows:

slapd-chain: first underlying database
"olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config"
cannot contain attribute "olcDbURI".
: config_add_internal:
DN="olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config"
no structural objectClass add function
config error processing
olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config: 
send_ldap_result: conn=-1 op=0 p=0
send_ldap_result: err=65 matched="" text=""
slapd destroy: freeing system resources.
slapd stopped.
connections_destroy: nothing to destroy.

The reason test022-ppolicy does not catch this is because an ldapsearch will
still work.  In fact, the chaining operations still succeed (writes are ferried
off to the upstream server).  But, this is a very grave problem, as it can cause
the slapd server to stop functioning completely.

Followup 1

Download message
Date: Wed, 28 Apr 2010 15:19:44 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: ryans@aweber.com
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
I should also note that a slapcat of the config database fails too in the exact
same manner.


I'm guessing this is somehow related to the fact that the chain module/overlay
is located within the back_ldap module,
but I'm not sure how to work around the issue yet, so at this time slapd will
not start on the slave server on which
slapd was stopped after adding the chaining configuration as is documented in
test022-ppolicy.



Followup 2

Download message
Date: Wed, 28 Apr 2010 21:20:38 +0200 (CEST)
Subject: ITS#6540
From: masarati@aero.polimi.it
To: ryans@aweber.com
Cc: openldap-its@openldap.org
Please do not report issues related to old releases.  Current is 2.4.22. 
Please test and report whether the issue persists.

p.



Followup 3

Download message
Date: Wed, 28 Apr 2010 16:09:32 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: ryans@aweber.com
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
The other thing I should note is initially, everything seemed to work, but
became prevalent the very first time I
restarted slapd after adding the chaining configurations from test022-ppolicy. 
That's when all the errors started with
slapcat.  The second time I restarted it (to try and fix the issue), slapd would
not start at all, as described above.



Followup 4

Download message
Date: Wed, 28 Apr 2010 16:12:57 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: ryans@aweber.com
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
> Date: Wed, 28 Apr 2010 21:20:38 +0200 (CEST)
> Subject: ITS#6540
> From: masarati@aero.polimi.it
> To: ryans@aweber.com
> Cc: openldap-its@openldap.org
>
> Please do not report issues related to old releases.  Current is 2.4.22.
> Please test and report whether the issue persists.
>
> p.


I checked my version of back-ldap/chain.c and back-ldap/config.c, and there have
been no changes that in any way apply
to or address the serious problem I've encountered.  For reference, my chain.c
is version 1.52.2.10, and config.c is
version 1.115.2.14



Followup 5

Download message
Date: Wed, 28 Apr 2010 16:49:59 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: ryans@aweber.com
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
It is also bears mentioning that test022-ppolicy has not changed at all since
2.4.18, and so would never detect this
problem, even using the most recent stable version.  As I mentioned, ldapsearch
does not trigger any errors - only
slapcat does (and of course, starting slapd, which fails as is detailed above).


If you really want to test this, you must restart the server on which the
chaining configuration from test022-ppolicy
has been added, and then run slapcat.  I am not the only one to run in to this;
for example, this individual has the
exact same issue, with no resolution:
http://www.openldap.org/lists/openldap-software/200907/msg00118.html


I'd be more than happy to patch the source, if such a patch existed.  But since
nothing seems to have changed in the
relevant bits of code that would affect this behavior, and no corresponding ITS
exists, that would lead me to believe it
is thusly undocumented, except for similar on-list complaints.  If you can show
me where in the back-ldap code you
believe I've missed said patch or refactoring, I've got no problem admitting I
missed it, but I don't see any such
modification(s).


I'm sure not everybody has the time to build from completely new source every
time a bug is unearthed, and while I do
not object to doing that as soon as I can, right now I have a slapd server that
is down because of this, so top priority
is to evaluate the code to figure out where this bug is stemming from and fix it
so I can restart slapd.  Next priority
is building and testing new packages.  I mean no disrespect by this, but
dropping everything to do this and work out any
new issues, bugs, and compatibility problems and separate those from this
pre-existing buggy behavior, seems hasty at
best, especially when there's no concretely identified reason for doing so
(i.e., this revision had change X, which
fixes problem Y).


I'm still reviewing the relevant code (which AFAICT hasn't changed) to figure
out what's going on here and put out the
immediate fire so that I can at least restore services, at which point I will
obviously have to make the time to
completely upgrade it all.

Thanks



Followup 6

Download message
Date: Thu, 29 Apr 2010 15:41:48 +0200 (CEST)
Subject: Re: (ITS#6540) test022-ppolicy is flawed,
      masks serious stability issue
From: masarati@aero.polimi.it
To: ryans@aweber.com
Cc: openldap-its@openldap.org
Let me first explain one thing.  If you have one problem and you don't
have time to rebuild from source, then it's very likely you'll have to
live with your problem.  The only difference between you and me is that if
I find a solution, I can commit the fix.  For the rest, your time is
valuable exactly as mine, and I probably get paid less than you get paid,
but to fix my problems, not yours.  It just happened that tracking
someone's issue, upgrading just solved it (ITS#6537, which ended up being
a duplicate of another, already solved issue).  In order to save the very
scarce, unpaid manpower we can dedicate to fixing others' issues, our
policy is to avoid tracking issues in old releases, because we don't
backport fixes so, in the best case, the issue will be solved in the next
one, and you'll have to upgrade anyway.

After this lengthy preamble, thanks to your prompt response I already
figured out myself how to reproduce the issue you're having, so you don't
need to rebuild yet.  You'll have to as soon as the issue is fixed.

p.

> It is also bears mentioning that test022-ppolicy has not changed at all
> since 2.4.18, and so would never detect this
> problem, even using the most recent stable version.  As I mentioned,
> ldapsearch does not trigger any errors - only
> slapcat does (and of course, starting slapd, which fails as is detailed
> above).
>
>
> If you really want to test this, you must restart the server on which the
> chaining configuration from test022-ppolicy
> has been added, and then run slapcat.  I am not the only one to run in to
> this; for example, this individual has the
> exact same issue, with no resolution:
> http://www.openldap.org/lists/openldap-software/200907/msg00118.html
>
>
> I'd be more than happy to patch the source, if such a patch existed.  But
> since nothing seems to have changed in the
> relevant bits of code that would affect this behavior, and no
> corresponding ITS exists, that would lead me to believe it
> is thusly undocumented, except for similar on-list complaints.  If you can
> show me where in the back-ldap code you
> believe I've missed said patch or refactoring, I've got no problem
> admitting I missed it, but I don't see any such
> modification(s).
>
>
> I'm sure not everybody has the time to build from completely new source
> every time a bug is unearthed, and while I do
> not object to doing that as soon as I can, right now I have a slapd server
> that is down because of this, so top priority
> is to evaluate the code to figure out where this bug is stemming from and
> fix it so I can restart slapd.  Next priority
> is building and testing new packages.  I mean no disrespect by this, but
> dropping everything to do this and work out any
> new issues, bugs, and compatibility problems and separate those from this
> pre-existing buggy behavior, seems hasty at
> best, especially when there's no concretely identified reason for doing so
> (i.e., this revision had change X, which
> fixes problem Y).
>
>
> I'm still reviewing the relevant code (which AFAICT hasn't changed) to
> figure out what's going on here and put out the
> immediate fire so that I can at least restore services, at which point I
> will obviously have to make the time to
> completely upgrade it all.
>
> Thanks
>
>
>
>




Followup 7

Download message
Date: Thu, 29 Apr 2010 17:30:42 +0200 (CEST)
Subject: Re: (ITS#6540) test022-ppolicy is flawed,
      masks serious stability issue
From: masarati@aero.polimi.it
To: ryans@aweber.com
Cc: openldap-its@openldap.org
> I already
> figured out myself how to reproduce the issue you're having, so you don't
> need to rebuild yet.  You'll have to as soon as the issue is fixed.

Try this patch:

<ftp://ftp.openldap.org/incoming/pierangelo-masarati-2010-04-29-chain.1.patch>

It removes the offending check.  I don't quite remember the reason (it
dates 4.5 years ago).  However, I've checked and all tests pass fine,
including the one you pointed out.  To reproduce, I modified test022 to
write the configuration on disk (-F option to consumer slapd); at the end
of the test, I tried slapd on the resulting database.  Fails with
2.4.22/HEAD, succeeds with this patch.

Please test and report.

p.




Followup 8

Download message
Date: Thu, 29 Apr 2010 13:38:07 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: ryans@aweber.com
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
> Date: Thu, 29 Apr 2010 15:41:48 +0200 (CEST)
> Subject: Re: (ITS#6540) test022-ppolicy is flawed,
>       masks serious stability issue
> From: masarati@aero.polimi.it
> To: ryans@aweber.com
> Cc: openldap-its@openldap.org
>
> Let me first explain one thing.  If you have one problem and you don't
> have time to rebuild from source, then it's very likely you'll have to
> live with your problem.


Yeah, I don't have a problem with rebuilding from source.  Even with a patch,
that would have to be done.  I was just
trying to, in the interest of urgency, avoid having to rebuild from 100% brand
new source, instead of being able to
patch the relevant code and tests and rebuild from that codebase.   This would
preclude having to worry about
distinguishing new problems from the existing problem, or diagnosing and fixing
additional and unrelated problems along
the way, making it potentially easier isolate the issue.


> The only difference between you and me is that if I find a solution, I
> can commit the fix.  For the rest, your time is valuable exactly as
> mine, and I probably get paid less than you get paid, but to fix my
> problems, not yours.


Sure, but in this kind of scenario, aren't they one and the same?  The burden of
identifying and fixing bugs in the code
is (in spirit, at least, unless there is a contract or legally binding agreement
involved) shared by both developers and
community users as we strive towards a common goal, although the nature of the
responsibility varies to some degree.  I
agree, however, that in most circumstances the contributions made by community
members and developers are of their own
volition and centric to, if not completely predicated upon, their needs and
interests.


Because commits do necessitate developer involvement and thus ultimately depend
on the allocation of their limited and
valuable time, it is my belief that community users bear some responsibility in
this process to help reduce the load on
the developers, and should exercise due diligence when submitting ITS's and
contributing to their solutions.  Up to this
point, I have tried to adhere to this self-subscribed policy as best as I can
for this ITS, and am more than happy to
contribute wherever possible, especially on things for which a seasoned OpenLDAP
developer's experience and knowledge
are a less important pre-requisite (like patching test022-ppolicy, or
documentation, for example).  So, please let me
know if there's anything I can do to help facilitate this process.  I know that
nobody here is obligated, nor are their
services free, so all time and effort expended is very valuable and highly
appreciated - I can't stress any of those
points enough!


Also, completely OT but with respect to the 'salary' comment, if you're not
getting paid at least as well or more than I
am to teach aerospace engineering, it is a shame, but not all that surprising. 
Nobody in academia (at least in the
United States) seems to get paid nearly what they're worth, regardless of the
position (grade school teacher like my
wife, or university professor such as yourself) or the degree of difficulty of
the subject.  We grumble about it every
so often in our household - but I digress...


> It just happened that tracking someone's issue, upgrading just solved
> it (ITS#6537, which ended up being a duplicate of another, already
> solved issue).  In order to save the very scarce, unpaid manpower we
> can dedicate to fixing others' issues, our policy is to avoid tracking
> issues in old releases, because we don't backport fixes so, in the best
> case, the issue will be solved in the next one, and you'll have to
> upgrade anyway.


Sure, and that's why I did my very best to review the source, and verify that
the changelogs, diffs, and source relevant
to the behavior had remained unchanged.  But I submit that even with the most
painstaking and diligent scrutiny of the
code, it still holds true that "the proof is in the pudding", and few test are
more ironclad than actually building the
newest source and comparing the behavior.  I did actually rebuild slapd this
morning as requested, though, and still had
the same issue (2.4.22), but when I went to update the ticket, I saw that you
had already confirmed what I suspected
prior to doing so.


> After this lengthy preamble, thanks to your prompt response I already
> figured out myself how to reproduce the issue you're having, so you don't
> need to rebuild yet.  You'll have to as soon as the issue is fixed.


Great, many thanks for your efforts in confirming.  Just out of curiosity, when
the fix is actually committed, will the
commit number or source code file be present in the ITS, so that one can easily
cross-reference the ITS with a
particular file or set of files from CVS HEAD?  And again, please let me know if
there's anything I can do to help
facilitate the process, or make it easier for you and/or the other develope

Message of length 7670 truncated


Followup 9

Download message
Date: Thu, 29 Apr 2010 13:40:39 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: ryans@aweber.com
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
> Date: Thu, 29 Apr 2010 17:30:42 +0200 (CEST)
> Subject: Re: (ITS#6540) test022-ppolicy is flawed,
>       masks serious stability issue
> From: masarati@aero.polimi.it
> To: ryans@aweber.com
> Cc: openldap-its@openldap.org
>
>> I already
>> figured out myself how to reproduce the issue you're having, so you
don't
>> need to rebuild yet.  You'll have to as soon as the issue is fixed.
>
> Try this patch:
>
> <ftp://ftp.openldap.org/incoming/pierangelo-masarati-2010-04-29-chain.1.patch>
>
> It removes the offending check.  I don't quite remember the reason (it
> dates 4.5 years ago).  However, I've checked and all tests pass fine,
> including the one you pointed out.  To reproduce, I modified test022 to
> write the configuration on disk (-F option to consumer slapd); at the end
> of the test, I tried slapd on the resulting database.  Fails with
> 2.4.22/HEAD, succeeds with this patch.
>
> Please test and report.
>
> p.
>


Sure, will do that now and report back.  Thanks.


-Ryan



Followup 10

Download message
Date: Thu, 29 Apr 2010 11:09:40 -0700
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: ryans@aweber.com, openldap-its@openldap.org
Subject: Re: (ITS#6540) test022-ppolicy is flawed,	masks serious stability
 issue
--On Thursday, April 29, 2010 5:38 PM +0000 ryans@aweber.com wrote:

> Great, many thanks for your efforts in confirming.  Just out of
> curiosity, when the fix is actually committed, will the commit number or
> source code file be present in the ITS, so that one can easily
> cross-reference the ITS with a particular file or set of files from CVS
> HEAD?  And again, please let me know if there's anything I can do to help
> facilitate the process, or make it easier for you and/or the other
> developers.  I'm happy to test, patch the test scripts, et cetera.

Unfortunately no, that type of commit -> bug integration does not currently 
exist, although I think it is a highly desirable feature and one that has 
been discussed for implementation.

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration



Followup 11

Download message
Date: Tue, 04 May 2010 11:16:37 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: ryans@aweber.com
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
After patching, and using the same configuration as I had when the chain overlay
was causing issues with slapcat and
restarting slapd, I now get prompted with a referral instead of it being
automatically chased.  However, it does
automatically fill in the DN and password to rebind with:


root@somehost:~# ldapvi -h localhost --bind=simple -D cn=admin,dc=example,dc=com
-w `cat /etc/ldap.secret` --discover
    159 entries read

add: 0, rename: 0, modify: 1, delete: 0
Action? [yYqQvVebB*rsf+?] y
Received referral to
ldap://ldapmaster.example.com/uid=ryans,ou=Users,dc=example,dc=com.
You are not logged in to ldap://ldapmaster.example.com:389 yet.
Type '!' or 'y' to do so.
Rebind? [y!nB*qQ?] y

--- Login
Type M-h for help on key bindings.

Filter or DN: cn=admin,dc=example,dc=com
    Password: ***********
Bound as cn=admin,dc=example,dc=com.
Done.


Before, I never got prompted with this message when using ldapvi, which makes me
think that chaining is no longer
working.  For reference, I am using the same configuration as is documented in
test022-ppolicy:

dn: cn=module{0},cn=config
objectClass: olcModuleList
cn: module{0}
olcModulePath: /usr/lib/ldap
olcModuleLoad: {0}back_hdb.la
olcModuleLoad: {1}autogroup.la
olcModuleLoad: {2}syncprov.la
olcModuleLoad: {3}back_ldap.la


dn: olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcOverlayConfig
objectClass: olcChainConfig
olcOverlay: {0}chain


dn: olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcLDAPConfig
objectClass: olcChainDatabase
olcDatabase: {0}ldap
olcDbURI: ldap://ldapmaster.example.com
olcDbIDAssertBind: bindmethod=simple binddn="cn=admin,dc=example,dc=com"
credentials=SECRET mode=self



I am still looking in to what might be causing this to fail.







Followup 12

Download message
Date: Tue, 04 May 2010 12:40:29 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: ryans@aweber.com
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
Actually, it would appear I'm hitting the same problem as the OP in this thread:

http://markmail.org/message/fhfhzy5uehh6e4us#query:openldap chain "modifications
require
authentication"+page:1+mid:fhfhzy5uehh6e4us+state:results


I say that because when I get prompted for authentication by the slave (instead
of having the referral handled
server-side), I see this corresponding entry in the master's logs:

May  4 12:22:43 ldap1 slapd[8794]: conn=226810 fd=50 ACCEPT from
IP=10.x.x.x:45081 (IP=0.0.0.0:389)
May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=0 BIND dn="" method=128
May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=0 RESULT tag=97 err=0 text=
May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=1 MOD
dn="uid=ryans,ou=Users,dc=example,dc=com"
May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=1 MOD attr=displayColor
May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=1 RESULT tag=103 err=8
text=modifications require authentication
May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=2 UNBIND
May  4 12:22:43 ldap1 slapd[8794]: conn=226810 fd=50 closed


So, it looks like the DN is not being passed through for some reason.  Still
working on trying to track it down...



Followup 13

Download message
Date: Tue, 04 May 2010 15:17:01 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
Ryan Steele wrote:
> Actually, it would appear I'm hitting the same problem as the OP in this
thread:
> 
> http://markmail.org/message/fhfhzy5uehh6e4us#query:openldap chain
"modifications require
> authentication"+page:1+mid:fhfhzy5uehh6e4us+state:results
> 
> 
> I say that because when I get prompted for authentication by the slave
(instead of having the referral handled
> server-side), I see this corresponding entry in the master's logs:
> 
> May  4 12:22:43 ldap1 slapd[8794]: conn=226810 fd=50 ACCEPT from
IP=10.x.x.x:45081 (IP=0.0.0.0:389)
> May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=0 BIND dn="" method=128
> May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=0 RESULT tag=97 err=0
text=
> May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=1 MOD
dn="uid=ryans,ou=Users,dc=example,dc=com"
> May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=1 MOD attr=displayColor
> May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=1 RESULT tag=103 err=8
text=modifications require authentication
> May  4 12:22:43 ldap1 slapd[8794]: conn=226810 op=2 UNBIND
> May  4 12:22:43 ldap1 slapd[8794]: conn=226810 fd=50 closed
> 
> 
> So, it looks like the DN is not being passed through for some reason. 
Still working on trying to track it down...
> 


The same issue affects ldappasswd (it refuses to change the passwd because it
thinks the user has not bound to the
directory), so it's not just third party utilities.  It really does appear as
though the chain overlay is stripping the
DN before sending the request out to the master, because the logs seem to
indicate that the request makes it to the
master and fails because there is no DN before a referral is presented to the
client that initially sent the update to
the slave.  I added the 'ACL' loglevel to what I already use ('stats' and
'sync'), just to confirm that it is wasn't
being denied by ACL's or anything like that.


I do see references to the 'authzFrom' and 'authzTo' attributes in the admin
guide's section on chaining, and on the FAQ
(http://www.openldap.org/faq/data/cache/1425.html), but the example in
test022-ppolicy does not use them, so I would not
think that they are required or causing this problem.  In any case, I did add
olcDbIDAssertAuthzFrom: {0}"*" to the
slaves (as the FAQ says) just to be absolutely sure, but it made no difference.


I initially thought that the uncommenting of that check was responsible for this
behavior, but the followup I sent just
prior to this has a link to an OpenLDAP mailing list thread that was made in
February, well before these changes that
you made, Pierangelo.  In either case, the behavior is broken.  Exactly where,
I'm still trying to figure out, but would
definitely appreciate any input from more seasoned OpenLDAP developers.




Followup 14

Download message
Date: Wed, 05 May 2010 11:06:30 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: ryans@aweber.com
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
I made an important discovery this morning (with a fresh set of eyes...):
without Pierangelo's patch, chaining does
*NOT* work, as originally stated.  So, the patch does fix the broken test that
started this ITS, making it both valid
and absolutely necessary, but alas, the functionality of the chaining
configuration is the same with both the patched
and unpatched package: the DN is not being passed through, so the automatic
chasing of the referral does not work.


Here is an example with ldapvi:

### Command issued on slave to change one attribute of user
uid=ryans,ou=Users,dc=example,dc=com
ryans@slave:~# ldapvi --ldap-conf --starttls -h localhost --bind=simple -D
cn=admin,dc=example,dc=com -w `cat
/etc/ldap.secret` --discover
    159 entries read
add: 0, rename: 0, modify: 1, delete: 0
Action? [yYqQvVebB*rsf+?] y
Received referral to
ldap://ldapmaster.example.com/uid=ryans,ou=Users,dc=example,dc=com.
You are not logged in to ldap://ldapmaster.example.com:389 yet.
Type '!' or 'y' to do so.
Rebind? [y!nB*qQ?]

### Logs on slave generated by ldapvi command
May  5 10:27:04 slave slapd[30408]: conn=44 fd=41 ACCEPT from IP=127.0.0.1:34118
(IP=0.0.0.0:389)
May  5 10:27:04 slave slapd[30408]: conn=44 op=0 EXT oid=1.3.6.1.4.1.1466.20037
May  5 10:27:04 slave slapd[30408]: conn=44 op=0 STARTTLS
May  5 10:27:04 slave slapd[30408]: conn=44 op=0 RESULT oid= err=0 text=
May  5 10:27:04 slave slapd[30408]: conn=44 fd=41 TLS established tls_ssf=128
ssf=128
May  5 10:27:04 slave slapd[30408]: conn=44 op=1 BIND
dn="cn=admin,dc=example,dc=com" method=128
May  5 10:27:04 slave slapd[30408]: conn=44 op=1 BIND
dn="cn=admin,dc=example,dc=com" mech=SIMPLE ssf=0
May  5 10:27:04 slave slapd[30408]: conn=44 op=1 RESULT tag=97 err=0 text=
May  5 10:27:04 slave slapd[30408]: conn=44 op=2 SRCH base="" scope=0 deref=0
filter="(objectClass=*)"
May  5 10:27:04 slave slapd[30408]: conn=44 op=2 SRCH attr=+ *
May  5 10:27:04 slave slapd[30408]: conn=44 op=2 ENTRY dn=""
May  5 10:27:04 slave slapd[30408]: conn=44 op=2 SEARCH RESULT tag=101 err=0
nentries=1 text=
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 SRCH base="dc=example,dc=com"
scope=2 deref=0 filter="(objectClass=*)"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY dn="dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY
dn="cn=admin,dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY
dn="ou=users,dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY
dn="ou=groups,dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY
dn="uid=xxxxx,ou=users,dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY
dn="uid=ryans,ou=users,dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY
dn="uid=xxxxx,ou=users,dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY
dn="cn=xxxxx,ou=groups,dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY
dn="cn=ryans,ou=groups,dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 ENTRY
dn="cn=xxxxxx,ou=groups,dc=example,dc=com"
May  5 10:27:04 slave slapd[30408]: conn=44 op=3 SEARCH RESULT tag=101 err=0
nentries=159 text=
May  5 10:27:22 slave slapd[30408]: conn=44 op=4 MOD
dn="uid=ryans,ou=Users,dc=example,dc=com"
May  5 10:27:22 slave slapd[30408]: conn=44 op=4 MOD attr=displayColor
May  5 10:27:22 slave slapd[30408]: conn=44 op=4 RESULT tag=103 err=10 text=


The referral is generated (err=10), and sent to the master, and here's what the
logs there look like:

### Logs on master generated by referral

May  5 10:43:04 ldap1 slapd[8794]: conn=402475 fd=273 ACCEPT from
IP=10.0.1.196:43376 (IP=0.0.0.0:389)
May  5 10:43:04 ldap1 slapd[8794]: conn=402475 op=0 BIND dn="" method=128
May  5 10:43:04 ldap1 slapd[8794]: conn=402475 op=0 RESULT tag=97 err=0 text=
May  5 10:43:04 ldap1 slapd[8794]: conn=402475 op=1 MOD
dn="uid=ryans,ou=Users,dc=example,dc=com"
May  5 10:43:04 ldap1 slapd[8794]: conn=402475 op=1 MOD attr=displayColor
May  5 10:43:04 ldap1 slapd[8794]: conn=402475 op=1 RESULT tag=103 err=8
text=modifications require authentication
May  5 10:43:04 ldap1 slapd[8794]: conn=402475 op=2 UNBIND
May  5 10:43:04 ldap1 slapd[8794]: conn=402475 fd=273 closed
May  5 10:43:04 ldap1 slapd[8794]: conn=402476 fd=273 ACCEPT from
IP=10.0.1.196:43377 (IP=0.0.0.0:389)


As you can see, the DN is empty.  It might be worth mentioning that I have tried
the olcDbURI with dotted-quads,
hostnames, and with and without quotes, and the value portions of the attr=value
pairs in the olcDbIDAssertBind
attribute with and without quotes, to no avail, though I did not expect it to. 
I have also tried with and without TLS
(my configuration supports both) in the chaining configuration, and explicitly
with olcDbChaseReferrals set to TRUE,
though that should be the default.  I've also tried with and without proxy authz
(authzTo/authzFrom) in a vain attempt
to get this working, as is mentioned above in this ITS (beca

Message of length 7631 truncated


Followup 15

Download message
Date: Wed, 28 Jul 2010 10:52:07 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: Pierangelo Masarati <masarati@aero.polimi.it>
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
Hi Pierangelo,

>> Try this patch:
>>
>> <ftp://ftp.openldap.org/incoming/pierangelo-masarati-2010-04-29-chain.1.patch>
>>
>> It removes the offending check.  I don't quite remember the reason (it
>> dates 4.5 years ago).  However, I've checked and all tests pass fine,
>> including the one you pointed out.  To reproduce, I modified test022 to
>> write the configuration on disk (-F option to consumer slapd); at the
end
>> of the test, I tried slapd on the resulting database.  Fails with
>> 2.4.22/HEAD, succeeds with this patch.
>>
>> Please test and report.
>>
>> p.

I just wanted to see if you'd had a chance to check into this yet?  As mentioned
in the last ITS response, your patch
does fix the test script, but unfortunately, referrals are still not being
automatically chased because the DN is not
passed correctly upstream to the read/write master.  Thoughts?

-- 
Ryan Steele                                    ryans@aweber.com
Systems Administrator                          +1 215-825-2196 x758
AWeber Communications                          http://www.aweber.com



Followup 16

Download message
Date: Wed, 28 Jul 2010 11:10:09 -0400
From: Ryan Steele <ryans@aweber.com>
To: openldap-its@OpenLDAP.org
CC: Pierangelo Masarati <masarati@aero.polimi.it>
Subject: Re: (ITS#6540) test022-ppolicy is flawed, masks serious stability
 issue
Hey Pierangelo,

I just noticed http://www.openldap.org/its/index.cgi?findid=6574 - this looks
like it fixes the issue I mentioned in
http://www.openldap.org/its/index.cgi?findid=6540?  Is that assertion correct?

Ryan Steele wrote:
> Hi Pierangelo,
> 
>>> Try this patch:
>>>
>>> <ftp://ftp.openldap.org/incoming/pierangelo-masarati-2010-04-29-chain.1.patch>
>>>
>>> It removes the offending check.  I don't quite remember the reason
(it
>>> dates 4.5 years ago).  However, I've checked and all tests pass
fine,
>>> including the one you pointed out.  To reproduce, I modified
test022 to
>>> write the configuration on disk (-F option to consumer slapd); at
the end
>>> of the test, I tried slapd on the resulting database.  Fails with
>>> 2.4.22/HEAD, succeeds with this patch.
>>>
>>> Please test and report.
>>>
>>> p.
> 
> I just wanted to see if you'd had a chance to check into this yet?  As
mentioned in the last ITS response, your patch
> does fix the test script, but unfortunately, referrals are still not being
automatically chased because the DN is not
> passed correctly upstream to the read/write master.  Thoughts?
> 

-- 
Ryan Steele                                    ryans@aweber.com
Systems Administrator                          +1 215-825-2196 x758
AWeber Communications                          http://www.aweber.com



Followup 17

Download message
Date: Sat, 31 Jul 2010 04:05:38 +0200 (CEST)
Subject: Re: (ITS#6540) test022-ppolicy is flawed,
      masks serious stability issue
From: masarati@aero.polimi.it
To: ryans@aweber.com
Cc: openldap-its@openldap.org
> Hey Pierangelo,
>
> I just noticed http://www.openldap.org/its/index.cgi?findid=6574 - this
> looks like it fixes the issue I mentioned in
> http://www.openldap.org/its/index.cgi?findid=6540?  Is that assertion
> correct?

I couldn't specifically look at your issue (apart from the simple fix to
allow cn=config to work).  ITS#6574 should have nothing to do with your
issue, as it affects back-meta in a part that has no code commonality with
back-ldap, and slapo-chain is built on top of back-ldap.

I'm sorry I went back through this thread and got somehow lost; what's the
exact topic now?  I infer (from followup #14) that there seems to be an
issue about chaining because slapo-chain seems not to rebind as expected. 
Could you isolate the problem in a minimal setup (slapd.conf or the
corresponding back-config's LDIF, a database and a simple operation that
can be performed with ldapmodify/ldappasswd or so) to clearly reproduce
the issue?

Thanks, p.



Followup 18

Download message
Date: Wed, 06 Oct 2010 16:59:10 -0400
From: Ryan Steele <ryans@aweber.com>
To: masarati@aero.polimi.it
CC: openldap-its@openldap.org
Subject: Re: (ITS#6540) test022-ppolicy is flawed,      masks serious stability
 issue
masarati@aero.polimi.it wrote:
>> Hey Pierangelo,
>>
>> I just noticed http://www.openldap.org/its/index.cgi?findid=6574 - this
>> looks like it fixes the issue I mentioned in
>> http://www.openldap.org/its/index.cgi?findid=6540?  Is that assertion
>> correct?
> 
> I couldn't specifically look at your issue (apart from the simple fix to
> allow cn=config to work).  ITS#6574 should have nothing to do with your
> issue, as it affects back-meta in a part that has no code commonality with
> back-ldap, and slapo-chain is built on top of back-ldap.


I apologize - it had occurred to me that since
http://www.openldap.org/doc/admin24/backends.html#Metadirectory states
that back-meta and back-ldap share a subset of code, and since the issue
described in ITS#6574 appears to be *exactly*
the same problem I'm seeing in ITS#6540, that they might have been one and the
same.


> I'm sorry I went back through this thread and got somehow lost; what's the
> exact topic now?  I infer (from followup #14) that there seems to be an
> issue about chaining because slapo-chain seems not to rebind as expected. 


ITS#6540 had two parts:

 1. The test script failed to identify a specific set of conditions in which
slapo-chain would cause slapcat to fail and
also prevent slapd from being restarted after slapo-chain was enabled.

 2. The problem that the test script failed to detect with slapo-chain is
directly related to the current problem:
slapo-chain does not properly rebind as the appropriate user as set in the
olcDbIDAssertBind attribute.  Instead, it
rebinds as 'anonymous' (an empty DN), which is of very little practical use,
since unauthenticated anonymous users
should not be allowed to modify anything.  As a result, slapd kicks the referral
back to the client instead of chasing
it automatically, which completely defeats the purpose of using slapo-chain.


> Could you isolate the problem in a minimal setup (slapd.conf or the
> corresponding back-config's LDIF, a database and a simple operation that
> can be performed with ldapmodify/ldappasswd or so) to clearly reproduce
> the issue?
> 
> Thanks, p.
> 

The configuration and test operations below should show exactly what's going on,
but first, allow me to describe the
process in a nutshell:

1. The client binds to the slave and issues a modification request

2. The slave creates the referral and tries to chase it automatically on the
client's behalf to the upstream master
using slapo-chain

3. The automatic referral chasing fails because the DN is not passed through and
the slave erroneously rebinds to the
master as anonymous (an empty DN) instead of the identity it's configured to
assert, as set by the olcDbIDAssertBind
attribute.

4. Because the automatic referral chasing fails, the slave kicks back the
referral to the client, stating that the
client "is not logged in" to the master and that "modifications require
authentication".

5. When the client provides credentials for the referral (manual referral
chasing), the rest of the operation works as
expected (updates made on master, which then cascades to the slaves).

The following LDIFs and sample operations exemplify the problem; please let me
know if this example suffices, or if you
need a more complete reference or further clarification:

#################################################
#                CONFIGURATION                  #
#################################################

### back-config base entry on slave (abbreviated)
dn: cn=config
objectClass: olcGlobal
cn: config
olcConfigDir: /etc/ldap/slapd.d
olcReadOnly: FALSE
olcReverseLookup: FALSE
olcTLSCACertificateFile: /etc/ldap/ssl/certs/cacert.pem
olcTLSCertificateFile: /etc/ldap/ssl/certs/openldap.cert.pem
olcTLSCertificateKeyFile: /etc/ldap/ssl/keys/openldap.key.pem
olcAuthzPolicy: none
olcLogLevel: stats sync
olcPasswordHash: {SSHA}SUPERSECRET
olcServerID: 1 ldap://master1.example.com
olcServerID: 2 ldap://slave1.example.com

### back-config modules entry
dn: cn=module{0},cn=config
objectClass: olcModuleList
cn: module{0}
olcModulePath: /usr/lib/ldap
olcModuleLoad: {0}back_hdb.la
olcModuleLoad: {1}autogroup.la
olcModuleLoad: {2}syncprov.la
olcModuleLoad: {3}back_ldap.la

### back-config chain overlay entry
dn: olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcOverlayConfig
objectClass: olcChainConfig
olcOverlay: {0}chain

### back-config chain database
dn: olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcLDAPConfig
objectClass: olcChainDatabase
olcDatabase: {0}ldap
olcDbURI: ldap://master1.example.com
olcDbIDAssertBind: bindmethod=simple binddn="cn=admin,dc=example,dc=com"
credentials=SECRET mode=self


#################################################
#        SIMPLE OPERATIONAL EXAMPLE             #
#################################################

### NOTE: This example uses ldapvi, but results are identical t

Message of length 9239 truncated


Followup 19

Download message
Date: Fri, 8 Oct 2010 23:09:45 +0200 (CEST)
Subject: Re: (ITS#6540) test022-ppolicy is flawed,
           masks serious stability issue
From: masarati@aero.polimi.it
To: "Ryan Steele" <ryans@aweber.com>
Cc: openldap-its@openldap.org
> The configuration and test operations below should show exactly what's
> going on, but first, allow me to describe the
> process in a nutshell:
>
> 1. The client binds to the slave and issues a modification request
>
> 2. The slave creates the referral and tries to chase it automatically on
> the client's behalf to the upstream master
> using slapo-chain
>
> 3. The automatic referral chasing fails because the DN is not passed
> through and the slave erroneously rebinds to the
> master as anonymous (an empty DN) instead of the identity it's configured
> to assert, as set by the olcDbIDAssertBind
> attribute.
>
> 4. Because the automatic referral chasing fails, the slave kicks back the
> referral to the client, stating that the
> client "is not logged in" to the master and that "modifications require
> authentication".
>
> 5. When the client provides credentials for the referral (manual referral
> chasing), the rest of the operation works as
> expected (updates made on master, which then cascades to the slaves).

What the case description and the logs suggest is that the referral is
chased anonymously because the protocol/host/port portion of the URI in
the referral does not match the one configured in the chain-uri directive.
 In this case, slapo-chain defaults to anonymous.

What slapo-chain does is take the referral string, pass it to URL parsing
routines, isolate protocol, host and port, pass it back to URL unparsing,
and compare the resulting string with the one configured in chain-uri
(which had to go through the same "normalization" when configured).

Perhaps if you try with a more detailed log level you'll be able to see
the two strings and figure out if and how they differ.  Hope this helps.

p.


>
> The following LDIFs and sample operations exemplify the problem; please
> let me know if this example suffices, or if you
> need a more complete reference or further clarification:
>
> #################################################
> #                CONFIGURATION                  #
> #################################################
>
> ### back-config base entry on slave (abbreviated)
> dn: cn=config
> objectClass: olcGlobal
> cn: config
> olcConfigDir: /etc/ldap/slapd.d
> olcReadOnly: FALSE
> olcReverseLookup: FALSE
> olcTLSCACertificateFile: /etc/ldap/ssl/certs/cacert.pem
> olcTLSCertificateFile: /etc/ldap/ssl/certs/openldap.cert.pem
> olcTLSCertificateKeyFile: /etc/ldap/ssl/keys/openldap.key.pem
> olcAuthzPolicy: none
> olcLogLevel: stats sync
> olcPasswordHash: {SSHA}SUPERSECRET
> olcServerID: 1 ldap://master1.example.com
> olcServerID: 2 ldap://slave1.example.com
>
> ### back-config modules entry
> dn: cn=module{0},cn=config
> objectClass: olcModuleList
> cn: module{0}
> olcModulePath: /usr/lib/ldap
> olcModuleLoad: {0}back_hdb.la
> olcModuleLoad: {1}autogroup.la
> olcModuleLoad: {2}syncprov.la
> olcModuleLoad: {3}back_ldap.la
>
> ### back-config chain overlay entry
> dn: olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config
> objectClass: olcOverlayConfig
> objectClass: olcChainConfig
> olcOverlay: {0}chain
>
> ### back-config chain database
> dn:
> olcDatabase={0}ldap,olcOverlay={0}chain,olcDatabase={-1}frontend,cn=config
> objectClass: olcLDAPConfig
> objectClass: olcChainDatabase
> olcDatabase: {0}ldap
> olcDbURI: ldap://master1.example.com
> olcDbIDAssertBind: bindmethod=simple binddn="cn=admin,dc=example,dc=com"
> credentials=SECRET mode=self
>
>
> #################################################
> #        SIMPLE OPERATIONAL EXAMPLE             #
> #################################################
>
> ### NOTE: This example uses ldapvi, but results are identical to
> ldapmodify, etc.
> ### NOTE: The client binds initially to the slave as the admin here, but
> results are identical to scenarios in which the
> client binds as a regular user
>
> ### Attempting to modify 'displayColor' attribute belonging to entry
> 'uid=ryans,ou=Users,dc=example,dc=com'
> root@slave1:~# ldapvi -h localhost --bind=simple -D
> cn=admin,dc=example,dc=com -w `cat /etc/ldap.secret` --discover
>     159 entries read
>
> add: 0, rename: 0, modify: 1, delete: 0
> Action? [yYqQvVebB*rsf+?] y
> Received referral to
> ldap://master1.example.com/uid=ryans,ou=Users,dc=example,dc=com.
> You are not logged in to ldap://master1.example.com:389 yet.
> Type '!' or 'y' to do so.
> Rebind? [y!nB*qQ?] y
>
> --- Login
> Type M-h for help on key bindings.
>
> Filter or DN: cn=admin,dc=example,dc=com
>     Password: ***********
> Bound as cn=admin,dc=example,dc=com.
> Done.
>
> #################################################
> #        LOGS FOR OPERATIONAL EXAMPLE           #
> ##################

Message of length 8653 truncated

Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest


The OpenLDAP Issue Tracking System uses a hacked version of JitterBug

______________
© Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org