Full_Name: Chris Mikkelson Version: 2.4.32 OS: FreeBSD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (204.147.85.37) When a back-mdb database is closed, the POSIX semaphores created when the mdb database was opened are left behind. This prevents a different user from opening the database. For example, if you run slapd as an unprivileged user, stop slapd, and do an offline slapcat as root, slapd will not start up again as that unprivileged user. The only ways to recover I've found are: a) reboot the machine b) restore the database from the slapcat, ensuring that slapadd runs as the correct user, or c) write a small program to re-generate the semaphore names an remove them. The patch at: http://mikk.net/~chris/patches/0002-Remove-POSIX-semaphores-when-the-last-user-closes-th.patch Attempts to upgrade the lockfile lock to exclusive when closing the environment. If that upgrade succeeds, it removes the semaphores. Patch has been tested on a BSD system. WIN32 has not been tested.
cmikk@qwest.net wrote: > Full_Name: Chris Mikkelson > Version: 2.4.32 > OS: FreeBSD > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (204.147.85.37) > > > When a back-mdb database is closed, the POSIX semaphores created when the mdb > database was opened are left behind. This prevents a different user from opening > the database. > > For example, if you run slapd as an unprivileged user, stop slapd, and do an > offline slapcat as root, slapd will not start up again as that unprivileged > user. The only ways to recover I've found are: This sounds like a bug in your platform's sem_open() syscall. If you first started slapd as an unprivileged user, the semaphore should be owned by that user. Running slapcat as root should not change the semaphore owner uid, and the unprivileged owner should still be able to access the semaphore later. Either that, or your description of the bug scenario is incomplete. I won't commit your patch without a more complete understanding of the bug. > a) reboot the machine > b) restore the database from the slapcat, ensuring that slapadd runs as the > correct user, or > c) write a small program to re-generate the semaphore names an remove them. > > The patch at: > > http://mikk.net/~chris/patches/0002-Remove-POSIX-semaphores-when-the-last-user-closes-th.patch > > Attempts to upgrade the lockfile lock to exclusive when closing the environment. > If that upgrade succeeds, it removes the semaphores. > > Patch has been tested on a BSD system. WIN32 has not been tested. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
hyc@symas.com wrote: > cmikk@qwest.net wrote: >> Full_Name: Chris Mikkelson >> Version: 2.4.32 >> OS: FreeBSD >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (204.147.85.37) >> >> >> When a back-mdb database is closed, the POSIX semaphores created when the mdb >> database was opened are left behind. This prevents a different user from opening >> the database. >> >> For example, if you run slapd as an unprivileged user, stop slapd, and do an >> offline slapcat as root, slapd will not start up again as that unprivileged >> user. The only ways to recover I've found are: > > This sounds like a bug in your platform's sem_open() syscall. If you first > started slapd as an unprivileged user, the semaphore should be owned by that > user. Running slapcat as root should not change the semaphore owner uid, and > the unprivileged owner should still be able to access the semaphore later. My mistake. Your patch looks good, committed. Thanks. > > Either that, or your description of the bug scenario is incomplete. I won't > commit your patch without a more complete understanding of the bug. > >> a) reboot the machine >> b) restore the database from the slapcat, ensuring that slapadd runs as the >> correct user, or >> c) write a small program to re-generate the semaphore names an remove them. >> >> The patch at: >> >> http://mikk.net/~chris/patches/0002-Remove-POSIX-semaphores-when-the-last-user-closes-th.patch >> >> Attempts to upgrade the lockfile lock to exclusive when closing the environment. >> If that upgrade succeeds, it removes the semaphores. >> >> Patch has been tested on a BSD system. WIN32 has not been tested. > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
changed notes changed state Open to Test moved from Incoming to Software Bugs
On Wed, Aug 22, 2012 at 03:23:16PM -0700, Howard Chu wrote: > cmikk@qwest.net wrote: > > For example, if you run slapd as an unprivileged user, stop slapd, and do an > > offline slapcat as root, slapd will not start up again as that unprivileged > > user. [snip] > > This sounds like a bug in your platform's sem_open() syscall. If you first > started slapd as an unprivileged user, the semaphore should be owned by that > user. Running slapcat as root should not change the semaphore owner uid, and > the unprivileged owner should still be able to access the semaphore later. In the scenario described, slapd is not running when slapcat starts as root, so slapcat gets an exclusive lock on the lockfile, sem_unlink()s the semaphores created by the unprivileged slapd user, and recreates them, now owned by root. When slapcat finishes, it closes the database and does not remove the semaphores. When slapd starts up, it gets an exclusive lock, attempts to sem_unlink() the semaphores created as (and owned by) root, which fails. In timeline form: 1: slapcat gets exclusive lock on lock.mdb 2: slapcat (as root) unlinks /MDBrXXXX and /MDBwXXXX 3: slapcat (as root) creates /MDBrXXXX and /MDBwXXXX 4: slapcat exits, releasing lock on lock.mdb, leaving behind /MDBrXXXX and /MDBwXXXX (owned by root) 5: slapd gets exclusive lock on lock.mdb 6: slapd (as unpriv) unlinks /MDBrXXXX and /MDBwXXXX. This fails because of the semaphores are owned by root. 7: slapd fails startup. Thanks, -- Chris Mikkelson | "Unfortunately, simplicity is a complicated mess cmikk@qwest.net | of a concept." --Taner Edis
changed notes changed state Test to Release
cmikk@qwest.net writes: > The patch at: > > http://mikk.net/~chris/patches/0002-Remove-POSIX-semaphores-when-the-last-user-closes-th.patch > > Attempts to upgrade the lockfile lock to exclusive when closing the environment. > If that upgrade succeeds, it removes the semaphores. That won't help if the mdb proccess crashes. Sounds like mdb needs some cleanup API calls. Maybe one which does the same as your patch after opening the environment - except it does not create any files, and fails if the exclusive lock fails. -- Hallvard
h.b.furuseth@usit.uio.no wrote: > cmikk@qwest.net writes: >> The patch at: >> >> http://mikk.net/~chris/patches/0002-Remove-POSIX-semaphores-when-the-last-user-closes-th.patch >> >> Attempts to upgrade the lockfile lock to exclusive when closing the environment. >> If that upgrade succeeds, it removes the semaphores. > > That won't help if the mdb proccess crashes. If which process crashes? If some other process crashes then this lock will succeed and remove the semaphores. > Sounds like mdb needs some > cleanup API calls. Maybe one which does the same as your patch after > opening the environment - except it does not create any files, and fails > if the exclusive lock fails. Sounds OK. Could add an invocation of it as an option for mdb_stat. mdb_stat needs to be extended anyway. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Howard Chu writes: > h.b.furuseth@usit.uio.no wrote: >> That won't help if the mdb proccess crashes. > > If which process crashes? Sorry, I meant the process which would have removed the semaphore. -- Hallvard
Reopening this. This is worse with a database with is intended to be used by different users (A and B): A opens the DB and creates the semaphores with e.g. mode 0660. B opens it, A closes, B closes - and fails sem_unlink() which only A and root can do. Next, B (or C) fails mdb_env_open() because sem_unlink() fails again. The work-around I can think of is a "multi-uid" mode which instead resets the semaphore with sem_post() if sem_getvalue() returns 0. I don't know how ugly that is considered to be. Could ask comp.programming.unix, or check what Berkeley DB does. This mode should use mode 0666 for the semaphores (temporarily setting umask 0, yuck), or it should not sem_unlink() since next user may create the semaphores with a group which gives the wrong users access. Or root may give it root-only access, as in the original report. If not unlinking, we need a special "remove database + semaphores" API call. Or use some other sync primitive, like file locks. The Linux manual says these work over NFS though, which sounds like they must be rather slow. flock() does not, but this time mdb would need to seek first. I don't know which sync variants BSD offers. Other matters with the current implementation - I'll patch these: mdb_env_excl_lock() need not retry getting a non-exclusive lock when closing. mdb_env_close() can pass *excl = -1 to tell it not to. mdb_env_setup_locks() can sem_unlink both semaphores before doing anything else, so that reopening a database as root will clean up. Drop the error checks of sem_unlink (so both get called), instead use O_EXCL in sem_open(,O_CREAT,,). Unless I'm missing something, the error checks just work like an emulation of O_EXCL anyway.
changed notes changed state Release to Open
h.b.furuseth@usit.uio.no wrote: > Reopening this. > > > This is worse with a database with is intended to be used by > different users (A and B): This is pretty much never a use case we would worry about. In most applications, a single userID creates and operates on the DB. > A opens the DB and creates the semaphores with e.g. mode 0660. > B opens it, A closes, B closes - and fails sem_unlink() which > only A and root can do. > > Next, B (or C) fails mdb_env_open() because sem_unlink() fails > again. > > The work-around I can think of is a "multi-uid" mode which instead > resets the semaphore with sem_post() if sem_getvalue() returns 0. > I don't know how ugly that is considered to be. Could ask > comp.programming.unix, or check what Berkeley DB does. That would be OK in general. It still leaves the question of how to remove the semaphore if the DB is being destroyed. But it's probably not worth the trouble to worry about this so much. Those OSes should just get their act together and support the POSIX process-shared mutexes. > This mode > should use mode 0666 for the semaphores (temporarily setting umask > 0, yuck), Definitely not. The caller specifies a mode; if they want 0666 they should configure it as such. > or it should not sem_unlink() since next user may create > the semaphores with a group which gives the wrong users access. Same as today where running slapadd with the wrong uid causes trouble for the following slapd. The answer is obvious: use the right uid when accessing the DB. > Other matters with the current implementation - I'll patch these: > > mdb_env_excl_lock() need not retry getting a non-exclusive lock when > closing. mdb_env_close() can pass *excl = -1 to tell it not to. > > mdb_env_setup_locks() can sem_unlink both semaphores before doing > anything else, so that reopening a database as root will clean up. > Drop the error checks of sem_unlink (so both get called), instead > use O_EXCL in sem_open(,O_CREAT,,). Unless I'm missing something, > the error checks just work like an emulation of O_EXCL anyway. The sem_unlink() and sem_open() sequence isn't ideal, certainly. I would prefer to just use the existing semaphore. Another possibility is to just use SysV semaphores instead of POSIX semaphores. Then you can use the ipcs(1) command to manually cleanup. BerkeleyDB uses SysV shared memory when you specify a shared memory environment and it appears that SysV semaphore support is actually more widespread than POSIX semaphores. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
-----Original Message----- From: openldap-bugs-bounces@OpenLDAP.org [mailto:openldap-bugs-bounces@OpenLDAP.org] On Behalf Of hyc@symas.com >> Another possibility is to just use SysV semaphores instead of POSIX semaphores. >> Then you can use the ipcs(1) command to manually cleanup. BerkeleyDB uses >> SysV shared memory when you specify a shared memory environment and it >> appears that SysV semaphore support is actually more widespread than POSIX semaphores. Just to mention that at least on HPUX, Posix semaphores are more efficient than SysV ones. ++Cyrille
Maucci, Cyrille wrote: > -----Original Message----- > From: openldap-bugs-bounces@OpenLDAP.org [mailto:openldap-bugs-bounces@OpenLDAP.org] On Behalf Of hyc@symas.com >>> Another possibility is to just use SysV semaphores instead of POSIX semaphores. >>> Then you can use the ipcs(1) command to manually cleanup. BerkeleyDB uses >>> SysV shared memory when you specify a shared memory environment and it >>> appears that SysV semaphore support is actually more widespread than POSIX semaphores. > > Just to mention that at least on HPUX, Posix semaphores are more efficient than SysV ones. I'm also reminded that there is no defined behavior for SysV semaphores in threads, they are only speciried for inter-process synchronization. So forget that... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Howard Chu writes: > h.b.furuseth@usit.uio.no wrote: >> Reopening this. >> >> >> This is worse with a database with is intended to be used by >> different users (A and B): > > This is pretty much never a use case we would worry about. In most > applications, a single userID creates and operates on the DB. I'm fine with just documenting that as a restriction on some systems. If not: >> (...) >> The work-around I can think of is a "multi-uid" mode which instead >> resets the semaphore with sem_post() if sem_getvalue() returns 0. >> I don't know how ugly that is considered to be. Could ask >> comp.programming.unix, or check what Berkeley DB does. > > That would be OK in general. It still leaves the question of how to remove the > semaphore if the DB is being destroyed. But it's probably not worth the > trouble to worry about this so much. Those OSes should just get their act > together and support the POSIX process-shared mutexes. > >> This mode >> should use mode 0666 for the semaphores (temporarily setting umask >> 0, yuck), > > Definitely not. The caller specifies a mode; if they want 0666 they > should configure it as such. 0666 would likely be wrong for the database file(s). But this flag could just as well consist of specifying a mode parameter for the semaphores. It's a threaded library doing umask() I dislike. And, as you say, the need to remove the semaphore afterwards. >> or it should not sem_unlink() since next user may create >> the semaphores with a group which gives the wrong users access. > > Same as today where running slapadd with the wrong uid causes trouble > for the following slapd. The answer is obvious: use the right uid when > accessing the DB. We're talking about the case where there is no single "right" uid. Not relevant for slapd, only libmdb by itself. >> Other matters with the current implementation - I'll patch these: >> >> mdb_env_excl_lock() need not retry getting a non-exclusive lock when >> closing. mdb_env_close() can pass *excl = -1 to tell it not to. >> >> mdb_env_setup_locks() can sem_unlink both semaphores before doing >> anything else, so that reopening a database as root will clean up. >> Drop the error checks of sem_unlink (so both get called), instead >> use O_EXCL in sem_open(,O_CREAT,,). Unless I'm missing something, >> the error checks just work like an emulation of O_EXCL anyway. > > The sem_unlink() and sem_open() sequence isn't ideal, certainly. I would > prefer to just use the existing semaphore. ...followed by 'if (sem_getvalue() shows 0) sem_post()' as above, then. -- Hallvard
Howard Chu writes: >h.b.furuseth@usit.uio.no wrote: >> This is worse with a database with is intended to be used by >> different users (A and B): > (...) >> mdb_env_setup_locks() can sem_unlink both semaphores before doing >> anything else, so that reopening a database as root will clean up. >> Drop the error checks of sem_unlink (so both get called), instead >> use O_EXCL in sem_open(,O_CREAT,,). Unless I'm missing something, >> the error checks just work like an emulation of O_EXCL anyway. > > The sem_unlink() and sem_open() sequence isn't ideal, certainly. I would > prefer to just use the existing semaphore. I just realized reusing the semaphores is a security problem: Anyone who can compute/read the semaphore names, can create the semaphores owned by themselves and then muck with them to sabotage someone else's mdb. I suppose the same goes for SysV semaphores. I don't know what the accepted way to deal with that is, but I suggest to unlink + O_EXCL as above for now, document the multi-uid restriction, and punt the rest until anyone who cares shows up. > Another possibility is to just use SysV semaphores instead of POSIX > semaphores. Then you can use the ipcs(1) command to manually cleanup. > (...) [later] > I'm also reminded that there is no defined behavior for SysV > semaphores in threads, they are only speciried for inter-process > synchronization. So forget that... I expect the work-around is to use them for IPC only, and wrap them in process-local mutexes in the MDB_env for thread sync. -- Hallvard
libmdb now removes semaphores more thoroughly - at init, and after ITS#7377 at exit failure. Anything else - like SysV semaphores - can wait until someone shows up who cares about it. [Re-sent after originally sending to wrong ITS: #7363] -- Hallvard
partial fix in master partial fix in RE24