Full_Name: Ralf Haferkamp Version: HEAD, RE24 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (89.166.134.124) When slapd is shutdown while the online indexing task is active it will stop the indexing task immediately leaving a half created index files behind. Searches that use the index afterwards will return imcomplete results. I wonder if we should wait for the indexing task to complete when shutting down the daemon or if there are any other ways to put the databases and the configuration back into a consistant state. (Apart from manually running slapindex)
On Tue, Dec 02, 2008 at 03:14:28PM +0000, rhafer@suse.de wrote: > Full_Name: Ralf Haferkamp > Version: HEAD, RE24 > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (89.166.134.124) > > > When slapd is shutdown while the online indexing task is active it will stop the > indexing task immediately leaving a half created index files behind. Searches > that use the index afterwards will return imcomplete results. > > I wonder if we should wait for the indexing task to complete when shutting down > the daemon or if there are any other ways to put the databases and the > configuration back into a consistant state. (Apart from manually running > slapindex) That's documented behavior. (See the Admin Guide: If slapd is stopped before the index task completes, indexing will have to be manually completed using the slapindex tool. Generally when people issue a shutdown command to slapd they want it to Stop Right Now. I think it would be a bad idea to wait, and I see nothing to change here. This ITS should be closed.
Am Mittwoch 03 Dezember 2008 04:10:36 schrieb hyc@symas.com: > On Tue, Dec 02, 2008 at 03:14:28PM +0000, rhafer@suse.de wrote: > > Full_Name: Ralf Haferkamp > > Version: HEAD, RE24 > > OS: > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (89.166.134.124) > > > > > > When slapd is shutdown while the online indexing task is active it will > > stop the indexing task immediately leaving a half created index files > > behind. Searches that use the index afterwards will return imcomplete > > results. > > > > I wonder if we should wait for the indexing task to complete when > > shutting down the daemon or if there are any other ways to put the > > databases and the configuration back into a consistant state. (Apart from > > manually running slapindex) > > That's documented behavior. (See the Admin Guide: > If slapd is stopped before the index task completes, indexing > will have to be manually completed using the slapindex tool. > > Generally when people issue a shutdown command to slapd they want it to > Stop Right Now. I think it would be a bad idea to wait, and I see nothing > to change here. This ITS should be closed. Ok, waiting for the index task to complete is a bad idea as it can take very long to complete. But could we indicate the condition somehow that warn the user about an incomplete run upon next startup? -- Ralf
rhafer@suse.de wrote: > Am Mittwoch 03 Dezember 2008 04:10:36 schrieb hyc@symas.com: >> On Tue, Dec 02, 2008 at 03:14:28PM +0000, rhafer@suse.de wrote: >>> Full_Name: Ralf Haferkamp >>> Version: HEAD, RE24 >>> OS: >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (89.166.134.124) >>> >>> >>> When slapd is shutdown while the online indexing task is active it will >>> stop the indexing task immediately leaving a half created index files >>> behind. Searches that use the index afterwards will return imcomplete >>> results. >>> >>> I wonder if we should wait for the indexing task to complete when >>> shutting down the daemon or if there are any other ways to put the >>> databases and the configuration back into a consistant state. (Apart from >>> manually running slapindex) >> That's documented behavior. (See the Admin Guide: >> If slapd is stopped before the index task completes, indexing >> will have to be manually completed using the slapindex tool. >> >> Generally when people issue a shutdown command to slapd they want it to >> Stop Right Now. I think it would be a bad idea to wait, and I see nothing >> to change here. This ITS should be closed. > Ok, waiting for the index task to complete is a bad idea as it can take very > long to complete. But could we indicate the condition somehow that warn the > user about an incomplete run upon next startup? - the index should be invalidated - slapd, at the next manual startup, should complain about the need to run slapindex first. - in order to allow consistent batch restart with no human intervention, slapd could allow to be restarted without running slapindex first, but without making use of the inconsistent index. my 2c, not volunteering right now. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando@sys-net.it -----------------------------------
ando@sys-net.it writes: > - the index should be invalidated > - slapd, at the next manual startup, should complain about the need to > run slapindex first. If it knows enough to complain, why not restart the indexing task? -- Hallvard
h.b.furuseth@usit.uio.no wrote: > ando@sys-net.it writes: >> - the index should be invalidated >> - slapd, at the next manual startup, should complain about the need to >> run slapindex first. > > If it knows enough to complain, why not restart the indexing task? There should be an option in doing that. In fact, I suspect running slapindex -q can be much more efficient than letting slapd index while serving without index. Of course, slapd cannot serve while slapindex'ing, so I think human intervention is needed to determine what's most appropriate on a case by case basis. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando@sys-net.it -----------------------------------
ando@sys-net.it wrote: > h.b.furuseth@usit.uio.no wrote: >> ando@sys-net.it writes: >>> - the index should be invalidated >>> - slapd, at the next manual startup, should complain about the need to >>> run slapindex first. >> If it knows enough to complain, why not restart the indexing task? > > There should be an option in doing that. In fact, I suspect running > slapindex -q can be much more efficient than letting slapd index while > serving without index. Of course, slapd cannot serve while > slapindex'ing, so I think human intervention is needed to determine > what's most appropriate on a case by case basis. I suppose we should save the last entryID that was indexed at shutdown time, so that the indexer task can resume at the next startup. We'd also need some coordination with slapindex, so that if you ran slapindex after shutdown, it would erase the saved status. Rather than add any fields to the DB index files, it would probably be OK to write the status in the cn=config tree. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
hyc@symas.com writes: >ando@sys-net.it wrote: >>h.b.furuseth@usit.uio.no wrote: >>> If it knows enough to complain, why not restart the indexing task? >> >> There should be an option in doing that. I don't quite see why. If you add an index via cn=config, that starts an indexing task while slapd is running. A following shutdown-restart is not necessarily related to a desire to not run that task. >> In fact, I suspect running slapindex -q can be much more efficient >> than letting slapd index while serving without index. (...) If you shut down slapd because you want to use slapindex, then you know what you want - so just go ahead and use slapindex. No extra flag needed. I suppose there could be a flag to _not_ start the indexing task when you add an index via cn=config, so you always must slapindex. > I suppose we should save the last entryID that was indexed at shutdown time, > so that the indexer task can resume at the next startup. We'd also need some > coordination with slapindex, so that if you ran slapindex after shutdown, it > would erase the saved status. Rather than add any fields to the DB index > files, it would probably be OK to write the status in the cn=config tree. Or a status file in the database directory. A cn=config attribute would make it messy to swap database directories around, since one must also modify cn=config when doing so. -- Hallvard
Hallvard B Furuseth wrote: > hyc@symas.com writes: > I suppose there could be a flag to _not_ start the indexing task when > you add an index via cn=config, so you always must slapindex. I'd rather not. >> I suppose we should save the last entryID that was indexed at shutdown time, >> so that the indexer task can resume at the next startup. We'd also need some >> coordination with slapindex, so that if you ran slapindex after shutdown, it >> would erase the saved status. Rather than add any fields to the DB index >> files, it would probably be OK to write the status in the cn=config tree. > > Or a status file in the database directory. > > A cn=config attribute would make it messy to swap database directories > around, since one must also modify cn=config when doing so. OK, a status file. It would record the old and new index masks for the attributes being reindexed, and the entryID where it stopped. If slapindex is run without a specific list of attributes, (reindex everything) it should remove and ignore the status file. If slapindex is run with a specific list of attributes, and those attributes are in the status file, it should use the saved status and remove the specified attributes from the status file. (Deleting the file if there are no other attributes remaining.) Otherwise, when slapd starts, it should read the status file and resume the indexing task. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
moved from Incoming to Development
I believe this affects back-mdb as well, so leaving open
Is this resolved with ITS#8958?
(In reply to Ondřej Kuzník from comment #11) > Is this resolved with ITS#8958? Yes *** This issue has been marked as a duplicate of issue 8958 ***