Full_Name: Andrew Schorr Version: lmdb 0.9.18 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (38.76.0.51) I patched mdb_load to add an option to set the mapsize with -T, as per open issue #8417. I then used it to load a 7 GB text file, and I found that the calls to mdb_txn_commit after every 100 mdb_cursor_put calls really slows down performance. If one comments out the batching, it goes much faster. And even faster is to avoid the cursor and just call mdb_put in a loop. Is there a reason mdb_load commits after every 100 puts?
Hi, sorry, please close this issue. I am unable to duplicate my initial results. The use of a cursor seems to be marginally helpful, and I no longer see any slowdown from batching the commits. I guess my server must have had some other loads on it that impacted my previous test results. I do see one odd result of batching -- the process ru_maxrss value reported by getrusage is much higher (10 GB vs 3 GB for my test data). I'm not sure why that's the case. Perhaps that resulted in some swapping during my initial test. Sorry for wasting your time. Regards, Andy On Thu, May 05, 2016 at 09:56:10PM +0000, openldap-its@OpenLDAP.org wrote: > > *** THIS IS AN AUTOMATICALLY GENERATED REPLY *** > > Thanks for your report to the OpenLDAP Issue Tracking System. Your > report has been assigned the tracking number ITS#8418. > > One of our support engineers will look at your report in due course. > Note that this may take some time because our support engineers > are volunteers. They only work on OpenLDAP when they have spare > time. > > If you need to provide additional information in regards to your > issue report, you may do so by replying to this message. Note that > any mail sent to openldap-its@openldap.org with (ITS#8418) > in the subject will automatically be attached to the issue report. > > mailto:openldap-its@openldap.org?subject=(ITS#8418) > > You may follow the progress of this report by loading the following > URL in a web browser: > http://www.OpenLDAP.org/its/index.cgi?findid=8418 > > Please remember to retain your issue tracking number (ITS#8418) > on any further messages you send to us regarding this report. If > you don't then you'll just waste our time and yours because we > won't be able to properly track the report. > > Please note that the Issue Tracking System is not intended to > be used to seek help in the proper use of OpenLDAP Software. > Such requests will be closed. > > OpenLDAP Software is user supported. > http://www.OpenLDAP.org/support/ > > -------------- > Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved. > -- Andrew Schorr e-mail: aschorr@telemetry-investments.com Telemetry Investments, L.L.C. phone: 917-305-1748 545 Fifth Ave, Suite 1108 fax: 212-425-5550 New York, NY 10017-3630
changed state Open to Closed