[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: LMDB: MDB_MAP_FULL doesn't allow deletions
That makes sense, but I'm also seeing a similar problem with the freelist entries. Once I start pruning (btw I'm not deleting in a single commit -- my granularity is also 50 deletes per transaction, then I commit and reset the cursor before continuing), the freepages freezes at a certain value and doesn't refresh until some critical point is reached.
When my DB pages hit 80%:
Freelist:
Entries: 3
Free pages: 157
Main DB
Tree depth: 3
Branch pages: 33
Leaf pages: 6521
Overflow pages: 0
Entries: 611087
Then I waited for the DB pages - free pages to hit 80%:
Freelist:
Entries: 5
Free pages: 78
Main DB:
Tree depth: 3
Branch pages: 34
Leaf pages: 6599
Overflow pages: 0
And now I'm in this purge loop again, where my Leaf Pages on the main DB remain stable at 6599, and my free pages remain at 78 (5 entries), while the number of entries in my main DB is being reduced by ~20000 each purge cycle. Puts and dels are getting committed at least every 50 transactions, and I'm closing and reopening the entire environment after every purge.
I'm using the same technique as in mdb_stat.c to determine the free pages, and running mdb_stat.c on the db while I'm in the debugger confirms my numbers. So it really looks like something isn't getting updated, but I'm not really able to judge.
Maybe this isn't worth the trouble, and I should just backup the full base and start a new one, but that feels like surrender to me. :-)
Thanks for any insight, much appreciated,
Jeremy
Am 12.06.2013 um 18:43 schrieb Howard Chu <hyc@symas.com>:
>> Does that make any sense/ring any bells? Thanks again for your help so far.
>
> That's normal for a B+tree with multiple nodes on a page. And if you're only looking at mdb_stat() of the main DB, you're only seeing part of the picture. Again, look at the output of the mdb_stat command line tool. Use "mdb_stat -ef" so you also see the freelist info.
>
> The trick, in your case, is to make sure that the number of free pages is always sufficient, and also the number number of freelist entries. A freelist entry is created by a single commit, and you want to always have at least 3 of them (because the 2 most recent ones are not allowed to be used). If you do all of your deletes in a single commit you will not free up usable space as quickly as doing them in several commits.