[Date Prev][Date Next]
Re: LMBD questions (stat & copy)
- To: Bruno Freudensprung <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>
- Subject: Re: LMBD questions (stat & copy)
- From: Howard Chu <email@example.com>
- Date: Thu, 7 Jan 2016 21:25:13 +0000
- In-reply-to: <00E3569E6FC51948817E09FD3696983F0981F5C7@EXMBX01.hosting.local>
- References: <00E3569E6FC51948817E09FD3696983F0981E950@EXMBX01.hosting.local> <00E3569E6FC51948817E09FD3696983F0981F5C7@EXMBX01.hosting.local>
- User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0 SeaMonkey/2.42a1
Bruno Freudensprung wrote:
I am still experimenting on this area and noticed that I can actually put more
data in LMDB by commiting after each put (for the sake of the test) than by
doing one commit in the middle of the data set.
In the end, in terms of amount of data inserted into the database, the big
1 commit at the end > 1 commit after each put > 1 commit in the middle of
the data set + 1 commit at the end
And mdb_stat can't really help anticipating MDB_MAP_FULL (maybe that wasn't
The docs tell you to configure a large mapsize and go about your business.
Doing anything else is a waste of time.
Anyway, if anybody had a hint about the maximum "reserved pages" (for internal
purpose) that I could subtract from the map size, I would be very grateful to
hear about it.
Your sample code gives useless results because you're only counting pages in
the Main DB. As plainly stated in the LMDB documentation, there is also a Free
DB which is used to track free pages, and it obviously consumes pages itself
in order to store the free page info.
With best regards,
Bruno (lmdb enthousiast).
*De :* openldap-technical [firstname.lastname@example.org] de la part
de Bruno Freudensprung [email@example.com]
*Envoyé :* jeudi 31 décembre 2015 10:02
*À :* firstname.lastname@example.org
*Objet :* LMBD questions (stat & copy)
I am trying to detect MDB_MAP_FULL based on mdb_stat(txn, dbi, &stat) results
and I am wondering if I am on the right track or doing things correctly.
I have a small and simple test program (http://pastebin.com/SPCYgWMC)
exhibiting the following behavior (consistent between Windows 7 64-bit and
Ubuntu 15.10 64-bit - latest version of lmdb cloned and recompiled on Linux):
* Inserting 20990 key value pairs in a database is OK when using one
transaction. Just before commiting the write transaction, mdb_stat(txn,
dbi, &stat) shows that the total number of pages (253) is very close the
the number of pages of the map (256). Indeed, adding another entry leads
to MDB_MAP_FULL. To put it shortly, this scenario looks OK (I am assuming
there might be 3 "internal-purpose" pages not shown by mdb_stat) and can
be tested by setting the stat_test variable to 0 at line 20 of the test
* Inserting same 20990 key value pairs in a database fails (MDB_MAP_FULL
encountered at 19357) when using two consecutive write transactions. Just
before MDB_MAP_FULL, mdb_stat(txn, dbi, &stat) shows that the total number
of pages (232) is "quite far" from the the number of pages of the map
(256), leading my MDB_MAP_FULL detection heuristic to fail. It means that
only 92% of the total amount of data can be sucessfully added to the
database. Other scenarii (with same key/value lengths though) stop at 85%
only. This can be tested by setting the stat_test variable to 1 (line 20).
By "total number of pages" I mean ms_branch_pages + ms_leaf_pages +
Do you think I am on a wrong track? (maybe mdb_stat does not show 24
"internal-purpose" pages in the second case? or I am forgetting something?).
One last remark: it seems that mdb_env_copy(db, copy_dir) is possible during a
write transaction on Windows (like stated in the documentation), but not on
Linux (the test program seems to be stopped on a mutex at line 62). This can
be tested by changing the copy_test variable to 1 at line 19. Is it expected
given my program?
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/