[Date Prev][Date Next]
RE:LMBD questions (stat & copy)
Thank you very much for your patient answers, and my apologies for my late reply.
The "reserved space" was indeed occupied by the FREE_DBI, thanks again for pointing this out for me.
Since I will have hundreds of envs (of unpredictable size) in my application, I am a bit reluctant in allocating too much space for all of them, and that's why I was trying to find a way emit a warning when room is about to lack. In this regard I think I have read something very promising in the docs: the mt_next_pgno field of the txn. It looks perfect. Do you think it would be safe to expose it in a read-only way?
In any case I will follow you advice and try to find a safe compromise for the map size.
With best regards,
De : openldap-technical [firstname.lastname@example.org] de la part de Howard Chu [email@example.com]
Envoyé : jeudi 7 janvier 2016 22:25
À : Bruno Freudensprung; firstname.lastname@example.org
Objet : Re: LMBD questions (stat & copy)
Bruno Freudensprung wrote:
> I am still experimenting on this area and noticed that I can actually put more
> data in LMDB by commiting after each put (for the sake of the test) than by
> doing one commit in the middle of the data set.
> In the end, in terms of amount of data inserted into the database, the big
> picture is:
> 1 commit at the end > 1 commit after each put > 1 commit in the middle of
> the data set + 1 commit at the end
> And mdb_stat can't really help anticipating MDB_MAP_FULL (maybe that wasn't
> its purpose).
The docs tell you to configure a large mapsize and go about your business.
Doing anything else is a waste of time.
> Anyway, if anybody had a hint about the maximum "reserved pages" (for internal
> purpose) that I could subtract from the map size, I would be very grateful to
> hear about it.
Your sample code gives useless results because you're only counting pages in
the Main DB. As plainly stated in the LMDB documentation, there is also a Free
DB which is used to track free pages, and it obviously consumes pages itself
in order to store the free page info.
> With best regards,
> Bruno (lmdb enthousiast).
> *De :* openldap-technical [email@example.com] de la part
> de Bruno Freudensprung [firstname.lastname@example.org]
> *Envoyé :* jeudi 31 décembre 2015 10:02
> *À :* email@example.com
> *Objet :* LMBD questions (stat & copy)
> I am trying to detect MDB_MAP_FULL based on mdb_stat(txn, dbi, &stat) results
> and I am wondering if I am on the right track or doing things correctly.
> I have a small and simple test program (http://pastebin.com/SPCYgWMC)
> exhibiting the following behavior (consistent between Windows 7 64-bit and
> Ubuntu 15.10 64-bit - latest version of lmdb cloned and recompiled on Linux):
> * Inserting 20990 key value pairs in a database is OK when using one
> transaction. Just before commiting the write transaction, mdb_stat(txn,
> dbi, &stat) shows that the total number of pages (253) is very close the
> the number of pages of the map (256). Indeed, adding another entry leads
> to MDB_MAP_FULL. To put it shortly, this scenario looks OK (I am assuming
> there might be 3 "internal-purpose" pages not shown by mdb_stat) and can
> be tested by setting the stat_test variable to 0 at line 20 of the test
> * Inserting same 20990 key value pairs in a database fails (MDB_MAP_FULL
> encountered at 19357) when using two consecutive write transactions. Just
> before MDB_MAP_FULL, mdb_stat(txn, dbi, &stat) shows that the total number
> of pages (232) is "quite far" from the the number of pages of the map
> (256), leading my MDB_MAP_FULL detection heuristic to fail. It means that
> only 92% of the total amount of data can be sucessfully added to the
> database. Other scenarii (with same key/value lengths though) stop at 85%
> only. This can be tested by setting the stat_test variable to 1 (line 20).
> By "total number of pages" I mean ms_branch_pages + ms_leaf_pages +
> Do you think I am on a wrong track? (maybe mdb_stat does not show 24
> "internal-purpose" pages in the second case? or I am forgetting something?).
> One last remark: it seems that mdb_env_copy(db, copy_dir) is possible during a
> write transaction on Windows (like stated in the documentation), but not on
> Linux (the test program seems to be stopped on a mutex at line 62). This can
> be tested by changing the copy_test variable to 1 at line 19. Is it expected
> given my program?
> Best regards,
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/