[Date Prev][Date Next] [Chronological] [Thread] [Top]

RE:LMBD questions (stat & copy)

To: Howard Chu <hyc@symas.com>, "openldap-technical@openldap.org" <openldap-technical@openldap.org>
Subject: RE:LMBD questions (stat & copy)
From: Bruno Freudensprung <bruno.freudensprung@temis.com>
Date: Wed, 13 Jan 2016 20:15:10 +0000
Accept-language: fr-FR, en-US
Content-language: fr-FR
In-reply-to: <568ED7B9.8010500@symas.com>
References: <00E3569E6FC51948817E09FD3696983F0981E950@EXMBX01.hosting.local> <00E3569E6FC51948817E09FD3696983F0981F5C7@EXMBX01.hosting.local>, <568ED7B9.8010500@symas.com>
Thread-index: AdFDqMg6LcQv+SBCRsWy6YjAuJnnqwEShkoXAGWocIABLKjp6g==
Thread-topic: LMBD questions (stat & copy)

Hi Howard,

Thank you very much for your patient answers, and my apologies for my late reply.
The "reserved space" was indeed occupied by the FREE_DBI, thanks again for pointing this out for me.
Since I will have hundreds of envs (of unpredictable size) in my application, I am a bit reluctant in allocating too much space for all of them, and that's why I was trying to find a way emit a warning when room is about to lack. In this regard I think I have read something very promising in the docs: the mt_next_pgno field of the txn. It looks perfect. Do you think it would be safe to expose it in a read-only way?
In any case I will follow you advice and try to find a safe compromise for the map size.

With best regards,

Bruno.
________________________________________
De : openldap-technical [openldap-technical-bounces@openldap.org] de la part de Howard Chu [hyc@symas.com]
Envoyé : jeudi 7 janvier 2016 22:25
À : Bruno Freudensprung; openldap-technical@openldap.org
Objet : Re: LMBD questions (stat & copy)

Bruno Freudensprung wrote:
>
> Hi,
>
> I am still experimenting on this area and noticed that I can actually put more
> data in LMDB by commiting after each put (for the sake of the test) than by
> doing one commit in the middle of the data set.
> In the end, in terms of amount of data inserted into the database, the big
> picture is:
>
>     1 commit at the end > 1 commit after each put > 1 commit in the middle of
> the data set + 1 commit at the end
>
> And mdb_stat can't really help anticipating MDB_MAP_FULL (maybe that wasn't
> its purpose).

The docs tell you to configure a large mapsize and go about your business.
Doing anything else is a waste of time.

> Anyway, if anybody had a hint about the maximum "reserved pages" (for internal
> purpose) that I could subtract from the map size, I would be very grateful to
> hear about it.

Your sample code gives useless results because you're only counting pages in
the Main DB. As plainly stated in the LMDB documentation, there is also a Free
DB which is used to track free pages, and it obviously consumes pages itself
in order to store the free page info.
>
> With best regards,
>
> Bruno (lmdb enthousiast).
>
> ------------------------------------------------------------------------------
> *De :* openldap-technical [openldap-technical-bounces@openldap.org] de la part
> de Bruno Freudensprung [bruno.freudensprung@temis.com]
> *Envoyé :* jeudi 31 décembre 2015 10:02
> *À :* openldap-technical@openldap.org
> *Objet :* LMBD questions (stat & copy)
>
> Hi,
>
> I am trying to detect MDB_MAP_FULL based on mdb_stat(txn, dbi, &stat) results
> and I am wondering if I am on the right track or doing things correctly.
> I have a small and simple test program (http://pastebin.com/SPCYgWMC)
> exhibiting the following behavior (consistent between Windows 7 64-bit and
> Ubuntu 15.10 64-bit - latest version of lmdb cloned and recompiled on Linux):
>
>   * Inserting 20990 key value pairs in a database is OK when using one
>     transaction. Just before commiting the write transaction, mdb_stat(txn,
>     dbi, &stat) shows that the total number of pages (253) is very close the
>     the number of pages of the map (256). Indeed, adding another entry leads
>     to MDB_MAP_FULL. To put it shortly, this scenario looks OK (I am assuming
>     there might be 3 "internal-purpose" pages not shown by mdb_stat) and can
>     be tested by setting the stat_test variable to 0 at line 20 of the test
>     program.
>   * Inserting same 20990 key value pairs in a database fails (MDB_MAP_FULL
>     encountered at 19357) when using two consecutive write transactions. Just
>     before MDB_MAP_FULL, mdb_stat(txn, dbi, &stat) shows that the total number
>     of pages (232) is "quite far" from the the number of pages of the map
>     (256), leading my MDB_MAP_FULL detection heuristic to fail. It means that
>     only 92% of the total amount of data can be sucessfully added to the
>     database. Other scenarii (with same key/value lengths though) stop at 85%
>     only. This can be tested by setting the stat_test variable to 1 (line 20).
>
> By "total number of pages" I mean ms_branch_pages + ms_leaf_pages +
> ms_overflow_pages.
>
> Do you think I am on a wrong track? (maybe mdb_stat does not show 24
> "internal-purpose" pages in the second case? or I am forgetting something?).
>
> One last remark: it seems that mdb_env_copy(db, copy_dir) is possible during a
> write transaction on Windows (like stated in the documentation), but not on
> Linux (the test program seems to be stopped on a mutex at line 62). This can
> be tested by changing the copy_test variable to 1 at line 19. Is it expected
> given my program?
>
> Best regards,
>
> Bruno.
>


--
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Follow-Ups:
- Re: LMBD questions (stat & copy)
  - From: Howard Chu <hyc@symas.com>

References:
- RE:LMBD questions (stat & copy)
  - From: Bruno Freudensprung <bruno.freudensprung@temis.com>
- Re: LMBD questions (stat & copy)
  - From: Howard Chu <hyc@symas.com>

Prev by Date: Re: add sortvals after database loaded?
Next by Date: Re: LMBD questions (stat & copy)
Index(es):
- Chronological
- Thread