RE: LMBD questions (stat & copy)

[Bruno Freudensprung <bruno.freudensprung@temis.com>]

Well... maybe am I a too "old school": I do appreciate lmdb a lot for its clean and simple API, but a parsimonious side of myself is kind of "shocked" demanding 1 TB upfront. After all, my application might not be the only one running on such a system, and there are many technologies using memory mapped files (like Lucene) that might need this addressing space as well.
So I have the impression that we might agree to disagree on this point.
In any case, thanks again for your support and answers! Very appreciated.

With best regards,

Bruno.

-----Message d'origine-----
De : Howard Chu [mailto:hyc@symas.com] 
Envoyé : jeudi 14 janvier 2016 10:25
À : Bruno Freudensprung <bruno.freudensprung@temis.com>; openldap-technical@openldap.org
Objet : Re: LMBD questions (stat & copy)

Bruno Freudensprung wrote:
>
> Hmmm... doing it this way has several drawbacks (if I am understanding the docs correctly):
> - the mutex of the write transactions prevents from writing 
> simultaneously in my dbis (I am inserting big graphs and sometimes it 
> takes from seconds to minutes to do so, plus those graphs are really 
> independent and modified by independent users)
> - the doc of mdb_env_set_maxdbs mentions a cost when there are more 
> than a "moderate number" of dbis
> - and, in the end, this adds another "maximum number of" to the 
> solution
>
> However don't get me wrong: lmdb has the best seek/get performances I have seen so far... this is why I really would like to use it (iterating over graph edges is so fast with lmdb and its zero copy API design). To me it is a the perfect fit. The last thing I really would like to do is being able to detect when I am getting close to the map size (thus my question on the mt_next_pgno field).

Even so, on a current x86-64 server you have 48 bits of virtual address space
- 256 terabytes worth. Even with hundreds of DBs open you can give them a terabyte each and not have to worry about it.

>
> Best regards,
>
> Bruno.
>
> -----Message d'origine-----
> De : Howard Chu [mailto:hyc@symas.com]
> Envoyé : mercredi 13 janvier 2016 23:35
> À : Bruno Freudensprung <bruno.freudensprung@temis.com>; openldap-technical@openldap.org
> Objet : Re: LMBD questions (stat & copy)
>
> Bruno Freudensprung wrote:
>> Hi Howard,
>>
>> Thank you very much for your patient answers, and my apologies for my late reply.
>> The "reserved space" was indeed occupied by the FREE_DBI, thanks again for pointing this out for me.
>> Since I will have hundreds of envs (of unpredictable size) in my
>> application,
>
> Sounds like a misuse. For hundreds of tables you should be using individual named DBs within a single env. The point is to use a single env and not have to worry about hundreds of configuration details.
>
>> I am a bit reluctant in allocating too much space for all of them, and that's why I was trying to find a way emit a warning when room is about to lack. In this regard I think I have read something very promising in the docs: the mt_next_pgno field of the txn. It looks perfect. Do you think it would be safe to expose it in a read-only way?
>> In any case I will follow you advice and try to find a safe compromise for the map size.
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/