[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: [LMDB] Large transactions
- To: Howard Chu <hyc@symas.com>, openldap-technical@openldap.org
- Subject: Re: [LMDB] Large transactions
- From: Jürgen Baier <baier@semedy.com>
- Date: Wed, 29 Nov 2017 14:43:17 +0100
- Content-language: en-US
- Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semedy-com.20150623.gappssmtp.com; s=20150623; h=subject:references:from:to:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=Ye/FfAbnPOOO5NgI89iPOr+/LgTu8jcYCbqAlH1x21U=; b=DKc3IOQ9NEmMKaif2El169Acd1OpxDLvH8b/s8nAcbx1yMC/qcoxFIuelvWUdZtxWd IQ+Yf0vTuXYSP4QVrVzxwQICQMpuesUS4udmWFCwArYsT4IPY3xx5qMu9rprtHsqbAWI zgEdt862WkkVSBh5NKnHeF/otm5FKcAzau6vP1tcimpeV+b4dZ65UIh0iqbxS1tHyqEu BYj8rqQDq9sujy7Y5lyo8Ozhdved19OwE0NG74U1DzkfyyURHXnGzjlBHonXGOAsu4JU mWzqtMMgSeRNzXhxjqoSBC6bVV2+0XzH4zgEK44XgxpmAtQYlQ3a9T5l36e1AhFH/8qw v5rA==
- In-reply-to: <ebae0de1-bfc6-8d38-0db5-e65d39d7fa72@symas.com>
- References: <555c7e78-a5e4-23ee-5b5c-2720b54e69e4@semedy.com> <WM!354c5fbb04a8692d321d46697249756ed1ce65dc4cdbf592366138261b88ed7902879da025f923c184d6c4eb02473dc9!@mailstronghold-2.zmailcloud.com> <ebae0de1-bfc6-8d38-0db5-e65d39d7fa72@symas.com>
- User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.4.0
Hi,
thanks for the answer. However, I still have a follow-up question on
this benchmark.
When I add 1 billion key/value pairs (16 byte MD5) to the LMDB database
(in a single transaction (but I also get similar results when I add the
same data in multiple transactions)) I get the following results:
Windows, without MDB_WRITEMAP: 46h
Windows, with MDB_WRITEMAP: 6h (!)
Linux (ext4), without MDB_WRITEMAP: 75h
Linux (ext4), with MDB_WRITEMAP: 73h
MDB_WRITEMAP seems to have a huge impact on write performance on
Windows, but on Linux I do not see similar improvements.
So I have two questions:
1) Could the the difference between Linux and Windows performance
regarding the MDB_WRITEMAP option be related to the fact that LMDB
currently uses sparse files on Linux, but not on Windows?
2) Is there a way to speed up Linux? Is there a way to pre-allocate the
data.mdb on startup?
Thanks,
Jürgen
On 21.11.17 21:17, Howard Chu wrote:
Jürgen Baier wrote:
Hi,
I have a question about LMDB (I hope this is the right mailing list
for such a question).
I'm running a benchmark (which is similar to my intended use case)
which does not behave as I hoped. I store 1 billion key/value pairs
in a single LMDB database. _In a single transaction._ The keys are
MD5 hash codes from random data (16 bytes) and the value is the
string "test".
The documentation about mdb_page_spill says (as far as I understand)
that this function is called to prevent MDB_TXN_FULL situations. Does
this mean that my transaction is simply too large to be handled
efficiently by LMDB?
Yes.
--
Juergen Baier