Re: (ITS#7958) LMDB: LIFO-reclaiming, write-performance improvement & bugfixes

[leo@yuriev.ru]

For informaion only - Nowadays 'lifo' and 'coalesce' features
implemented in ReOpenLDAP fork.
1) lmdb: MDB_LIFORECLAIM & MDB_COALESCE modes.
https://github.com/ReOpen/ReOpenLDAP/commit/829c2063b602238b5c93ea36a981de3=
d0d7994bc
2) lmdb-backend: support config for 'lifo' and 'coalesce' envflags.
https://github.com/ReOpen/ReOpenLDAP/commit/08b4a41b5b837548444ef0fef761494=
0c41d882a

With the couple of issues:
1)  lmdb in 'writemap' mode may inconsistent even with checkpoints
https://github.com/ReOpen/ReOpenLDAP/issues/1
2) lifo feature should be synchonized with checkpoints
https://github.com/ReOpen/ReOpenLDAP/issues/2

However currently it gives a reasonable boost (5-10 times) of
write-performance in our use case.

Leonid.

2014-10-20 0:27 GMT+03:00 =D0=9B=D0=B5=D0=BE=D0=BD=D0=B8=D0=B4 =D0=AE=D1=80=
=D1=8C=D0=B5=D0=B2 <leo@yuriev.ru>:
>>>> We are using the combination:
>>>>    envflags writemap nosync lifo
>>>>    checkpoint 0 1
>>>>
>>>> If the checkpoint is set in seconds, it gives us the assurance
>>>> consistent state database on disk.
>>>> However, without this patch meta-pages can be written by the kernel
>>>> before the data.
>>>>
>>>> In fact, for a full guarantee in case of death slapd process,
>>>> meta-page should be written explicitly.
>>
>>
>> No, the DB can never go inconsistent due to a process crashing - the pag=
es
>> in OS cache are always correct. It can only go inconsistent if the OS
>> crashes and a proper sync has not occurred.
>
> Yes, Howard, you are right.
> But apparently I need to be more precise.
> Talking about "death" of slapd, I meant all the reasons, including power =
off.
>
> For example, a power-off case:
> - The main power is turned off and the system switches to the UPS.
> - Given the notice, OS starts an emergency stop processes.
> - For some reason (does not have enough time to stop) slapd receives SIGK=
ILL.
> - OS tries to write mmap-region of the DB-file and begins with the
> lower address.
> - let the meta-pages has written completely, but for the rest of the
> data is not enough battery power.
> - Now DB is completely destroyed on the disk.
>
> To avoid this, the meta-pages should not be included in the rw-mapped
> region, and should be written explicity after a data pages.
>
>
>>>>>> commit 8ddd63161aeb2689822d1a8d27385d62e4e341ae
>>>>>> Author: Leo Yuriev <leo@yuriev.ru>
>>>>>> Date:   2014-09-19 22:47:19 +0400
>>>>>>
>>>>>>        BUGFIX - lmdb: properly sync meta-pages in mdb_sync_env().
>>>>>>
>>>>>>        Meta-pages may be updated during data-syncing in mdb_sync_env=
(),
>>>>>>        in this case database would be inconsistent.
>>>>>>
>>>>>>        Check-and-retry if lead txn-id changed during flushing data i=
n
>>>>>> mdb_sync_env().
>>
>> Fundamentally, you are trying to make an inherently unsafe configuration
>> "safer", but it's impossible. Assume you have mlock'd the meta pages int=
o
>> memory, so the OS never flushes them itself any more, and you're running
>> with NOSYNC. That means, within 3 transactions, the data pages on disk w=
ill
>> be out of sync with the meta pages on disk. If the OS crashes at that po=
int,
>> the entire DB will be lost.
>
> Not a problem.
> I had explained above - we should write meta-pages explicitly after
> the data sync.
> But also we should not perform reclaiming ahead of the last checkpoint.
>
>> The only way to make this mode of operation somewhat safe is to defer
>> reclaiming pages for even longer. E.g., instead of halting at current_tx=
nid
>> - 3, halt at current_txnid - 22, in which case the data pointed to by th=
e
>> on-disk meta pages cannot get obsolete until 20 transactions have occurr=
ed.
>>
>> Note that in combination with your LIFO patch, it's pretty much guarante=
ed
>> that the on-disk meta pages will be useless after only 2 un-sync'd
>> transactions.
>
> Yes, Howard, you are right.
> But I think there is confusion in the discussion because of mixing of
> LIFO-feature and changes for checkpoints consistency in a NOSYNC and
> WRITEMAP+NOSYNC modes.
> For a "NOSYNC + checkpoints" topic I will submit a separate ITS (like
> a 'volaile' related 7969,7970,7971).
> My opinion - it is a flaw, and no reason to don't fix it.
>
> Continuing the conversation about checkpoints in a LIFO context.
> I saw the problem, that you specified, and thinking over its solution,
> but have not yet found "golden ratio".
> And since we are having a serious problem with syncrepl, then I put
> off this task with an excuse "LIFO-patch not does worse than it was."
>
> In general, we should do not reclaim anything ahead of the txn, that
> is synced to the disk (let this be named a R-rule).
> To do so we need a second field like mti_txnid, but which will be
> update only at the end of mdb_env_write_meta().
> Finally we should start search in mdb_find_oldest() from value of this
> new field instead of the current txn number.
> This seems to will be work fine.
>
> However, I stopped on a reasoning - about the purpose of the
> checkpoints, about design LMDB as a product, about the expectations of
> the user and the necessary configuration parameters:
> - checkpoints are needed ONLY in nosync modes;
> - if the user does NOT activate the checkpoints, he do not care about
> consistency;
> - but if it is turned on, we MUST provide consistency on the checkpoints;
> - otherwise a checkpoints feature is thoughtless and should be REMOVED.
> Therefore implementation of checkpoints & reclaiming should be updated
> to conform to the "R-rule", that noted above.
>
> From this point of view a LIFO-feature also should be refined, but
> nevertheless can be very useful.
> - SYNC mode =3D takes a benefit from storage with write-back cache
> (assume powered by battery).
> - ASYNC/NOSYNC without checkpoint =3D significant reduction of
> committed/dirty pages and thereby much less write-iops.
> - ASYNC/NOSYNC with checkpoint =3D seems to same as a SYNC case.
>
> Total of all the above - I think first we need to fix a reclaiming or
> delete the checkpoints, and then I will complete LIFO.
>
>>   -- Howard Chu
>>   CTO, Symas Corp.           http://www.symas.com
>>   Director, Highland Sun     http://highlandsun.com/hyc/
>>   Chief Architect, OpenLDAP  http://www.openldap.org/project/
>
> Thank for conversation.
> Leonid.