Re: LMDB and multiple processes

["Brian G. Merrell" <bgmerrell@gmail.com>]

On Wed, Jun 4, 2014 at 10:22 AM, Howard Chu <hyc@symas.com> wrote:

Brian G. Merrell wrote:

Hi all,

First, I'm having trouble finding resources to answer a question like this
myself, so please forgive me if I've missed something.

http://symas.com/mdb/doc/

Thanks. ÂI did see and skim the API portion of the docs before asking, but I was just having trouble knowing how the pieces fit together to solve a problem.

Â

I'm considering using LMDB (versus LevelDB) for a project I'm working on where
I'll be receiving a high volume (hundreds per second) of high priority
requests (over HTTP) and issuing multiple (<10) database queries per request.

I'll also have a separate process receiving updates for the data and writing
to the database. ÂThis will happen often (several times a minute, perhaps),
but the priority is much lower than the read requests.

LMDB appealed to me because of the read performance and that I could have one
processing reading data from LMDB and another process writing data updates to
LMDB.

For proof of concept, I hacked up the following (I'll use pseudocode since I
used the Go bindings for my actual programs, and hopefully my question is
sufficiently abstract not to matter):

Process 1, the writer, simply writes a random integer (from 0 to 1000) to a
defined set of keys:

env = NewEnv()
env.Open("/tmp/foo", 0, 0664)
txn = env.BeginTxn(nil, 0)
dbi = txn.DBIOpen(nil, 0)
txn.Commit()

txn = env.BeginTxn(nil, 0)
n_entries = 5
for i = 0; i < n_entries; i++ {
  Âkey = sprintf("Key-%d", i)
  Âval = sprintf("Val-%d", rand.Int(1000))
  Âtxn.Put(dbi, key, val, 0)
}
txn.Commit()
env.DBIClose(dbi)
env.Close()

Process 2, the reader, simply loops forever and does random access reads on
the data from process 1 (I won't benefit from a cursor for my actual problem),
and prints out that data occasionally:

env = NewEnv()
env.Open("/tmp/foo", 0, 0664)
while {
  Âtxn = BeginTxn(nil, 0)
  Âdbi = txn.DBIOpen(nil, 0)
  Âtxn.Commit()
  Âfor i = 0; i < n_entries; i++ {
    Âkey = sprintf("Key-%d", i)
    Âval = txn.Get(dbi, key)
    Âprint("%s: %s", key, value)
  Â}
  Âenv.DBIClose(dbi)
  Âsleep(5)
}

So my high level question is: What am I doing wrong? ÂThis seems to work OK,
but a lot of it was guesswork, so I'm sure I'm doing some silly things.

Your reader process should be using read transactions.Â

OK, I interpret this as meaning that I need to pass the MDB_RDONLY flag to mdb_txn_begin. ÂIs that correct?

Â

For example, first I put the BeginTxn() and DBIOpen() calls in process 2
outside of the while loop, but when I did that, I never saw the updates values
upon running process 1 simultaneously. ÂIn my real-world application, it seems
like adding these calls to every request (to be sure the data being read is
up-to-date) could be an unnecessary performance penalty.

In the actual LMDB API read transactions can be reused by their creating thread, so they are zero-cost after the first time. I don't know if any of the other language wrappers leverage this fact.

This helps a lot. ÂI will investigate what the case is with gomdb.

Â

Opening a DBI only needs to be done once per process. Opening per transaction would be stupid, like reopening a file handle on every request.

I suspected so. ÂThe fact that mdb_dbi_open takes a transaction had me confused a bit, because I thought I would need to pass in the new transaction every time I got a transaction from mdb_txn_begin.

I've refactored the reader to look like this:

env = NewEnv()

env.Open("/tmp/foo", 0, 0664)

txn = BeginTxn(nil, mdb.RDONLY) // parent txn is the nil arg

dbi = txn.DBIOpen(nil, 0)

txn.Abort()

while {

  Âtxn = BeginTxn(nil, mdb.RDONLY) // parent txn is the nil arg

  Âfor i = 0; i < n_entries; i++ {

    Âkey = sprintf("Key-%d", i)

    Âval = txn.Get(dbi, key)

    Âprint("%s: %s", key, value)

  Â} ÂÂ

  Âtxn.Commit()

  Âsleep(5)

}

env.DBIClose(dbi)

Now, I guess the big question that BeginTxn inside the loop is zero-cost.

Thanks for the tips so far Howard; it has been very helpful.

Â

I was suspect there are flags that I can/should be using, but I'm not sure.

Thanks for any input.

    ÂBrian

--
 -- Howard Chu
 CTO, Symas Corp.      http://www.symas.com
 Director, Highland Sun   http://highlandsun.com/hyc/
 Chief Architect, OpenLDAP Âhttp://www.openldap.org/project/

    BrianÂ