Re: (ITS#8392) Opening multiple databases from the same process

[hyc@symas.com]

matthew@wellquite.org wrote:
> Full_Name: Matthew Sackman
> Version: LMDB 0.9.18
> OS: Linux
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (109.149.185.170)
>
>
> Linux koba 4.4.0-1-amd64 #1 SMP Debian 4.4.6-1 (2016-03-17) x86_64 GNU/Linux
> LMDB 0.9.18 (compiled locally)
>
> I'm currently writing a program that iterates through multiple LMDB databases in
> order to check differences between them. On each database I have one
> long-running read-only txn with a cursor, and I'm walking through all the
> databases at the same time.
>
> I'm seeing some very odd behaviour. For example, cursor.Get with FIRST is not
> always returning the first item in the table (which I happen to know exactly
> what it should be). The behaviour is non-deterministic - different runs produce
> different results.
>
> I've searched through the docs and I can't find any statements regarding opening
> multiple dbs from the same process. Is this supported?
>
> I'm writing this in Go, using my own bindings. I'm pretty certain that I'm doing
> all the threading correctly and I certainly get this same issue regardless of
> whether I'm using NOTLS or not (with my bindings, it shouldn't be necessary
> because I'm locking go-routines to os-threads), plus these are the same bindings
> that I've used throughout GoshawkDB and I've not come across this problem until
> now.
>
> I can guarantee that each DB that I open will be opened in a different OS thread
> and that no single OS thread will ever access different DBs.

Is this entire task read-only, nobody is modifying any of the DBs while these 
readers are iterating thru the DBs?

I've only used Go very briefly. I would be quite suspicious of that goroutine 
behavior but don't have any insight there.

You might consider compiling LMDB with MDB_DEBUG defined, enabling debug 
output, and then watching the cursor activity. Of course, the act of writing 
debug messages will impact thread scheduling, so it may not reveal what's 
going on.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/