[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: zero copy jni lib for lmdb

Kristoffer SjÃgren wrote:

Ah, I overlooked that flag. This means I can return the pointer to Java and
write directly into LMDB for the duration of the current transaction. Sadly,
after trying it, there is no noticeable effect on latency. Commits
are still occupying majority of total time.

Could be some kind of overhead calling JNI repeatably in a tight loop? So
instead I tried to batch all 1 million addresses over to JNI in one go and
manage puts+begin+commit from the C side. Again, this did not have much
notable effect.

What's even more weird is that when I clock the batch time from C I get around
1.8 - 1.7 sec, but the total execution time is actually more like 6 sec!
Clearly there is something I don't understand here.
Dunno what's going on there. Have you tried first with a smaller batch, 500K, 100K? 

For reads I already let pass the value pointer to Java and read directly from
it (like you suggest) and this part is actually ~1.5 times faster than the
buffer copy approach in lmdbjni. Just to be sure, are the value pointers
managed by LMDB or do I need to free them manually per transaction? Right now
I just read data from them.
All pointers returned from LMDB are owned/managed by LMDB. That's already stated in the docs. 


BTW, forgot to say thanks for a great product Howard!


On Sun, Jun 29, 2014 at 4:56 PM, Howard Chu <hyc@symas.com
<mailto:hyc@symas.com>> wrote:

    Kristoffer SjÃgren wrote:

        Hi

        I'm experimenting a bit with developing a JNI library for LMDB that
        utilize
        zero copy buffers. As you may know there is a project called lmdbjni [1]
        already but it does buffer copy only.

        I must admit that my C and JNI skills is not really that great, so please
        forgive any stupidity on my part.


    Can't offer much advice on java or JNI. But your test doesn't really
    leverage LMDB's zero-copy writes. To do that you have to use the
    MDB_RESERVE write flag before generating the output value. Otherwise
    there's still a copy from the mdb_put argument into the actual DB.

    I don't know what other buffer copies occur before you finally reach
    mdb_put, but I don't see why you need to do anything special to pass a
    user value into mdb_put. zero-copy really only has significant benefit for
    readers, and that's where you have to play games with non-GC'd pointers.

    Perhaps other java users on this list can offer more advice.


        Anyway, i'm taking the sun.misc.Unsafe (no bounds checking) approach for
        memory allocation and pass memory addresses through JNI to LMDB for both
        writes and reads and at the moment I have implemented put, get, begin,
        commit [2].


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/