[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Issues arising from creating powerdns backend based on LMDB

Howard Chu wrote:
Mark Zealey wrote:
I'm not doing *any* commits just one big txn for all the data...

The below C works fine up until i=4m (ie 500mb of residential memory
shown in top), then has massive slowdown, shared memory (again, as seen
in top) increases, waits about 20-30 seconds and then disks get hammered
writing 10mb/sec (200txns) when they are capable of 100-200mb/sec
streaming writes... Does it do the same for you?

Kyoto writes async by default. You should do the same here, use
MDB_NOSYNC on the env_open.

MDB_NOSYNC makes no difference in my test case above - seeing exactly
the same memory, speed and disk patterns. Are you able to reproduce it?

Yes, I see it here, and I see the problem. LMDB was not originally designed to
handle transactions of unlimited size. It originally had a txn sizelimit of
about 512MB. In 0.9.7 we added some code to raise this limit, and it's
performing quite poorly here. I've tweaked my copy of the code to alleviate
that problem but your test program still fails here because the volume of data
being written also exceeds the map size. You were able to run this to completion?

Two things... I've committed a patch to mdb.master to help this case out. It sped up my run of your program, using only 10M records, from 19min to 7min.

Additionally, if you change your test program to commit every 2M records, and avoid running into the large txn situation, then the 10M records are stored in only 1m51s.

Running it now with the original 100M count. Will see how it goes.

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/