[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: (ITS#8972) Performance/freezing issues using NTDLL on Windows Server on Azure



kriszyp@gmail.com wrote:
> Full_Name: Kristopher William Zyp
> Version: LMDB 0.9.70
> OS: Windows Server 2012 R2
> URL: 
> Submission from: (NULL) (71.199.6.148)

I guess Windows users are never happy.

Note that you're using the LMDB work-in-progress code. You ought to be using the
LMDB 0.9 release branch, which just uses Win32.

The fact is that you get optimal performance by using LMDB as originally designed
and documented - choose as large a mapsize as possible and then forget about it.
Preallocation is always more efficient than incremental growth.

Clearly Windows will struggle to allocate disk space to grow files and address
space to grow maps while there's competing system activity occurring, and this
problem is worse when running under a hypervisor. This is the reason we never
merged the NTDLL code into the release branch; we knew from the outset that it
would destroy performance.

https://openldap.org/lists/openldap-technical/201511/msg00107.html

As a general rule, we're opposed to adding platform-specific API options to LMDB.
The API is supposed to be uniform across all supported platforms. If you want to
submit a patch for compile-time selection of this feature, I guess that could
work. As it is, LMDB is clearly not going to be a preinstalled Windows DLL so
it should be being built privately for every app that uses it anyway, so this
shouldn't cause compatibility issues.

> We have experienced significant performance issues in the form of strange pauses
> or freezing of any processes that are using our LMDB databases. These pauses are
> periodic, often every few minutes, and typically completely freeze a process for
> about 5 seconds, sometimes less, sometimes as much as a minute. They often occur
> when the process is completely idle (we have a sleep timer to monitor for them),
> but we also see a lot of unusually large pauses on commits too. We have never
> been able to reproduce any of these issues with our local machines running
> Windows 10, only on the virtual Azure Windows servers.
> 
> After weeks of investigation, we finally realized that all of these strange
> performance issues seemed to be due to the use of NTDLL for memory mapping. I
> swapped out the NtCreateSection/NtMapViewOfSection calls for standard Win32
> CreateFileMapping/MapViewOfFile calls to create the memory map, and suddenly
> these issues went away, and performance dramatically improved. Making this
> switch does mean you lose the progressive file allocation functionality of
> NtCreateSection, but this is a small price to pay for the vast performance
> benefit of using standard file mapping.
> 
> Anyway, my request is to simply to provide an option to use CreateFileMapping
> API instead of NtCreateSection. I recognize that the NTDLL based APIs may be
> more convenient for many users, but an option to use the standard APIs, even if
> it means the entire map is pre-allocated in the file size, is well worth it for
> the gains. I also recognize that Windows isn't the first choice of LMDB, but
> performance with these standard APIs is very good.
> 
> I'd be happy to work on putting together a branch/pull request to address this.
> However, your repository is a little different than typical github repositories,
> so I wanted to know if that was doable, and if there was a proper way to submit
> a PR, or if it needs to be submitted through this ticket system as a patch.

http://www.openldap.org/devel/contributing.html

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/