[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
RE: (ITS#8972) Performance/freezing issues using NTDLL on WindowsServer on Azure
- To: openldap-its@OpenLDAP.org
- Subject: RE: (ITS#8972) Performance/freezing issues using NTDLL on WindowsServer on Azure
- From: kriszyp@gmail.com
- Date: Tue, 05 Feb 2019 19:20:55 +0000
- Auto-submitted: auto-generated (OpenLDAP-ITS)
--_22B17C00-48BC-46BE-A758-3013CC9A68A3_
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="utf-8"
Oh wow, this is just a matter of me using the wrong LMDB version? I had no =
idea I wasn=E2=80=99t using the release version. I=E2=80=99ve been using ht=
tps://github.com/Venemo/node-lmdb, which includes LMDB directly in it, and =
had assumed that the version number in the code implied a release. I can pu=
t together a PR for that project to get it on the right version.
Thank you so much the pointer, I appreciate! And thank you so much for the =
great software, its a fantastic package!
Thanks,
Kris
From: Howard Chu
Sent: February 5, 2019 12:03 PM
To: kriszyp@gmail.com; openldap-its@OpenLDAP.org
Subject: Re: (ITS#8972) Performance/freezing issues using NTDLL on WindowsS=
erver on Azure
kriszyp@gmail.com wrote:
> Full_Name: Kristopher William Zyp
> Version: LMDB 0.9.70
> OS: Windows Server 2012 R2
> URL:=20
> Submission from: (NULL) (71.199.6.148)
I guess Windows users are never happy.
Note that you're using the LMDB work-in-progress code. You ought to be usin=
g the
LMDB 0.9 release branch, which just uses Win32.
The fact is that you get optimal performance by using LMDB as originally de=
signed
and documented - choose as large a mapsize as possible and then forget abou=
t it.
Preallocation is always more efficient than incremental growth.
Clearly Windows will struggle to allocate disk space to grow files and addr=
ess
space to grow maps while there's competing system activity occurring, and t=
his
problem is worse when running under a hypervisor. This is the reason we nev=
er
merged the NTDLL code into the release branch; we knew from the outset that=
it
would destroy performance.
https://openldap.org/lists/openldap-technical/201511/msg00107.html
As a general rule, we're opposed to adding platform-specific API options to=
LMDB.
The API is supposed to be uniform across all supported platforms. If you wa=
nt to
submit a patch for compile-time selection of this feature, I guess that cou=
ld
work. As it is, LMDB is clearly not going to be a preinstalled Windows DLL =
so
it should be being built privately for every app that uses it anyway, so th=
is
shouldn't cause compatibility issues.
> We have experienced significant performance issues in the form of strange=
pauses
> or freezing of any processes that are using our LMDB databases. These pau=
ses are
> periodic, often every few minutes, and typically completely freeze a proc=
ess for
> about 5 seconds, sometimes less, sometimes as much as a minute. They ofte=
n occur
> when the process is completely idle (we have a sleep timer to monitor for=
them),
> but we also see a lot of unusually large pauses on commits too. We have n=
ever
> been able to reproduce any of these issues with our local machines runnin=
g
> Windows 10, only on the virtual Azure Windows servers.
>=20
> After weeks of investigation, we finally realized that all of these stran=
ge
> performance issues seemed to be due to the use of NTDLL for memory mappin=
g. I
> swapped out the NtCreateSection/NtMapViewOfSection calls for standard Win=
32
> CreateFileMapping/MapViewOfFile calls to create the memory map, and sudde=
nly
> these issues went away, and performance dramatically improved. Making thi=
s
> switch does mean you lose the progressive file allocation functionality o=
f
> NtCreateSection, but this is a small price to pay for the vast performanc=
e
> benefit of using standard file mapping.
>=20
> Anyway, my request is to simply to provide an option to use CreateFileMap=
ping
> API instead of NtCreateSection. I recognize that the NTDLL based APIs may=
be
> more convenient for many users, but an option to use the standard APIs, e=
ven if
> it means the entire map is pre-allocated in the file size, is well worth =
it for
> the gains. I also recognize that Windows isn't the first choice of LMDB, =
but
> performance with these standard APIs is very good.
>=20
> I'd be happy to work on putting together a branch/pull request to address=
this.
> However, your repository is a little different than typical github reposi=
tories,
> so I wanted to know if that was doable, and if there was a proper way to =
submit
> a PR, or if it needs to be submitted through this ticket system as a patc=
h.
http://www.openldap.org/devel/contributing.html
--=20
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
--_22B17C00-48BC-46BE-A758-3013CC9A68A3_
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset="utf-8"
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of=
fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40";><head><meta ht=
tp-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta name=
=3DGenerator content=3D"Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=3DEN-CA link=3Dblue vlink=3D"#954F72"><div cla=
ss=3DWordSection1><p class=3DMsoNormal>Oh wow, this is just a matter of me =
using the wrong LMDB version? I had no idea I wasn=E2=80=99t using the rele=
ase version. I=E2=80=99ve been using <a href=3D"https://github.com/Venemo/n=
ode-lmdb">https://github.com/Venemo/node-lmdb</a>, which includes LMDB dire=
ctly in it, and had assumed that the version number in the code implied a r=
elease. I can put together a PR for that project to get it on the right ver=
sion.</p><p class=3DMsoNormal>Thank you so much the pointer, I appreciate! =
And thank you so much for the great software, its a fantastic package!</p><=
p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Thanks,<br>Kr=
is</p><p class=3DMsoNormal><o:p> </o:p></p><div style=3D'mso-element:p=
ara-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm=
0cm 0cm'><p class=3DMsoNormal style=3D'border:none;padding:0cm'><b>From: <=
/b><a href=3D"mailto:hyc@symas.com";>Howard Chu</a><br><b>Sent: </b>February=
5, 2019 12:03 PM<br><b>To: </b><a href=3D"mailto:kriszyp@gmail.com";>kriszy=
p@gmail.com</a>; <a href=3D"mailto:openldap-its@OpenLDAP.org";>openldap-its@=
OpenLDAP.org</a><br><b>Subject: </b>Re: (ITS#8972) Performance/freezing iss=
ues using NTDLL on WindowsServer on Azure</p></div><p class=3DMsoNormal><o:=
p> </o:p></p><p class=3DMsoNormal>kriszyp@gmail.com wrote:</p><p class=
=3DMsoNormal>> Full_Name: Kristopher William Zyp</p><p class=3DMsoNormal=
>> Version: LMDB 0.9.70</p><p class=3DMsoNormal>> OS: Windows Server =
2012 R2</p><p class=3DMsoNormal>> URL: </p><p class=3DMsoNormal>> Sub=
mission from: (NULL) (71.199.6.148)</p><p class=3DMsoNormal><o:p> </o:=
p></p><p class=3DMsoNormal>I guess Windows users are never happy.</p><p cla=
ss=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Note that you're u=
sing the LMDB work-in-progress code. You ought to be using the</p><p class=
=3DMsoNormal>LMDB 0.9 release branch, which just uses Win32.</p><p class=3D=
MsoNormal><o:p> </o:p></p><p class=3DMsoNormal>The fact is that you ge=
t optimal performance by using LMDB as originally designed</p><p class=3DMs=
oNormal>and documented - choose as large a mapsize as possible and then for=
get about it.</p><p class=3DMsoNormal>Preallocation is always more efficien=
t than incremental growth.</p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal>Clearly Windows will struggle to allocate disk space to g=
row files and address</p><p class=3DMsoNormal>space to grow maps while ther=
e's competing system activity occurring, and this</p><p class=3DMsoNormal>p=
roblem is worse when running under a hypervisor. This is the reason we neve=
r</p><p class=3DMsoNormal>merged the NTDLL code into the release branch; we=
knew from the outset that it</p><p class=3DMsoNormal>would destroy perform=
ance.</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>htt=
ps://openldap.org/lists/openldap-technical/201511/msg00107.html</p><p class=
=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>As a general rule, w=
e're opposed to adding platform-specific API options to LMDB.</p><p class=
=3DMsoNormal>The API is supposed to be uniform across all supported platfor=
ms. If you want to</p><p class=3DMsoNormal>submit a patch for compile-time =
selection of this feature, I guess that could</p><p class=3DMsoNormal>work.=
As it is, LMDB is clearly not going to be a preinstalled Windows DLL so</p=
><p class=3DMsoNormal>it should be being built privately for every app that=
uses it anyway, so this</p><p class=3DMsoNormal>shouldn't cause compatibil=
ity issues.</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorm=
al>> We have experienced significant performance issues in the form of s=
trange pauses</p><p class=3DMsoNormal>> or freezing of any processes tha=
t are using our LMDB databases. These pauses are</p><p class=3DMsoNormal>&g=
t; periodic, often every few minutes, and typically completely freeze a pro=
cess for</p><p class=3DMsoNormal>> about 5 seconds, sometimes less, some=
times as much as a minute. They often occur</p><p class=3DMsoNormal>> wh=
en the process is completely idle (we have a sleep timer to monitor for the=
m),</p><p class=3DMsoNormal>> but we also see a lot of unusually large p=
auses on commits too. We have never</p><p class=3DMsoNormal>> been able =
to reproduce any of these issues with our local machines running</p><p clas=
s=3DMsoNormal>> Windows 10, only on the virtual Azure Windows servers.</=
p><p class=3DMsoNormal>> </p><p class=3DMsoNormal>> After weeks of in=
vestigation, we finally realized that all of these strange</p><p class=3DMs=
oNormal>> performance issues seemed to be due to the use of NTDLL for me=
mory mapping. I</p><p class=3DMsoNormal>> swapped out the NtCreateSectio=
n/NtMapViewOfSection calls for standard Win32</p><p class=3DMsoNormal>> =
CreateFileMapping/MapViewOfFile calls to create the memory map, and suddenl=
y</p><p class=3DMsoNormal>> these issues went away, and performance dram=
atically improved. Making this</p><p class=3DMsoNormal>> switch does mea=
n you lose the progressive file allocation functionality of</p><p class=3DM=
soNormal>> NtCreateSection, but this is a small price to pay for the vas=
t performance</p><p class=3DMsoNormal>> benefit of using standard file m=
apping.</p><p class=3DMsoNormal>> </p><p class=3DMsoNormal>> Anyway, =
my request is to simply to provide an option to use CreateFileMapping</p><p=
class=3DMsoNormal>> API instead of NtCreateSection. I recognize that th=
e NTDLL based APIs may be</p><p class=3DMsoNormal>> more convenient for =
many users, but an option to use the standard APIs, even if</p><p class=3DM=
soNormal>> it means the entire map is pre-allocated in the file size, is=
well worth it for</p><p class=3DMsoNormal>> the gains. I also recognize=
that Windows isn't the first choice of LMDB, but</p><p class=3DMsoNormal>&=
gt; performance with these standard APIs is very good.</p><p class=3DMsoNor=
mal>> </p><p class=3DMsoNormal>> I'd be happy to work on putting toge=
ther a branch/pull request to address this.</p><p class=3DMsoNormal>> Ho=
wever, your repository is a little different than typical github repositori=
es,</p><p class=3DMsoNormal>> so I wanted to know if that was doable, an=
d if there was a proper way to submit</p><p class=3DMsoNormal>> a PR, or=
if it needs to be submitted through this ticket system as a patch.</p><p c=
lass=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>http://www.openl=
dap.org/devel/contributing.html</p><p class=3DMsoNormal><o:p> </o:p></=
p><p class=3DMsoNormal>-- </p><p class=3DMsoNormal>=C2=A0=C2=A0-- Howard Ch=
u</p><p class=3DMsoNormal>=C2=A0 CTO, Symas Corp.=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http://www.symas.com</p><p class=3DMso=
Normal>=C2=A0 Director, Highland Sun=C2=A0=C2=A0=C2=A0=C2=A0 http://highlan=
dsun.com/hyc/</p><p class=3DMsoNormal> =C2=A0Chief Architect, OpenLDAP=C2=
=A0 http://www.openldap.org/project/</p><p class=3DMsoNormal><o:p> </o=
:p></p></div></body></html>=
--_22B17C00-48BC-46BE-A758-3013CC9A68A3_--