[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#8095) Patch
--047d7bfcf1deec63470514165cd2
Content-Type: text/plain; charset=UTF-8
My Testing does show it is not worth it for the data size in the db.
Also there is a race condition using sendfile on linux that could happen.
It is avoidable but it would require changes to the protocol to avoid it
which I am not willing to do.
So dismiss my patch with extreme prejudice
On Wed, Apr 15, 2015 at 10:37 PM, Howard Chu <hyc@symas.com> wrote:
> rouzier@gmail.com wrote:
>
>> Full_Name: James Rouzier
>> Version:
>> OS:
>> URL: ftp://ftp.openldap.org/incoming/james-rouzier-150331.patch
>> Submission from: (NULL) (192.222.129.11)
>>
>>
>> I am working on a proof of concept of a simple key value store service
>> using
>> lmdb as the back end.
>> I am using sendfile to avoid an extra copy.
>> I accomplished this by creating function to extract get the mmap ptr from
>> the
>> env so I can calculate the offset of data.
>>
>
> I'm quite reluctant to return the mmap pointer. In a WRITEMAP environment
> that invites all manner of undetectable/unpreventable corruptions.
>
> Have you actually measured the performance difference between using
> sendfile() here and plain write() or send()? The kernel doesn't physically
> copy the written data from user space to kernel space, it just maps in a
> particular page. Since LMDB data is already mmap'd, the kernel has very
> little work to do since the relevant pages are already present in the
> kernel's page tables.
>
> Ultimately sending data over TCP involves byte-by-byte iteration anyway,
> since TCP checksums must be computed on all outgoing data. Doesn't seem to
> me like sendfile can be a very significant optimization here.
>
>>
>> I figured that function might be useful to the bigger community.
>>
>> If it does not fit in the bigger vision of lmdb it is fine not to include
>> it.
>>
>>
>> I, James Rouzier, hereby place the following modifications to OpenLDAP
>> Software
>> (and only these modifications) into the public domain. Hence, these
>> modifications may be freely used and/or redistributed for any purpose
>> with or
>> without attribution and/or other notice.
>>
>>
>>
>>
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
--047d7bfcf1deec63470514165cd2
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr">My Testing doe=
s show it is not worth it for the data size in the db.<br><br><div>Also the=
re is a race condition using sendfile on linux that could happen.<br>It is =
avoidable but it would require changes to=C2=A0the protocol to avoid it whi=
ch I am not willing to do.<br><br>So dismiss my patch with extreme prejudic=
e<br><br><br></div></div><div><div><div class=3D"gmail_extra"><br><div clas=
s=3D"gmail_quote">On Wed, Apr 15, 2015 at 10:37 PM, Howard Chu <span dir=3D=
"ltr"><<a href=3D"mailto:hyc@symas.com"; target=3D"_blank">hyc@symas.com<=
/a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:=
0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><a href=3D"mailto:r=
ouzier@gmail.com" target=3D"_blank">rouzier@gmail.com</a> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Full_Name: James Rouzier<br>
Version:<br>
OS:<br>
URL: <a href=3D"ftp://ftp.openldap.org/incoming/james-rouzier-150331.patch"=
target=3D"_blank">ftp://ftp.openldap.org/incoming/james-rouzier-150331.pat=
ch</a><br>
Submission from: (NULL) (192.222.129.11)<br>
<br>
<br>
I am working on a proof of concept of a simple key value store service usin=
g<br>
lmdb as the back end.<br>
I am using sendfile to avoid an extra copy.<br>
I accomplished this by creating function to extract get the mmap ptr from t=
he<br>
env so I can calculate the offset of data.<br>
</blockquote>
<br>
I'm quite reluctant to return the mmap pointer. In a WRITEMAP environme=
nt that invites all manner of undetectable/unpreventable corruptions.<br>
<br>
Have you actually measured the performance difference between using sendfil=
e() here and plain write() or send()? The kernel doesn't physically cop=
y the written data from user space to kernel space, it just maps in a parti=
cular page. Since LMDB data is already mmap'd, the kernel has very litt=
le work to do since the relevant pages are already present in the kernel=
9;s page tables.<br>
<br>
Ultimately sending data over TCP involves byte-by-byte iteration anyway, si=
nce TCP checksums must be computed on all outgoing data. Doesn't seem t=
o me like sendfile can be a very significant optimization here.<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<br>
I figured that function might be useful to the bigger community.<br>
<br>
If it does not fit in the bigger vision of lmdb it is fine not to include i=
t.<br>
<br>
<br>
I, James Rouzier, hereby place the following modifications to OpenLDAP Soft=
ware<br>
(and only these modifications) into the public domain. Hence, these<br>
modifications may be freely used and/or redistributed for any purpose with =
or<br>
without attribution and/or other notice.<br>
<br>
<br>
<br><span><font color=3D"#888888">
</font></span></blockquote><span><font color=3D"#888888">
<br>
<br>
-- <br>
=C2=A0 -- Howard Chu<br>
=C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"=
http://www.symas.com"; target=3D"_blank">http://www.symas.com</a><br>
=C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland=
sun.com/hyc/" target=3D"_blank">http://highlandsun.com/hyc/</a><br>
=C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p=
roject/" target=3D"_blank">http://www.openldap.org/project/</a><br>
</font></span></blockquote></div><br></div>
</div></div></div><br></div>
--047d7bfcf1deec63470514165cd2--