[Date Prev][Date Next]
Re: LMDB ITS 8324 WriteMap performance on Windows
- To: Howard Chu <firstname.lastname@example.org>
- Subject: Re: LMDB ITS 8324 WriteMap performance on Windows
- From: Victor Baybekov <email@example.com>
- Date: Thu, 31 Dec 2015 00:23:48 +0000
- Cc: firstname.lastname@example.org
- Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=c3I7r5rARuom9Slki2q9hqaztfk1s+pRgKAz5DDOZkA=; b=DBxS9UVmaZV7BwOFNKKzJaLq8t8LpyNBa2MmcW5GBS16epVbc+K0hdqCX2pvw1ferU lcFDSZKG8MEymXBC8DxkLiUs38VFfHjTS1v1YwMEAgd64FfbO85LzgJGh5HpDqfgk4q3 MiD/ULf9xbX78AD9KQdeMkCQsXvEETQGcAsOyC7MNCRp4P1su/3LJnNlaQ7Vpktl1vJF HyXRNA4uObXB34mUkxpy0Hw7tA9o7e55Ho5ITM6ldFMsLb6m3f86mWbGWi2734eCBamC 3yCNTiQDGdCVu6pXlqWH9ZsJDo8SbpfTFb3yz8Uj/ITO4N0yQh2vNHVEJFa8F9h59Wrl S5Hg==
- In-reply-to: <email@example.com>
- References: <CAPBrOryip1Gu16RiCU_ypEhNGDh=nHPEd3JSA0kme90QMufSfirstname.lastname@example.org> <email@example.com>
Thanks for the explanation! The ultimate speed is needed for server usage where, e.g. now, I have several TB free space and could allocate the entire space at start. For clients, where I use LMDB for replicated cache of data subsets, even the 3x perf drop is kind of tolerable given the absolute numbers. Could we have a flag to use the patch optionally? So that on a server where I could allocate 1+Tb and use hacks with WriteMap as we discussed previously, I could enjoy the speedup of WriteMap and the fact that the pointer addresses from MDB_RESERVE remain fixed over the lifetime of a process?
Another question: with this patch, do addresses of pointers remain the same after the map file size increases, or we must follow the LMDB rules as if there is no WriteMap flag is used? Does Windows reserve the entire virtual memory space, and the patch just affects the file size on a disk, or file growth with the patch is equivalent to remapping and pointer addresses become invalid every time? I am referring to this discussion
we had before.
P.S. I know your hate for Windows and am starting to share it given rich tools available on *nix, but for small non-hardcore-programmers teams doing exploratory number crunching the benefits of RDP & GUIs etc. outweigh the benefits *nix gives for high load production systems, so we use Windows as an internal server. LMDB is just a perfect off-heap data structure for many use cases.