[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Can't open env with required mapsize on windows .

Mohammed Muneer wrote:


     "Windows is like that."

     I know.


Please keep your replies on the mailing list.


     "Have no idea what you're actually doing, so it's impossible to answer.
Learn to ask better questions and provide better information."

     Sorry that I didn't sound intelligent. I was in a hurry when I wrote it.
I will now try to give you a detailed account of what I am doing.


     "What revision of LMDB?"

     The latest hosted at https://github.com/LMDB/lmdb.
"The latest" in which branch? Note that mdb.master is the main development branch and contains work-in-progress code that is probably not stable. As a non-LMDB-developer you should only be using actual tagged releases. You may occasionally want to use the mdb.RE/xx release engineering branches if they contain a fix for a specific issue that you encountered. 


     "Which version of Windows?"

     All are Windows 7 Ultimate x64


     "How much free disk space is there?"

     Machine 1 has around 26 GB free. Machine 2 has around 40 GB free.
Virtualbox guest has only has around 10 GB free.


     "What else is your application code doing?"

     This is silly. My application was trying to open an env on windows. Here
is the code for it.


       #define MB * 1024 * 1024
       #define GB * 1024 * 1024 * 1024

       MDB_env *env = 0;

       bool openEnv() {
           mdb_size_t mapSize = 390 MB;
           MDB_dbi maxNamedDatabases = 100;


           if (mdb_env_create(&env) != MDB_SUCCESS)
           return false;

           if (mdb_env_set_mapsize(env, mapSize) != MDB_SUCCESS)
           return false;

           if (mdb_env_set_maxdbs(env, maxNamedDatabases) != MDB_SUCCESS)
           return false;

           int err = mdb_env_open(env, "test", 0, 0664);

           if (err != MDB_SUCCESS) {
           fprintf(stderr, "Error opening env %i", err);
           return false;
           }
           fprintf(stderr, "Success opening env %i", err);

           return true;
       }

     int main() {

         if (!openEnv())
           return 1;

     }




     "How do you know that only XX MB were "opened"

     Trial and error on windows showed that mdb_env_open returns success when
I give a mapSize of 390 MB on Machine 1, etc...
     But on further testing, we can open 2 GB sometimes. But after a system
restart, situation is still the same.


     "what stats are you checking? "

     Well as you can see from the above code, I am checking the return value
of mdb_env_open().



     "What other differences exist between machine 1 and machine 2?"

     Not much. One being a core-i3 and the other core-i5. Both have 4 GB RAM.
Both are similiar in that none of them have
     so many applications installed and all have used memory around 500 MB



     "What other processes are running,"

      As I stated its mostly a default installation of Windows x64 on all
machines.



     "how much memory are they consuming on each machine?"

      Are you referring to other processes or my application. Assuming you are
talking about my application, it is just a
      main function. So that might be irrelevant



     "Note that just because you create a 1GB memory map of a file doesn't
mean that it will actually consume 1GB of RAM.
      That's not how memory mapping works."

      Never said that. Didn't even imply it. Don't know where you are getting
this. I know how memory map works. Having implemented
      one in a custom hobby OS 5 years ago. Your implementation leveraging COW
along with performance and MVCC was the whole reason I was
      interested in your project.



      "Read your Windows developer documentation, MSDN is always there for you
to read. We're not here to teach you how your OS works."

      You don't have to teach me how my OS works. Sorry if I gave you that
impression.


Apparently we do. See below.


      "Hangs where, exactly? Have you gotten a stack trace from such a hang?
That's the obvious first step that any application developer should know to do."

      On windows, if I dont set env_set_mapsize(env, 0) just before beginning
a write transaction, then first insert will complete successfully. On the second
      write transaction while the application is running, and we try to for
eg: mdb_cursor_get(),  it doesn't return. This can be reproduced with the below
      stated toolchain.


      -----------

      For further notes, I am using official QtSDK-mingw on windows which is a
32 bit compiler. But sometimes I can get LMDB to open the env
      succesffully for 2 gb, etc. But on a system restart, the situation is
still the same.
You can't address more than 2GB of memory in 32 bit code. You generally won't be able to create a 2GB mmap in a 32 bit process because other DLLs will already be occupying large chunks of the 32 bit address space. 


      But I downloaded an unofficial QtSDK-mingw-64 with the same result. But
      when I change
     mdb_size_t mapSize = 2147483648;    // 2 GB
      to
     mdb_size_t mapSize = 2147483648LL;

     it works.


     But setting 2147483648LL does not work (meaning does not return
MDB_SUCCESS) most of the times on mingw-32-bit


That is expected. Again, you should already know how your OS works.


     So the quesstion really is how to get it working when compiled on a
32-bit compiler and executed on both 32 and 64 bit Windows?

     Note: I am not referring to vl32 branch because I don't want to have to
use more than 2GB as the size of my env file.


It can't be done.



On Mon, Dec 28, 2015 at 2:17 PM, Howard Chu <hyc@symas.com
<mailto:hyc@symas.com>> wrote:

    Mohammed Muneer wrote:


        I can env_set_mapsize(env, 1 GB) without any problems in linux.

        But on windows, it is a different story


    Windows is like that.

           Machine 1) Windows 64 only opens with upto 10 MB (apporx)
           Machine 2 Windows 64 only opens with upto 500 MB (apporx)
           Machine 3) Virtual box Windows 64 running on linux works with 1 GB

        Q1) So why is this happening. Am I doing anything wrong.


    Have no idea what you're actually doing, so it's impossible to answer.
    Learn to ask better questions and provide better information.

    What revision of LMDB? Which version of Windows? How much free disk space
    is there? What else is your application code doing? How do you know that
    only XX MB were "opened", what stats are you checking? What other
    differences exist between machine 1 and machine 2?

    What other processes are running, how much memory are they consuming on
    each machine? Note that just because you create a 1GB memory map of a file
    doesn't mean that it will actually consume 1GB of RAM. That's not how
    memory mapping works. Read your Windows developer documentation, MSDN is
    always there for you to read. We're not here to teach you how your OS works.

        Q2) And setting env_set_mapsize before calling open is not enough. I
        need to set
                env_set_mapsize(env, 0) just before beginning a write transaction
        inorder to
               insert something or otherwise the application indefinitely
        hangs on the
        second insert transaction.


    Hangs where, exactly? Have you gotten a stack trace from such a hang?
    That's the obvious first step that any application developer should know
    to do.


        Both the above questions concerns just Windows. Everything is fine
        with linux.


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/