[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Enhancements to back-sql


your improvements sound interesting, and are definitely welcome. The form of your submission is up to you; it should be thru the ITS, in patch form. I don't mind if you send one single patch or separate ones for each feature. About the tree, HEAD is not __much__ different from 2.2.18, but there are indeed some changes, so HEAD is preferable. I guess in any case careful reviewing will be required for such a big patch, so 2.2.18 is fine as well, if HEAD costs you too much. In any case, we're too late for 2.2.19 and probably for 2.2.X at all :)

About the improvements:
- I just committed the entry_get feature. If your functionality is better than mine, I suggest you harmonize your commit accordingly, or remove it.
- about "uid" as a reserved word, I've never tried it, but I recall that the problem in Oracle could be fixed by wrapping alias words in double quotes. If this is correct, all you need to do is uncomment #define BACKSQL_ALIASING_QUOTE '"' in back-sql.h; I think this solution is cleaner and more efficient (on the slapd side, at least).

I'm very curious about the connection pooling.

Sincerely, p.

Mark Adamson wrote:

Here at Carnegie Mellon we're running the back-sql backend on our production master OpenLDAP server and we've reached a very stable environment. I have made about 10 enhancements to the backend and I would like to begin sending them back up to the repository. I have them in my source tree added to 2.2.18.

Two questions:

Is 2.2.18 a good tree to have them in, or is there a better tree, e.g. HEAD or the theoretical 2.3?

How would you like to see the changes? A series of unified diff's, each building on the previous? A tar of the whole back-sql?

Here's a summary of the enhancements, sorted somewhat in order of their code impact:

* Added an option to set the SQL query for the "id_query" operation.

* Added the backend API functions for the entry_get and entry_release
entry points. The functions are needed for performing ACLs which employ "dnattr", "group", and other clauses.

* Added an option to the SQL backend called "use_subtree_shortcut".
When a search is performed, the SQL query includes a WHERE clause
which says the DN must be "LIKE %<searchbase>". The LIKE operation
can be slow in an RDBM. This shortcut option says that if the
searchbase of the LDAP search is the root DN of the SQL backend,
and thus all objects will match the LIKE operator, do not include
the "LIKE %<searchbase>" clause in the SQL query (it is replaced
instead by the always true "1=1" clause to keep the "AND"'s working correctly). This option is off by default, and should be
turned on only if all objects to be found in the RDBM are under the
same root DN. Multiple backends working within the same RDBM table
space would encounter problems. LDAP searches whose searchbase are
not at the root DN will bypass this shortcut and employ the LIKE clause.

* Oracle has reserved the keyword "uid", and so tables cannot be
created with a column with that name. This is unfortunately a common
word for an LDAP server that stores information about computer accounts.
In order to have an LDAP attribute named "uid", a special exception was added for fetching a column named "uid". The SQL column can be
called anything, e.g. "username", but when the SQL query to fetch an attribute named "uid" is made, the "AS" clause changes the column name to
"uidtmp" instead of "uid". When ODBC is reading back results for the
"uid" attribute, it will seek the result column "uidtmp". This option is
shielded by a #define macro.

* Improved sr_text messages to tell what went wrong in backend,
instead of just "SQL-backend error".

* Added a "create_hint" column to ldap_oc_mappings table. Allows
taking the value of an attr named in "create_hint" and passing it to
the create_proc procedure.  This is necessary for when an objectClass's
table is partition indexed by some indexing column and thus the value
in that indexing column cannot change after the row is created. The
value for the indexed column is passed into the create_proc, which
uses it to fill in the indexed column as the new row is created.

* When loading the values of an attribute, the count(*) of the number
of values is fetched first and memory is allocated for the array of
values and normalized values. The old system of loading the values one
by one and running realloc() on the array of values and normalized
values each time was badly fragmenting memory. The array of values and
normalized values would be side by side in memory, and realloc()'ing
them over and over would force them to leapfrog each other through all
of available memory. Attrs with a large number of values could not be
loaded without crashing the slapd daemon.

* Added code to interpret the value returned by stored procedures
which have expect_return set. Returned value is interpreted as an LDAP
return code. This allows the distinction between the SQL failing to
execute and the SQL running to completion and returning an error code
which can indicate a policy violation.

* Added RDBM connection pooling. Once an operation is finished the
connection to the RDBM is returned to a pool rather than closing.
Allows the next operation to skip the initialization and authentication
phases of contacting the RDBM. Also, if licensing with ODBC places
a limit on the number of connections, an LDAP thread can block waiting
for another thread to finish, so that no LDAP errors are returned
for having more LDAP connections than allowed RDBM connections. An
RDBM connection which receives an SQL error is marked as "tainted"
so that it will be closed rather than returned to the pool.
  Also, RDBM connections must be bound to a given LDAP connection AND
operation number, and NOT just the connection number.  Asynchronous
LDAP clients can have multiple simultaneous LDAP operations which
should not share the same RDBM connection.  A given LDAP operation can
even make multiple SQL operations (e.g. a BIND operation which
requires SASL to perform an LDAP search to convert the SASL ID to an
LDAP DN), so each RDBM connection now has a refcount that must reach
zero before the connection is returned to the free pool.

* Added ability to change the objectClass of an object. Required considerable work to copy all attributes out of old object and into
new object. Does a schema check before proceeding. Creates a new
object, fills it in, deletes the old object, then changes the oc_map_id and keyval of the entry in the "ldap_entries" table.

   -Mark Adamson
    Carnegie Mellon

SysNet - via Dossi,8 27100 Pavia Tel: +390382573859 Fax: +390382476497