Attached is the specification for the LDAP Client Update Protocol (LCUP). The draft will be submitted to IETF later this week. LDUP will consider adding LCUP to its agenda at the coming IETF meeting.
Comments are welcome,
Olga Natkovich
Software Engineer, Sun-Netscape Alliance
Internet Draft O. Natkovich
Document: <draft-natkovich-ldap-lcup-00.txt> M. Smith
Category: Proposed Standard Netscape Communications
Corp.
February 2000
LDAP Client Update Protocol
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026 [1].
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet- Drafts as
reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-
Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
1. Abstract
This document defines the LDAP Client Update Protocol (LCUP). The
protocol is intended to allow an LDAP client to synchronize with the
content of a directory information tree (DIT) stored by an LDAP
server and to be notified about the changes to that content.
2. Conventions used in this document
In the protocol flow definition, the notation C->S and S->C
specifies the direction of the data flow from the client to the
server and from the server to the client respectively.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in RFC-2119
[KEYWORDS].
3. Overview
The LCUP protocol is intended to allow LDAP clients to synchronize
with the content stored by LDAP servers.
The problem areas addressed by the protocol include:
- mobile clients that maintain a local read-only copy of the
directory data. While off-line, the client uses the local copy of
the data. When the client connects to the network, it
synchronizes with the current directory content and can be
optionally notified about the changes that occur while it is on-
line. For example, a mail client can maintain a local copy of the
corporate address book that it synchronizes with the master copy
whenever the client gets connected to the corporate network.
- applications intending to synchronize heterogeneous data stores.
A meta directory application, for instance, would periodically
retrieve a list of modified entries from the directory, construct
the changes and apply them to a foreign data store.
- clients that need to take certain actions when a directory entry
is modified. For instance, an electronic mail repository may want
to perform a "create mailbox" task when a new person entry is
added to an LDAP directory and a "delete mailbox" task when a
person entry is removed.
The problem areas not being considered:
- directory server to directory server synchronization. The LDUP
replication protocol [LDUPPROT] should be used for this purpose.
Several features of the protocol distinguish it from LDUP
replication. First, the server does not maintain any state
information on behalf of its clients. The clients are responsible
for storing the information about how up to date they are with
respect to the server's content. Second, no predefined agreements
exist between the clients and the servers. The client decides when
and from where to retrieve the changes. Finally, the server never
pushes the data to the client; the client always initiates the
update session during which it pulls the changes from the server.
The set of clients that are allowed to synchronize with an LDAP
server is determined by the server defined policy.
There are, currently, several protocols available for LDAP client
server synchronization. While each protocol addresses the needs of a
particular group of clients (on-line clients in case of Persistent
[PSEARCH] and Triggered [TSEARCH] Search, off-line clients in case
of DirSync [DIRSYNC]), none satisfies the requirements of all
clients in the target group. For instance, a mobile client that was
off-line and wants to become up to date with the server and stay up
to date while connected can't be easily supported by any of the
above protocols.
4. Protocol Specification
This section describes the protocol elements and the protocol flow.
4.1 Protocol Elements
Natkovich Proposed Standard - Expires: August 2000 2
A client initiates a synchronization session with a server by
attaching a clientUpdate control to a search operation. The search
specification determines the part of the directory information tree
(DIT) the client wishes to synchronize with, the set of attributes
it is interested in and the amount of data the client is willing to
receive. The clientUpdate control contains the client's
synchronization specification. The control has the following format:
clientUpdateControlValue ::= SEQUENCE{
cookie OCTET STRING OPTIONAL
keepConnection BOOLEAN DEFAULT FALSE
changesOnly BOOLEAN DEFAULT FALSE
}
cookie - an opaque cookie that represents the current state of the
client's data.
keepConnection - if set to TRUE, indicates that the server should
keep the connection open after the initial synchronization and
should notify the client of modifications to the data. The
connection should stay open until the client abandons the search
operation, sends the stopClientUpdate extended operation, or
closes the connection.
changesOnly - if set to TRUE, the keepConnection and cookie fields
of the control are ignored by the server. In response, the server
skips the initial synchronization and only notifies the client
about the changes that occur to the data while the client is
connected. This feature is useful if the client is not interested
in data synchronization but needs to trigger events in response
to data modifications.
In response to the client's synchronization request, the server
returns a set of SearchResultEntries that fits the client's
specification. To represent deleted entries, the server attaches an
entryUpdate control to the SearchResultEntry. Furthermore, the
server may elect to periodically return to the client the cookie
that represents the state of the client's data. This information is
useful in case the client crashes or gets disconnected. The cookie
is also provided in the entryUpdate control. The control has the
following format:
entryUpdateControlValue ::= SEQUENCE{
cookie OCTET STRING OPTIONAL
stateUpdate BOOLEAN DEFAULT FALSE
entryDeleted BOOLEAN DEFAULT FALSE
}
cookie - an opaque cookie that represents the current state of the
client's data.
stateUpdate - if set to TRUE, indicates that the entry to which the
control is attached contains no changes and it is sent only to
Natkovich Proposed Standard - Expires: August 2000 3
communicate to the client the new cookie. In this case, the
entryDeleted field MUST be ignored and the cookie field WILL
contain the updated cookie. This feature allows updating the
client's cookie when there is no changes that effect the client's
data store. Note that the server MUST attach the control to a
valid entry. The server COULD always send the entry at the root
of the client's tree.
entryDeleted - if set to TRUE, indicates that the entry to which
the control is attached was deleted.
When the server has finished processing the client's request, it
attaches a clientUpdateDone control to the SearchResult message and
sends it to the client. The control has the following format:
clientUpdateDoneControlValue ::= SEQUENCE{
cookie OCTET STRING OPTIONAL
reload BOOLEAN DEFAULT FALSE
}
cookie - an opaque cookie that represents the current state of the
client's data.
reload - if set to TRUE, indicates that the server does not contain
sufficient information to synchronize the client or that the
server's data was reloaded since the last synchronization
session. This field indicates to the client that the client's
data store needs to be reinitialized.
If the client needs to terminate the synchronization process and it
wishes to obtain the cookie that represents the current state of its
data, it issues a stopClientUpdateRequest extended operation. The
operation carries no data. The server responds with a
stopClientUpdateResponse extended operation that has the following
format:
stopClientUpdateResponseValue ::= SEQUENCE {
cookie OCTET STRING
}
cookie - an opaque cookie that represents the current state of the
client's data.
If the client is not interested in the state information, it can
simply abandon the search operation or disconnect from the server.
If server resources become tight, the server can terminate one or
more search operations by sending a SearchResult message to the
client(s). Unless the client sets the changesOnly field to TRUE, the
server attaches a clientUpdateDone control that contains the cookie
that corresponds to the current state of the client's data and the
reload flag set to 0. A server set policy is used to decide which
searches to terminate. This can also be used as a security
Natkovich Proposed Standard - Expires: August 2000 4
mechanism to disconnect clients that are suspected of malicious
actions.
4.2 Protocol Flow
The client server interaction can proceed in three different ways
depending on the client's requirements.
If the client's intent is not to synchronize data but to trigger
actions in response to directory modifications, the protocol
proceeds as follows:
C->S Sends a search operation with a clientUpdate control attached.
The search specification determines the part of the DIT the
client wishes to synchronize with and the set of attributes it
is interested in. The changesOnly field of the control should
be set to TRUE; other fields are ignored.
S->C Sends change notification to the client for each change to the
data within the client's search specification.
S->C If the server starts to run out of resources or the client is
suspected of malicious actions, the server can terminate the
search operation by sending a SearchResult message to the
client.
C->S Abandons the search operation or disconnects from the server.
S->C Stops sending changes to the client and closes the connection.
If the client's intent is to synchronize with the server and then
disconnect, the protocol proceeds as follows:
C->S Sends a search operation with the clientUpdate control
attached. The search specification determines the part of the
DIT the client wishes to synchronize with, the set of
attributes it is interested in and the amount of data the
client is willing to receive. If this is the initial
synchronization session, the client does not provide a cookie;
otherwise, the cookie field of the control is set to the
cookie received from the server at the end of the last
synchronization session. (Note that the client can synchronize
with different servers during different synchronization
sessions.) The keepConnection and changesOnly fields are set
to FALSE.
S->C If no cookie is specified in the clientUpdate control, the
server sends all data that matches the client's search
specification followed by the SearchResult message with a
clientUpdateDone control attached to it. The control contains
the cookie that corresponds to the current state of the
client's data and the reload flag set to FALSE.
If an invalid cookie is specified the server sends back an
unwillingToPerform error.
If a valid cookie is specified and the data that matches the
search specification has been reloaded or the server does not
contain enough state information to synchronize the client,
the server sends a clientUpdateDone control with the reload
field set to TRUE and no cookie.
Natkovich Proposed Standard - Expires: August 2000 5
If the client is up to date, the server sends a success
response to the client.
If the cookie is valid and there is data to be sent, the
server sends the modified entries to the client. Each
SearchResultEntry contains the attributes requested by the
client in the search specification regardless of whether they
were modified. An entryUpdate control with the entryDeleted
field set to TRUE is attached to every deleted entry. The
server may also periodically attach an entryUpdate control to
the entries sent to the client to indicate the current state
of the client's data. In that case, the cookie field of the
control represents the state of the client's data including
the entry to which the control is attached. Once all the
changes are sent, the server sends a SearchResult with the
clientUpdateDone control attached. The control contains the
cookie that represents the current state of the client's data.
The reload field of the control is set to FALSE.
C->S If the reload field of the control is set to TRUE, the client
clears its data store and repeats the synchronization process
by sending the search operation with clientUpdate control that
contains no cookie. Otherwise, the client stores the cookie
received from the server until the next synchronization
session.
If the client's intent is to be synchronized with the server and
stay notified about data modifications, the protocol proceeds as
follows:
C->S The client behaves exactly as in the previous case except it
sets the keepConnection control field to TRUE.
S->C The server behaves exactly as in the previous case except the
connection is kept open after the initial set of changes is
sent to the client. A SearchResult message is not sent to the
client; instead, the server keeps sending changes to the
client.
S->C If the server starts to run out of resources or the client is
suspected of malicious actions, the server can terminate the
search operation by sending a SearchResult message with the
clientUpdateDone control back to the client.
C->S Sends a stopClientUpdateRequest extended operation to the
server to terminate the synchronization session.
S->C Responds with a stopClientUpdateResponse extended operation
with the cookie representing the current state of the client's
data.
4.3 Size and Time Limits
The search request size or the time limits can only be imposed for
non-persistent operations, those that set keepConnection field of
the clientUpdateControlValue to FALSE. All other operations SHOULD
set both limits to 0. The server SHOULD ignore the limits set for
persistent operations.
4.4 Changes vs. Operations
Natkovich Proposed Standard - Expires: August 2000 6
Since the server sends to the client the modified entries rather
than the operations, a MODDN operation performed on a subtree will
be seen by the client as a sequence of added or modified entries
depending on whether the operation moved the entries into the scope
of the client's search specification.
5.0 Additional Features
There are several features present in other protocols or considered
useful by clients that are currently not included in the protocol
primarily because they are difficult to implementing on the server.
These features are briefly discussed in this section. This section
is intended to open a discussion on the merits of including and
approaches of implementing these features.
5.1. Change Type
This feature is present in the Triggered Search [TSEARCH]
specification. A flag is attached to each entry returned to the
client indicating the reason why this entry is returned. The
possible reasons from the draft are
"- notChange: the entry existed in the directory and matched the
search at the time the operation is being performed,
- enteredSet: the entry entered the result set for one of the
reasons defined in section 4 above,
- leftSet: the entry left the result set for one of the reasons
defined in section 4 above,
- modified: the entry was part of the result set, was modified or
renamed, and still is in the result set."
The leftSet feature is particularly useful because it indicates to
the client that an entry is no longer within the client's search
specification and the client can remove the associated data from its
data store. Ironically, this feature is the hardest to implement on
the server because the server does not keep track of the client's
state and has no easy way of telling which entries moved out of
scope between synchronization sessions with the client.
A compromise could be reached by only providing this feature for the
operations that occur while the client is connected to the server.
This is easier to accomplish because the decision about the change
type can be made based only on the change without need for any
historical information. This, however, would add complexity to the
protocol.
5.2. Sending Changes
The DirSync protocol [DIRSYNC] sends to the clients only the
modified attributes of the entry rather than the entire entry. While
this approach can significantly reduce the amount of data returned
to the client, it has several disadvantages. First, unless a
separate mechanism (like the change type described above) is used to
notify the client about entries moving into the search scope,
Natkovich Proposed Standard - Expires: August 2000 7
sending only the changes can result in the client having an
incomplete version of the data. Let's consider an example. An
attribute of an entry is modified. As a result of the change, the
entry enters the scope of the client's search. If only the changes
are sent, the client would never see the initial data of the entry.
Second, this feature is hard to implement since the server might not
contain sufficient information to construct the changes based solely
on the server's state and the client's cookie. On the other hand,
this feature can be easily implemented by the client assuming that
the client has the previous version of the data and can perform
value by value comparisons.
5.3. Data Size Limits
The DirSync protocol [DIRSYNC] allows clients to control the amount
of data sent to them in the search response. The client can specify
the number of bytes it is willing to receive by setting the
maxReturnLength field of the DirSync control. This feature is
intended to allow clients with limited resources to process
synchronization data in batches. However, an LDAP search operation
already provides the means for the client to specify the size limit
by setting the sizeLimit field in the SearchRequest to the maximum
number of entries the client is willing to receive. While the
granularity is not the same, the assumption is that LCUP protocol
will be implemented by regular LDAP clients that can deal with the
limitations of the LDAP protocol.
5.4. Data Ordering
The DirSync protocol [DIRSYNC] allows a client to specify that
parent entries should be sent before the children for add operations
and children entries sent before their parents during delete
operations. This ordering helps clients to maintain a hierarchical
view of the data in their data store. While possibly useful, this
feature is relatively hard to implement and is expensive to perform.
6. The Protocol and the LDUP Architecture
The LDAP Client Update Protocol is defined within the framework of
the LDUP Architecture [LDUPARCH]. The following aspects of the
protocol are drawn from the architecture:
- The scope of each search operation is restricted to a single
replica as defined in the LDUP architecture document [LDUPARCH].
- Each entry returned to the client contains a unique identifier as
defined in the LDUP architecture document [LDUPARCH]. The client
can use the identifier to unambiguously cross reference objects
stored on the server with those in the client's store.
- One of the main criteria for selecting the protocol features is
that an LDUP compliant server can implement these features
efficiently.
Natkovich Proposed Standard - Expires: August 2000 8
7. Client Side Considerations
There are several issues that the implementors of a synchronization
client need to consider:
- The cookie received from the server after a synchronization
session can only be used with the same or more restrictive search
specification than the search that generated the cookie. The
server will reject the search operation with a cookie that does
not satisfy this condition. This is because the client can end up
with an incomplete data store otherwise. A more restrictive
search specification is the one that generates a subset of the
data produced by the original search specification.
- Because an LCUP client specifies the area of the tree with which
it wishes to synchronize through the standard LDAP search
specification, the client can be returned nsSuchObject error if
the root of the synchronization area was renamed between the
synchronization sessions. If this condition occurs, the client
can attempt to locate the root by using the root's uniqueid saved
in client's local data store. It then can repeat the
synchronization request using the new search base. In general, a
client can detect that an entry was renamed and apply the changes
received to the right entry by using uniqueid rather than DN
based addressing.
8. Server Implementation Considerations
By design, the protocol does not specify the format of the cookie.
This is to allow different implementations the flexibility of
storing any information applicable to their environment. A
reasonable implementation for an LDUP compliant server would be to
use the Replica Update Vector (RUV). For each master, RUV contains
the largest CSN seen from this master. In addition, the RUV
implemented by the iPlanet Directory Server (not yet in LDUP)
contains replica generation - an opaque string that identifies the
replica's data store. The replica generation value changes whenever
the replica's data is reloaded. Replica generation is intended to
signal the replication/synchronization peers that the replica's data
was reloaded and that all other replicas need to be reinitialized.
RUV satisfies the three most important properties of the cookie: (1)
it uniquely identifies the state of client's data, (2) it can be
used to synchronize with multiple servers, and (3) it can be used to
detect that the server's data was reloaded.
In addition, the cookie must contain enough information to allow the
server to determine whether the cookie can be safely used with the
search specification it is attached to. As discussed earlier in the
document, the cookie can only be used with the search specification
that is equally or more restrictive than the one for which the
cookie was generated.
Natkovich Proposed Standard - Expires: August 2000 9
An implementation must make sure that it can correctly update the
client's cookie when there is a size limit imposed on the search
results by either the client's request or by the server's
configuration. If RUV is used as the cookie, entries last modified
by a particular master must be sent to the client in the order of
their last modified CSN. This ordering guarantees that the RUV can
be updated after each entry is sent.
An implementation must be able to notify the client about all
entries deleted since the last implementation session. An LDUP
compliant implementation can achieve this through the use of entry
tombstones. The implementation should avoid aggressive tombstone
purging since lack of tombstones would cause client's data to be
reloaded. We suggest that only the tombstone content be removed
during the regular trimming cycle while tombstones themselves are
discarded much less frequently.
The specification makes no guarantees about how soon a server should
send notification of a changed entry to the client when the
connection between the client and the server is kept open. This is
intentional as any specific maximum delay would be impossible to
meet in a distributed directory service implementation. Server
implementors are encouraged to minimize the delay before sending
notifications to ensure that clients' needs for timeliness of change
notification are met.
9. Synchronizing Heterogeneous Data Stores
Clients synchronizing multiple writeable data stores, like iPlanet
Meta Directory, will only work correctly if each piece of
information is single mastered (for instance, only by an LDUP
compliant directory or only by Oracle). This is because different
systems have different notions of time and different update
resolution procedures. As a result, a change applied on one system
can be discarded by the other, thus preventing the data stores from
converging.
10. Security Considerations
In some situations, it may be important to prevent general exposure
of information about changes that occur in an LDAP server.
Therefore, servers that implement the mechanism described in this
document SHOULD provide a means to enforce access control on the
entries returned and MAY also provide specific access control
mechanisms to control the use of the controls and extended
operations defined in this document.
As with normal LDAP search requests, a malicious client can initiate
a large number of persistent search requests in an attempt to
consume all available server resources and deny service to
legitimate clients. The protocol provides the means to stop
malicious clients by disconnecting them from the server. The servers
that implement the mechanism SHOULD provide the means to detect the
Natkovich Proposed Standard - Expires: August 2000 10
malicious clients. In addition, the servers SHOULD provide the
means to limit the number of resources that can be consumed by a
single client.
Access control on the data can be modified in such a way that the
data is no longer visible to the client. The specification does not
specify how the server should handle this condition. Moreover, data
consistency is not guaranteed if access control is changed from a
more restrictive to a less restrictive one. This is because access
control can be considered as an additional filter on the search
specification and the protocol does not support going from a more to
a less restrictive search specification. See Client Side
Considerations Section for more detailed explanation of the problem.
11. References
[KEYWORDS] S. Bradner, "Keywords for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997.
[PSEARCH] M. Smith "A Simple LDAP Change Notification Mechanism",
INTERNET-DRAFT <draft-ietf-ldapext-psearch-01.txt>,
August 1998.
[TSEARCH] M.Whal "LDAPv3 Triggered Search Control", INTERNET-DRAFT
<draft-ietf-ldapext-trigger-01.txt>, August 1998.
[DIRSYNC] M. Armijo "Microsoft LDAP Control for Directory
Synchronization", INTERNET-DRAFT <draft-armijo-ldap-
dirsync-00.txt>, August 1999.
[LDUPARCH] J. Merrells, E. Reed, U. Srinivasan, "LDAP Replication
Architecture", INTERNET-DRAFT <draft-ietf-ldup-model-
02.txt>, October 1999.
[LDUPPROT] E. Stokes, G. Good "The LDUP Replication Update
Protocol", INTERNET-DRAFT <draft-ietf-ldup-protocol-
00.txt>, October 1999.
12. Author's Addresses
Olga Natkovich
Netscape Communications Corp.
501. E. Middlefield Rd., Mailstop MV068
Mountain View, CA 94043
Phone: +1 650 937-4788
Email: olga@netscape.com
Mark Smith
Netscape Communications Corp.
501. E. Middlefield Rd., Mailstop MV068
Mountain View, CA 94043
Phone: +1 650 937-3477
Natkovich Proposed Standard - Expires: August 2000 11
Email: mcs@netscape.com
Full Copyright Statement
"Copyright (C) The Internet Society (date). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implmentation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph
are included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Natkovich Proposed Standard - Expires: August 2000 12