CVS log for servers/slapd/overlays/syncprov.c

Up to [OpenLDAP] / servers / slapd / overlays

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

ITS#6812 cleanup error codes

cleanup: Rename rs_ensure_entry_modifiable().

The name rs_entry2modifiable() is more reasonably sized.
For now, leave behind a macro rs_ensure_entry_modifiable in proto-slap.h.

ITS#6795 plug sr_ctrls memleak

ITS#6717 refine mincsn check for MMR

ITS#6716 partial revert, initialize sl_mincsn at db_open

noop change: Silence signed vs unsigned warnings

More for #6503

ITS#6503 - record Adds in sessionlog too, to avoid excessive present phases

Happy New Year

ITS#6758 Use rs_*() to manage SlapReply entries.
Some noop changes + fixes entry leaks and REP_ENTRY_MODIFIABLE flag leaks.

ITS#6758 Avoid SlapReply reuse, simple cases.
Caller provides and reuses a SlapReply it does not read after
initialization.  So reinitialize or move the declaration inwards.
In case of refint, move the rs parameter into refint_repair().

noop commit for ITS#6758: simplify future commits

noop commit for ITS#6758: Remove no-effect code.
Unused variable resets, unused SlapReply parameters to some functions.

Remove lint (unused vars/labels/funcs/default:s)

Plug mutex/rwlock leaks (destroy them)

ITS#6716: Keep sessionlog sorted by csn, compare mincsn with head.

ITS#6718: Send NEW_COOKIE messages to all consumers.

ITS#6710: Force clients to refresh if context is dirty.

ITS#6709: Wait for active mods to finish before initiating new consumers.

ITS#6708: Mutex protect fixing the filter.

ITS#6606 send error if consumer is newer than provider

unregister controls when removing syncprov, ppolicy, valsort, chain or
sssvlv overlay

Send error to active psearches upon overlay removal

During the refresh phase the sync filter needs to be adjusted (skipping
the "(entrycsn>=cookie)" part that was inserted) when checking whether a
change needs to be replicated, otherwise we lose DELETES that happen during
the refresh phase (ITS#6555).

Reset BackendInfo to original value after checkpoint. Plugs a memory leak
when deleting the syncprov overlay from a database (when compiled with
-DSLAP_CONFIG_DELETE)

happy belated New Year

Plugged memory leak. Syncentry wasn't free'd when sending
LDAP_SYNC_NEW_COOKIE intermediate results. (ITS#6459)

ITS#6438 check for redundant syncops_free() ?

more for #6397

ITS#6397 cleanup psearch if cookie is stale

Revert prev commit, simpler fix...

Short-circuit backend_operational; sent entry's e_private doesn't belong
to the backend any more

ITS#6368 use dup'd entries in response queue

Fix prev commit

in op_response, rearrange contextCSN update checks

Mostly revert prev commit, broke test054

Always log outgoing cookie, just use op->o_csn instead of commit_csn since
all responses are now serialized.

ITS#6368 partially revert 1.273, always check for skips, not just on
first matchops call

Use fake_init2 in db_open/db_close since they can execute from a
back-config operation (and could wipe out the state otherwise)

Cleanup for #6373

Fix prev commit

ITS#6376 plug potential leak

ITS#6373 more for subentry creation

ITS#6373 create subentry if missing

ITS#6373 sync_use_subentry support

unparse checkpoint time in minutes (ITS#6370)

ITS#6346 serialize all responses

ITS#6335 Don't reuse a modtarget someone is about to remove

wrap gmtime for reentrancy (ITS#6262)

More for ITS#6045: clear o_callback as well.

Fixed previous again, using o_bd->bd_self is safer.

Fixed previous edit: Use op->o_bd, not a copy of ss->s_op->o_bd

ITS#6045: Use copy of db when testing filter in matchops.

ITS#6024 Don't send cookies without csn.

Fix prev 2 commits: make sure op is Modify before testing modlist,
don't checkpoint when adding the context entry to a DB

ITS#5988 allow entries without CSN to be replicated

ITS#5973 re-enable patch

Fix prev commit

ITS#5988 don't respond if consumer's state is newer than ours

ITS#5999 better modtarget tracking

Avoid shutdown problems

Cleanup prev commit

More for #6020

For #6020 log remote sid in syncprov_sendresp

More for prev commit

ITS#5973 hide this for now, it's broken

More for prev commit

Fix prev commit

ITS#6020 better tracking of where changes came from

Check rc in prev commit

ITS#5985 Only play one queued response at a time per psearch thread

ITS#5973 use the modify op to pass csn updates from syncrepl to syncprov

ITS#5972 Added newCookie sync info messages.

ITS#5969 update consumers csn when providers change.

Fix corrupted CSN issue

(Temporarily) define CHECK_CSN for all LDAP_DEVEL builds

More CHECK_CSN tests

Fix compare op

Add some asserts on CSN validity

Reverse lock order when detaching op

ITS#5881 consumer must use ManageDSAiT control to retrieve referrals
as entries. provider must preserve incoming ManageDSAiT status.

Update copyright notices

ITS#5827, release syncop after mod op completes

ITS#5710 fix stupid updateCookie hack, use explicit o_dont_replicate flag

Add DBFLAG_CLEAN for a DB that was cleanly shut down; don't bother
scanning to verify contextCSN for clean shutdowns (ITS#5640)

Really fix ITS#5709 - more explicit check for un-replicatable ops, and
don't send ctxcsn update if current maxcsn doesn't match current csn

ITS#5776 prevent duplicate removal of qtask from runqueue.

Warning cleanup: Remove no-op bv_len < 0 and bv_len >= 0 tests

silence warnings

ITS#5596 contextCSN updates are internal operations.

allow olcSpReloadHint in olcSyncProvConfig (ITS#5591)

ITS#5589 delcsn / NULL cookie

ITS#5564 ignore clients contextCSN when doing full reload.

ITS#5548 use syncrepl Connection when evaluating filter and acl.

Forced commit - prev was ITS#5537, not 5337

ITS#5337 fix maxcsn search at startup

Fix rev 1.220 - remove extra free_syncop()

ITS#5488 from Rein Tollevik: propagate syncrepl contextCSN to providers

ITS#5487 from Rein Tollevik: syncprov_findbase searched wrong backend

ITS#5465 ignore ops without queued CSNs

ITS#5506 reset op2.o_extra

ITS#5493 always send cookie with delta-sync entries

ITS#5486, fix rev 1.219. Must never use s_op directly, must always use
a copy.

Avoid queueing response on abandoned psearches

ITS#5451 glue deadlock fix

ITS#5484 remove syncop from runq before freeing it

Revert prev commit

ITS#5470 don't return any data at all until we have a ctxcsn

ITS#5437, #5444 if numcsns don't match, set changed flag

fix bitwise test (ITS#5445)

ITS#5434 again

ITS#5432 don't send cookie with delete set if there's no true delete CSN

ITS#5434 search thru other overlays when present

ITS#5433 checkpoint must write to master DB

Use <ldap_queue.h> macros for prev commit

ITS#5401 make sure connection is still alive before detaching op

use the replication persistent search identity to check whether a modification must be replicated (ITS#5418)

ITS#5407 more checks for pool pausing

Fix prev commit (thanks Ando)

ITS#5405 fix race condition

ITS#5367 revert prev commit

Cleanup

Happy New Year (belated)

ITS#5261 quick fix, don't checkpoint on first write to DB

ITS#5268 omit contextCSN when replicating

ITS#5211 spec says we must ignore invalid cookies

Forced commit - prev was for ITS#5210, not 5120

More for prev commit, ITS#5120.

Don't send cookie at end of refresh if nothing changed in refreshAndPersist

ITS#5177 pass new entries in search_response for Persist mode

Fix unprotected checks for psearches

Plug leak

Don't queue zero-length CSNs

log cookie (until we find what's going on...)

first round of s/sprintf/snprintf/; the rationale is: truncate error messages rather than buffer overflow; otherwise, assert that no buffer overflow occurred.  In some cases, error handling could be added. Please note: no real error in the code was found so far, apart from that in syncprov.c and from some config-time error logging; however, errors could slip thru again if things don't change consistently

Use attr_valfind instead of value_find_ex to optimize lookups

Preliminary sorted value support

More for a_numvals

Attribute/Modification value counters

add helper for internal searchs for objectClass=*

plug yet another leak

Reworked recent backend API changes, now using a separate struct,
(ConfigReply) for the error messages.

Added a new parameter (ConfigArgs*) to the _db_init, _db_open, _db_close and
_db_destroy functions.

ITS#5048 fully initialize fc_limits

ITS#5049: Initialize conn in syncprov_db_close() before connection_fake_init().

ITS#4977 check for cookie parse failure

more ITS#4973 initialization

Give OperationBuffer normal struct members instead of using
LBER_ALIGNED_BUFFER(OPERATION_BUFFER_SIZE), as discussed in ITS#4078.
Add a corresponding SyncOperationBuffer in overlays/syncprov.c.

ITS#4961 disable schema checking on contextCSN update

Distinguish absent control value from empty control value.

cleanup struct declarations

Plug leaks

Added "slapd" rewrite map handler, connection_fake_init2 to use existing
tmpmemctx without reinitializing

Cleanup si_ctxcsn and si_sids in db_destroy

Coverity leak

fix crash related to ITS#4839; other issues seem to presist

Improve prev commit

Re-fix ITS#4720, use rwlock

Partially revert prev commit, delete unused parameter, fix psearch response

ITS#4623, entry_get has to pass thru glue layer too

Fix typo

Fix prev commit - cannot shortcut initial sync

Optimization: send sid in cookie to avoid getting our changes sent
back to us.

Fix prev commit, don't generate a new ctxcsn if we're a consumer with
an empty DB

Preliminary multi-context support for syncrepl. Passes all single-master
tests, needs multi-master testing.

Fix debug args

Log the error code too

ITS#4810 log a message if be_entry_get doesn't return the entry we want

ITS#4813 don't limit the presence list to the current snapshot. add
some debug msgs for the sessionlog list.

Don't use ldap_pvt_thread_pool_context_reset(), it interferes with
dynamic adds.

happy new year

complain if instantiated outside of a database definition

ITS#4742 cleanup config error reporting

ITS#4720 release csn mutex before checkpointing.

Tweak prev commit, avoid referencing syncop when resetting qtask

ITS#4658 propagate errors from syncprov_sendresp; drop qtask on errors.
Note: need a way to close the connection too.

Entry/Attribute struct caching, to minimize malloc fragmentation
Note: this breaks LDAP_COMP_MATCH and SLAP_ZONE_MALLOC. But they
were probably broken already anyway.

TS#4622 deletes in psearch didn't always propagate

ITS#4622 if any present list was sent, must send refreshPresent

ITS#4622 make sure queued psearch responses get played after refresh
phase completes. Ignore empty sessionlog

Delete curcsn arg to slap_get_commit_csn, redundant with op->o_csn

ITS#4589 slap_get_commit_csn occasionally returns a curcsn that points
to already-freed memory. That bug still needs to be tracked down. This
patch deletes curcsn and just uses op->o_csn, which did not exist when
this code was originally written.

ITS#4622
  syncrepl: don't save contextCSN on incoming context entry
  syncprov: delete contextCSN on checkpoint if ctxcsn is empty

ITS#4613 - fail to start if lastmod is off

ITS#4562 - don't assume syncprov_op_response only runs in overlay context;
under a glue setup it may be get a real DB, don't corrupt the DB struct.

ITS#4534 propagate CSN for delete syncID sets

ITS#4423, fix sessionlog lower bound in playlog

More logging for ITS#4423

Init & destroy so.s_mutex around syncprov_findbase() call in syncprov_op_search

Cleanup

ITS#4417 backglue must pass intermediate responses transparently.
syncprov must prevent send_ldap_result from being called for an
in-progress persistent search.

ITS#4365 reset frs.sr_nentries before searching in syncprov_playlog

ITS#4364 add filter_dup, don't use str2filter to dup filters.

silence (harmless) warnings

wake listener when scheduling qtask so it can run immediately

resource release (more is needed: a report is coming...)

Happy new year!

typos in comments (very pedantic...)

ITS#4238 don't hide syncrepl control from rootDSE

a back-ldap consumer segfaults when accessing members of conn...

Improve previous commit

rework previous commit?

don't search max CSN if don't have a value; try the operational_attrs if entryCSN is not in attrs

plug leaks

Remove some unnecessary #ifdefs

streamline overlay configuration

use lutil_ato*() whenever appropriate

More for ITS#3939

Fix back-bdb inequality_candidates to use unchecked limit.
Fix syncprov_findcsn to set unchecked limit. (ITS#3939)

improve fix to ITS#4173

don't muck with SlapReply during response (ITS#4173)

gdb is much more clever with enums...

plug one-time leak

Update result code to IANA-assigned values
Provide some backwards compatibility.
Misc cleanup

Use generic filter for findbase

The rest of the reloadHint support...

Add option to enable using the reloadHint. 2.2 consumers never sent this
hint. The 2.3 consumer needs it for delta-syncrepl.

Move CSN invocations to backends

Align Operation buffers using LBER_ALIGNED_BUFFER

ITS#4068 fix queued UUIDs

Plug memleaks

Don't flush psearch queue in main search, let the qtask do it.

Fix abandon checks, syncop use count.

Fix prev commit op initialization, runqueue startup latency

Tweak prev commit, avoid rescheduling running qtask

Fix runq scheduling

Fix prev commit modrdn

Minimize base lookups

Always queue psearch responses (ITS#3671 revisited)

Add a "nopresent" keyword to skip the present phase; for backends that are
only used for logging (c.f. accesslog overlay) the present phase is useless.

ITS#3989 more fixes

ITS#3989 fix crash - don't access freed psearch records

ITS#3989 fix ID used for syncprov_findbase

Remove unused variable

need stricter sanity checks...

ITS#3961 better fix - provide a context for the main thread. The context
must be reset by the caller to clear out temp allocations etc.

ITS#3961 provide ldap_pvt_thread_pool_fake_context_init/destroy, don't
use arbitrary context pointers.

More optimization for CSN filter checks

Fix prev commit

Optimize FINDCSN case

ITS#3935 tweak sessionlog description

ITS#3922 don't leak ber buffer on decode/parse errors

ITS#3922 plug another cookie parsing leak

ITS#3922 fix syncrepl ctxcsn leak

Fix off-by-one pointer error. Garbage byte between uuid and csn in slog_entry.

assert expects int. (int)<nonnull ptr/long> can be 0. Use assert(arg!=0/NULL).

Silence warnings

fbuf in syncprov_findcsn() got used before it was fully filled in

ITS#3456 better fix, use a child thread.

ITS#3456 - omit FIND_MAXCSN call from syncprov_db_open. Will probably
delete all associated code soon; since the maxcsn will be updated as
soon as any write op occurs, it's no big deal for this to be out of date.
Also, since the maxcsn will get checkpointed on close, it will only get
out of date on an actual slapd crash, which is hopefully too rare to
worry about.

Add SLAP_MOD_INTERNAL flag to Modifications, allow internal ops to bypass
ACL checks when modifying user-modifiable attributes as non-root user.

Fix debug msg

Table-driven config

ITS#3671 more abandon/error checks

Mroe abandon checks for ITS#3671

More for ITS#3671, check for Abandon before replaying queued events

ITS#3682, dup entry and free the DB's copy before sending it

ITS#3671 make sure to flush queue when able

Fix modify deadlock

Better fix for #3671

ITS#3671 free all locks before calling send_search_*, queue entries if
the connection is busy. Probably need to set a queue limit here and
abandon the psearch if the limit is reached.

ITS#3616 store contextCSN in glue entry if database suffix is empty

Don't do ctxcsn checks in Tool mode, don't generate ctxcsn if it's missing
and we're also a consumer.

Fixing memory leakage in attribute aliasing

ITS#3596 partial fix - allow abandoning a locked operation

Fix cancel cleanup

ITS#3549 acknowledge CANCEL requests

fix dynamically registered controls check; NOTE: controls registered by global overlays don't work yet; better leave Kurt's kludge in place right now

silence warnings

More consolidation for single consumer-context syncrepl

Fix prev commit

Use a single sessionlog per context, delete sid

More cleanup for prev commit

ITS#3463 don't muck with frontend's operations, just our detached ones

Fix findmax when multiple matches are found

make sure the attribute exists

Happy New Year!

cleanup

Fix playlog uninit'd fields

Moved cookie validation into slap_parse_sync_cookie so consumer uses it too

Don't reject invalid cookies, just ignore them. partially revert prev.

Fix, don't record psearch until after it has been validated

Fix abandon, only drop our cloned ops, not real frontend ops

Fix abandon processing

Persist mode fixes

Fix 1.48 change

Add better validation of received cookie csn

A better approach for detach_op

ITS#3426, copy group ACL cache to saved Op

Fix playlog search

Added sessionlog support. consumer needs work...

Fix findpresent, include user's filter

Rearrange shortcut check for context with no changes

Preallocate syncuuid set instead of realloc/freeing

restore code in search_cleanup to free ctrls

Write contextCSN during db_open if it's newer than was previously recorded

Fix mod locking

Forced commit - add CVS ID tag

Restored findcsn( MAX ), used at db_open time

_operational should return SLAP_CB_CONTINUE

Cleanup - enable config, delete unused code

Cleanup... never replicate syncConsumersubentries. Plug memleaks.

Partially revert previous commit. Fix startup csn generation.
Fix findbase initialization.

generate contextCSN if not available; don't fail if a referral is sent (or don't accept referrals?)

Use rdwr lock for si_mods to avoid deadlock

Fix syncprov_db_close, only cleanup thrctx once

make unregistered known controls show up as undefined; move LDAPsync cid to frontend; some cleanup

Let the backend know there's a sync search

Add checkpointing to save contextCSN periodically. Read contextCSN
on startup, save on shutdown.

Cleanup controls

Fix modify serialization

Fix overlay aux_operational, add contextCSN as operational attribute
of suffix entry

Force a refresh if the search base has changed

Cleanup

Serialize multiple modifies of the same entry (ala seqmod.c)

Fix abandon sequencing

Added abandon/cancel support

More migration of syncrepl from frontend to syncprov overlay
define OPERATION_BUFFER_SIZE for dummy Operations

Self-contained detached operation

Added response queuing for updates occurring during a refresh

Fix prev commit

More syncprov fixes. test019 now passes.

Split Operation into Opheader and op

Disable back-bdb native syncrepl support, enable syncprov overlay,
remove syncrepl stuff from Operation struct

Merge op->o_sync_mode into op->o_sync

Cleanup, add some more descriptive comments.
Still needs: error checking, queuing results when refresh is in progress

Persist passes test018

Use normalized UUID...

First pass at persist, not working

Tweak findcsn conditions for validating cookie csn

Passes test017

More refresh working...

Getting refresh working

A bit more...

Incomplete skeleton

ITS#5401

Remove fix for ITS#5465, causes ITS#5385 to worsen.

ITS#5465 ignore ops without queued CSNs

ITS#5493

Happy new year! (belated)

ITS#5177 from HEAD

ITS#5048, #5049 from HEAD

ITS#4623 fix from HEAD

ITS#4977 fix from HEAD, check parse_sync_cookie failure

Cleanup prev commit, new ITS#4720 fix from HEAD

Partial sync with HEAD - ITS#4810, 4813, more debug msgs

happy new year

Complain if syncprov is instantiated outside of a database definition

ITS#4720 fix from HEAD

Additional ITS#4622 delete fix from HEAD

Fixed slapo-syncprov incomplete sync on restart issues (ITS#4622)
(synced syncprov.c with HEAD)

ITS#4589: code rewrite to use op->o_csn.  Bug may still occur.

ITS#4613 fix
more ITS#4622 fixes: syncprov: delete contextCSN on checkpoint if ctxcsn is empty

Import ITS#4562 fix from HEAD

+       Fixed slapd-shell (ITS#4495)
+       Added slapo-syncprov extra logging

From HEAD
+       Fixed slapd-bdb pre/post-read unavailable issue (ITS#4538)
+       Fixed slapo-refinit delete prohibit issue (ITS#4442)
+       Fixed slapo-syncprov/syncrepl sessionlog issue (ITS#4534)
+       Fixed test020-proxycache disabled debug issue (ITS#4491)

Suck in patches using Quanah's diffs
+       Fixed slapo-syncprov MODs cause DELs (ITS#4423)
+       Fixed slapd slap_send_ldap_result bug (ITS#4499)
+       Fixed slapd-ldif deadlock (ITS#4500)
+       Fixed liblutil strtoul(3) usage (ITS#4503)
+       Updated ldapsearch(1) BASE output (ITS#4504)
+       Fixed slapd cn=config (ITS#4512)
+       Fixed slapd thread pool init issue (ITS#4513)
+       Fixed slapd cn=config olcLimits (ITS#4515)
+       Fixed slapd runqueue use of freed memory (ITS#4517)

Additional changes - these suggested by Quanah

Update syncprov

From HEAD
+       Fixed slapo-syncprov/pache filter dup issue (ITS#4364)
+       Fixed slapo-syncprov playlog netries reset issue (ITS#4365)

Happy new year!

Changes from HEAD suggested by Howard (1st round, less tests)

Sync with HEAD

Sync with HEAD

Sync with HEAD

Sync with HEAD

Sync with HEAD

Sync with HEAD

Operation buffer align fix from HEAD

    Fixed slapd req_pwdexop bug
    Fixed slapo-syncprov queued UUIDs bug (ITS#4068)
    Fixed slapo-syncprov memory leak
    Fixed slapd anlist leak
    Removed lint
    Build Environment
        Updated testsuite to test only primary backends by default
        Disable test041-aci

Sync with HEAD

Sync with HEAD
Changes needs updating for release...

Sync with HEAD

Sync with HEAD

Sync with HEAD (need to update CHANGES)

Sync with HEAD

Sync with HEAD

Sync with HEAD

Sync with HEAD (ready for 2.3 "release"?)

Sync with HEAD in prep for 2.3.3beta

Initial round of changes for 2.3 beta

Sync with HEAD

Sync with HEAD

Sync with HEAD

Sync with HEAD