Fortress does not use Unbound's connection pool. Rather it uses a
hybrid approach combining the connection pool from the old mozilla
ldap library (ConnectionPool.java) + a pool manager class that was
created specifically for Fortress (PoolMgr.java).
The pool management code in place today was tested behind a load
balancer. It was designed to handle ldap server restarts or
switching between one ldap server ip to another based on various
events occurring (i.e. health). It was also designed to recover a
connection in the event a firewall or ldap server itself prematurely
terminates it. In these situations the pool &/or connection
should recover without requiring restart of process hosting fortress
ldap clients. Indeed requiring a process restart due to conditions
such as these would certainly be a critical defect in the fortress
library and would warrant serious attention by us. If this proves
to be the case we'll assist you in any way possible to get it
cleared up.
But I don't think this is happening based on the stack trace
presented.
Another possibility is the ldap client process might run out of
available connections because the number of pool connections isn't
set high enough to service all of the requests hitting it. Or,
network latency becomes a problem due to things happening on the
hosting server or the surrounding network. In any case your stack
trace indicates a null connection being returned from the connection
pool. This only happens when the ldap client wait exceeds 10
seconds for connection to return from the pool.
A couple things you can try:
1. bump up the max number of connections in the pool. This can be
done by increasing max in the fortress.properties:
# This is min/max settings for LDAP administrator pool which has
read/write access to all nodes under suffix:
min.admin.conn=1
max.admin.conn=10
Try doubling the max. If that doesn't work you can double again. I
have run with as many as 100 connections in the pool without
problems pertaining to resources. Keep in mind these default
settings are for testing and aren't optimized for production.
2. increase the wait time for connection pool before it returns null
connection, the default is 10 seconds.
add this to fortress.properties:
connTimeout=30
Which will allow the client to wait for 30 seconds. In actuality 30
seconds is FAR to long to wait for a connection but it may help us
figure out what is going wrong.
On 02/22/2014 02:47 PM, Shawn Smith
wrote:
We've been running into a semi-recurring problem with connection
pool corruption in the UnboundID stuff under Fortress. An example
of the stack dump is below. It looks like there's some stuff you
can do with UnboundID that might help in providing a graceful
recovery rather than us bouncing the app (which is the only fix we
have now). Before I dig too deep I was wondering if you had seen
this behavior and might already have a resolution. I'm willing to
put some time into fixing this if it's something new.
Thanks
The code:
public FortressUser getFortressUserByUserName(String userName) {
FortressUser userIn = new FortressUser();
userIn.setUserId(userName);
FortressUser userOut = null;
LOGGER.info("Retrieving user with userId: " + userName);
// Get the user
ReviewMgr reviewMgr;
try {
reviewMgr = ReviewMgrFactory.createInstance("");
userOut = new
FortressUser(reviewMgr.readUser(userIn)); //Line
that's blowing up
} catch (SecurityException e) {
LOGGER.error("The ReviewMgr could retrieve the requested
user");
}
// Output the user's data to the log
if(userOut != null) {
LOGGER.info(userOut.toString());
} else {
LOGGER.info("userOut is null");
}
return userOut;
}
The stack trace:
2014-02-20 10:53:09,850 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/fps-web].[FacesServlet]]
(ajp-/0.0.0.0:8009-506) JBWEB000236: Servlet.service() for servlet
FacesServlet threw exception: LDAPSDKUsageException(message='A
null object was provided where a non-null object is required
(non-null index 0). Thread stack trace:
getStackTrace(Thread.java:1568) /
ensureNotNull(Validator.java:119) /
connect(LDAPConnection.java:716) /
connect(LDAPConnection.java:681) /
connect(LDAPConnection.java:459) /
reconnect(LDAPConnection.java:626) /
checkConnection(PoolMgr.java:592) /
getConnection(PoolMgr.java:260) /
getAdminConnection(UnboundIdDataProvider.java:1192) /
getUser(UserDAO.java:747) / read(UserP.java:194) /
readUser(ReviewMgrImpl.java:263) /
getFortressUserByUserName(FortressServiceEjb.java:172) /
getRoles(FpsScimProvider.java:595) /
findUsers(FpsScimProvider.java:274) /
findPerson(PersonController.java:1079) /
loadAuthPerson(PersonController.java:947) /
loadAuthPerson(PersonController$Proxy$_$$_WeldClientProxy.java) /
invoke(null) / invoke(DelegatingMethodAccessorImpl.java:43) /
invoke(Method.java:606) / invokeMethod(BeanELResolver.java:735) /
invoke(BeanELResolver.java:467) /
invoke(CompositeELResolver.java:246) / getValue(AstValue.java:156)
/ getValue(ValueExpressionImpl.java:189) /
getValue(WeldValueExpression.java:50) / writeText(ELText.java:227)
/ writeText(ELText.java:150) / write(TextInstruction.java:85) /
encodeBegin(UIInstructions.java:82) / encodeAll(UILeaf.java:183) /
encodeAll(UIComponent.java:1782) /
encodeAll(UIComponent.java:1782) /
renderView(FaceletViewHandlingStrategy.java:439) /
renderView(MultiViewHandler.java:124) /
renderView(ViewHandlerWrapper.java:286) /
execute(RenderResponsePhase.java:120) / doPhase(Phase.java:101) /
render(LifecycleImpl.java:139) / service(FacesServlet.java:594) /
internalDoFilter(ApplicationFilterChain.java:295) /
doFilter(ApplicationFilterChain.java:214) /
invoke(StandardWrapperValve.java:230) /
invoke(StandardContextValve.java:149) /
event(JBossWebContext.java:91) / invoke(JBossWebContext.java:72) /
invoke(SecurityContextAssociationValve.java:169) /
invoke(StandardHostValve.java:145) /
invoke(ErrorReportValve.java:97) /
invoke(StandardEngineValve.java:102) /
service(CoyoteAdapter.java:336) / process(AjpProcessor.java:488) /
process(AjpProtocol.java:420) / run(JIoEndpoint.java:920) /
run(Thread.java:724)')
at
com.unboundid.util.Validator.ensureNotNull(Validator.java:119)
[unboundid-ldapsdk-2.3.3.jar:2.3.3]
at
com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:716)
[unboundid-ldapsdk-2.3.3.jar:2.3.3]
at
com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:681)
[unboundid-ldapsdk-2.3.3.jar:2.3.3]
at
com.unboundid.ldap.sdk.migrate.ldapjdk.LDAPConnection.connect(LDAPConnection.java:459)
[unboundid-ldapsdk-2.3.3.jar:2.3.3]
at
com.unboundid.ldap.sdk.migrate.ldapjdk.LDAPConnection.reconnect(LDAPConnection.java:626)
[unboundid-ldapsdk-2.3.3.jar:2.3.3]
at
us.jts.fortress.ldap.PoolMgr.checkConnection(PoolMgr.java:592)
[fortress-1.0-RC33.jar:]
at
us.jts.fortress.ldap.PoolMgr.getConnection(PoolMgr.java:260)
[fortress-1.0-RC33.jar:]
at
us.jts.fortress.ldap.UnboundIdDataProvider.getAdminConnection(UnboundIdDataProvider.java:1192)
[fortress-1.0-RC33.jar:]
at
us.jts.fortress.rbac.dao.unboundid.UserDAO.getUser(UserDAO.java:747)
[fortress-1.0-RC33.jar:]
at us.jts.fortress.rbac.UserP.read(UserP.java:194)
[fortress-1.0-RC33.jar:]
at
us.jts.fortress.rbac.ReviewMgrImpl.readUser(ReviewMgrImpl.java:263)
[fortress-1.0-RC33.jar:]
at
edu.psu.javaee.applications.fpsscim.ejbs.FortressServiceEjb.getFortressUserByUserName(FortressServiceEjb.java:172)
[classes:]
at
edu.psu.javaee.applications.fpsscim.FpsScimProvider.getRoles(FpsScimProvider.java:595)
[classes:]
--
Shawn Eion Smith
Software Engineering Manager/Research Engineer
Software Engineering Solutions
Applied Information Technology
Penn State University
shawn.eion.smith@psu.edu
814-867-3208
--
Shawn McKinney
shawn.mckinney@jts.us
|