[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Connection Pool Problems



Fortress does not use Unbound's connection pool.  Rather it uses a hybrid approach combining the connection pool from the old mozilla ldap library (ConnectionPool.java) + a pool manager class that was created specifically for Fortress (PoolMgr.java).

The pool management code in place today was tested behind a load balancer.  It was designed to handle ldap server restarts or switching between one ldap server ip to another based on various events occurring (i.e. health).  It was also designed to recover a connection in the event a firewall or ldap server itself prematurely terminates it.  In these situations the pool &/or connection should recover without requiring restart of process hosting fortress ldap clients.  Indeed requiring a process restart due to conditions such as these would certainly be a critical defect in the fortress library and would warrant serious attention by us.  If this proves to be the case we'll assist you in any way possible to get it cleared up.

But I don't think this is happening based on the stack trace presented.

Another possibility is the ldap client process might run out of available connections because the number of pool connections isn't set high enough to service all of the requests hitting it.  Or, network latency becomes a problem due to things happening on the hosting server or the surrounding network.  In any case your stack trace indicates a null connection being returned from the connection pool.  This only happens when the ldap client wait exceeds 10 seconds for connection to return from the pool.

A couple things you can try:

1. bump up the max number of connections in the pool.  This can be done by increasing max in the fortress.properties:

# This is min/max settings for LDAP administrator pool which has read/write access to all nodes under suffix:
min.admin.conn=1
max.admin.conn=10

Try doubling the max.  If that doesn't work you can double again.  I have run with as many as 100 connections in the pool without problems pertaining to resources.  Keep in mind these default settings are for testing and aren't optimized for production.

2. increase the wait time for connection pool before it returns null connection, the default is 10 seconds.

add this to fortress.properties:

connTimeout=30        

Which will allow the client to wait for 30 seconds.  In actuality 30 seconds is FAR to long to wait for a connection but it may help us figure out what is going wrong.

On 02/22/2014 02:47 PM, Shawn Smith wrote:
We've been running into a semi-recurring problem with connection pool corruption in the UnboundID stuff under Fortress.  An example of the stack dump is below.  It looks like there's some stuff you can do with UnboundID that might help in providing a graceful recovery rather than us bouncing the app (which is the only fix we have now).  Before I dig too deep I was wondering if you had seen this behavior and might already have a resolution.  I'm willing to put some time into fixing this if it's something new.

Thanks

The code:

 public FortressUser getFortressUserByUserName(String userName) {
    FortressUser userIn = new FortressUser();
    userIn.setUserId(userName);
    FortressUser userOut = null;
    LOGGER.info("Retrieving user with userId: " + userName);
   
    // Get the user
    ReviewMgr reviewMgr;
    try {
      reviewMgr = ReviewMgrFactory.createInstance("");
      userOut = new FortressUser(reviewMgr.readUser(userIn));  //Line that's blowing up
    } catch (SecurityException e) {
      LOGGER.error("The ReviewMgr could retrieve the requested user");
    }

    // Output the user's data to the log
    if(userOut != null) {
      LOGGER.info(userOut.toString());
    } else {
      LOGGER.info("userOut is null");
    }

    return userOut;
  }

The stack trace:

2014-02-20 10:53:09,850 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/fps-web].[FacesServlet]] (ajp-/0.0.0.0:8009-506) JBWEB000236: Servlet.service() for servlet FacesServlet threw exception: LDAPSDKUsageException(message='A null object was provided where a non-null object is required (non-null index 0).  Thread stack trace:  getStackTrace(Thread.java:1568) / ensureNotNull(Validator.java:119) / connect(LDAPConnection.java:716) / connect(LDAPConnection.java:681) / connect(LDAPConnection.java:459) / reconnect(LDAPConnection.java:626) / checkConnection(PoolMgr.java:592) / getConnection(PoolMgr.java:260) / getAdminConnection(UnboundIdDataProvider.java:1192) / getUser(UserDAO.java:747) / read(UserP.java:194) / readUser(ReviewMgrImpl.java:263) / getFortressUserByUserName(FortressServiceEjb.java:172) / getRoles(FpsScimProvider.java:595) / findUsers(FpsScimProvider.java:274) / findPerson(PersonController.java:1079) / loadAuthPerson(PersonController.java:947) / loadAuthPerson(PersonController$Proxy$_$$_WeldClientProxy.java) / invoke(null) / invoke(DelegatingMethodAccessorImpl.java:43) / invoke(Method.java:606) / invokeMethod(BeanELResolver.java:735) / invoke(BeanELResolver.java:467) / invoke(CompositeELResolver.java:246) / getValue(AstValue.java:156) / getValue(ValueExpressionImpl.java:189) / getValue(WeldValueExpression.java:50) / writeText(ELText.java:227) / writeText(ELText.java:150) / write(TextInstruction.java:85) / encodeBegin(UIInstructions.java:82) / encodeAll(UILeaf.java:183) / encodeAll(UIComponent.java:1782) / encodeAll(UIComponent.java:1782) / renderView(FaceletViewHandlingStrategy.java:439) / renderView(MultiViewHandler.java:124) / renderView(ViewHandlerWrapper.java:286) / execute(RenderResponsePhase.java:120) / doPhase(Phase.java:101) / render(LifecycleImpl.java:139) / service(FacesServlet.java:594) / internalDoFilter(ApplicationFilterChain.java:295) / doFilter(ApplicationFilterChain.java:214) / invoke(StandardWrapperValve.java:230) / invoke(StandardContextValve.java:149) / event(JBossWebContext.java:91) / invoke(JBossWebContext.java:72) / invoke(SecurityContextAssociationValve.java:169) / invoke(StandardHostValve.java:145) / invoke(ErrorReportValve.java:97) / invoke(StandardEngineValve.java:102) / service(CoyoteAdapter.java:336) / process(AjpProcessor.java:488) / process(AjpProtocol.java:420) / run(JIoEndpoint.java:920) / run(Thread.java:724)')
    at com.unboundid.util.Validator.ensureNotNull(Validator.java:119) [unboundid-ldapsdk-2.3.3.jar:2.3.3]
    at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:716) [unboundid-ldapsdk-2.3.3.jar:2.3.3]
    at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:681) [unboundid-ldapsdk-2.3.3.jar:2.3.3]
    at com.unboundid.ldap.sdk.migrate.ldapjdk.LDAPConnection.connect(LDAPConnection.java:459) [unboundid-ldapsdk-2.3.3.jar:2.3.3]
    at com.unboundid.ldap.sdk.migrate.ldapjdk.LDAPConnection.reconnect(LDAPConnection.java:626) [unboundid-ldapsdk-2.3.3.jar:2.3.3]
    at us.jts.fortress.ldap.PoolMgr.checkConnection(PoolMgr.java:592) [fortress-1.0-RC33.jar:]
    at us.jts.fortress.ldap.PoolMgr.getConnection(PoolMgr.java:260) [fortress-1.0-RC33.jar:]
    at us.jts.fortress.ldap.UnboundIdDataProvider.getAdminConnection(UnboundIdDataProvider.java:1192) [fortress-1.0-RC33.jar:]
    at us.jts.fortress.rbac.dao.unboundid.UserDAO.getUser(UserDAO.java:747) [fortress-1.0-RC33.jar:]
    at us.jts.fortress.rbac.UserP.read(UserP.java:194) [fortress-1.0-RC33.jar:]
    at us.jts.fortress.rbac.ReviewMgrImpl.readUser(ReviewMgrImpl.java:263) [fortress-1.0-RC33.jar:]
    at edu.psu.javaee.applications.fpsscim.ejbs.FortressServiceEjb.getFortressUserByUserName(FortressServiceEjb.java:172) [classes:]
    at edu.psu.javaee.applications.fpsscim.FpsScimProvider.getRoles(FpsScimProvider.java:595) [classes:]
-- 
Shawn Eion Smith
Software Engineering Manager/Research Engineer
Software Engineering Solutions
Applied Information Technology
Penn State University
shawn.eion.smith@psu.edu
814-867-3208

-- 
Shawn McKinney
shawn.mckinney@jts.us