Implications of using background-validation instead of validate-on-match for fault tolerance in JBoss EAP

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7
  • 6

Issue

• Which fault-tolerance validation method should be used?
• How do background-validation and validate-on-match compare in terms of reliability and performance?
• Does application code need to be changed if using background-validation instead of validate-on-match?

Resolution

Reliability Considerations

The validate-on-match setting tests each connection as it is handed out from the pool, but with background-validation there may be an extended period of time between background validation and the use of validated connections. Consequently, the background-validation mechanism is potentially less reliable because connections may fail between the periodic background validation and the time they are requested by/returned to the application for use. With background validation, the likelihood of getting a connection which has already failed depends on how frequently the validation runs. The problem is that more frequent background validation runs can often impose a higher penalty on the system than validate-on-match because background-validation will perform the validation operation/query for each connection in the pool that is not already reserved by the application for use (i.e. no validation will test connections that are already "checked out" of the pool for use by the application).

Performance Considerations

The question of which mechanism is more performant very much depends on the use of the system, network performance and the timing and scope of any connectivity issues. Consider the scenarios below:

• Users in systems that remain idle for long periods are more likely to see brief (or longer) delays when first requesting connections after an idle period when using validate-on-match compared to background-validation.
• Users of systems with well-performing networks using a more efficient validation mechanism (e.g. the JDBC 4 validation mechanism which relies on Connection.isValid(int) if supported by the driver) may notice few if any delays whether using validate-on-match or background-validation. This is especially true if the system is rarely idle and connections are less likely to time out.
• Following a wide-spread outage (impacting most or all of the connections in the pool), users of datasources configured with validate-on-match are more likely to encounter delays in getting connections (as the broken connections are iteratively validated and evicted while the user waits for a connection) whereas users of datasources configured with background-validation are more likely to encounter broken connections which will need to be returned, activity rolled back and retried (potentially multiple times). So, the performance difference may be negligible given that with validate-on-match the connection eventually retrieved (after some delay) should be working.

Coding for fault tolerance

If a connection is terminated by something outside the JVM after the last validation scan and that connection is provided to the application to satisfy a getConnection() request, the application will encounter an exception when it first uses the connection. The application is expected to catch the exception so that it can call Connection.close(). Calling close() will return the connection to the pool so that it can be evicted and destroyed. Failure to call close() will result in a leak of the connection which will (eventually) cause problems as the pool cannot create a new connection to replace the one that is leaked. After calling close() on the connection, the application should perform whatever cleanup it needs to do. If the application needs to do more work or start a new transaction, it must request a different connection from the pool.

In reality, the application logic should be the same whether using validate-on-match or background-validation because a connection could be externally terminated at any point (e.g. due to a network failure) even after it is obtained from the pool by the application. The difference between the two mechanisms is only the likelihood that a connection is already broken when it is obtained from the pool (i.e. it is more likely with background-validation since it would have just been tested if using validate-on-match).

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.