Hi,
Below are the details :
High Level Architecture
Its 3-Tier architecture consists of Web Layer, Application Layer, Database Layer. However, the Web & Application layer are hosted on the same physical server.
The Web Layer consists of Apache 2.2.24 which will acts as a reverse-proxy. All requests are directed to Application layer.
The Application layer consists of CF 10 Enterprise version (10,0,13,287689 ) which integrated with Content Management Server (MURA 6.1) through MURA plugin.
The Database layer – Oracle 11g. Hosted on the dedicated physical server.
Recently, we upgraded the existing Apache-CF9 environment to Apache-CF10 (& MURA) to support our business needs. The website hosted on the CF10 environment experienced severe performance issues when launched to customers. The users experienced slowness ranging from 25 seconds & the CPU usage on the web-application server reached 95% consistent.
After initial investigation & tuning options, the CPU usage was reduced to 50% under 450 users load test. As part of stabilization, Java thread dumps were taken during load test at various levels of CPU usage. Upon analyzing the thread dumps, it was identified that the below threads were in BLOCKED state intermittently which directly affected the CPU to a higher %. Below is the stack trace of the BLOCKED threads.
"ajp-bio-8012-exec-9297" daemon prio=3 tid=0x0000000105d7e800 nid=0x3d27 waiting for monitor entry [0xfffffffd8fcef000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at macromedia.jdbc.sqlserverbase.ddv.c(Unknown Source)
at macromedia.jdbc.sqlserverbase.BaseDriver.acceptsURL(Unknown Source)
at java.sql.DriverManager.getDriver(DriverManager.java:253)
at coldfusion.sql.JdbcImpl.getConnection(JdbcImpl.java:79)
at coldfusion.sql.JdbcImpl.getConnection(JdbcImpl.java:59)
at coldfusion.sql.SqlImpl.execute(SqlImpl.java:363)
at coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:1059)
at coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:688)
at cfpluginManager2ecfc2058009211$funcDISPLAYOBJECT.runFunction(/webapp/ virtual/mura/requirements/mura/plugin/pluginManager.cfc:1863)
...................
"ajp-bio-8012-exec-9209" daemon prio=3 tid=0x00000001036e0800 nid=0x3ccf waiting for monitor entry [0xfffffffd9b8f4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at macromedia.jdbc.oraclebase.ddv.c(Unknown Source)
at macromedia.jdbc.oraclebase.BaseDriver.acceptsURL(Unknown Source)
at java.sql.DriverManager.getDriver(DriverManager.java:253)
at coldfusion.sql.JdbcImpl.getConnection(JdbcImpl.java:79)
at coldfusion.sql.JdbcImpl.getConnection(JdbcImpl.java:59)
at coldfusion.sql.SqlImpl.execute(SqlImpl.java:363)
at coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:1059)
at coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:688)
at cfpluginManager2ecfc2058009211$funcGETPLUGIN.runFunction(/webapp/virt ual/mura/requirements/mura/plugin/pluginManager.cfc:780)
"ajp-bio-8012-exec-9280" daemon prio=3 tid=0x00000001052a8800 nid=0x3d16 waiting for monitor entry [0xfffffffd920fb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at macromedia.jdbc.oraclebase.ddv.c(Unknown Source)
at macromedia.jdbc.oraclebase.BaseDriver.acceptsURL(Unknown Source)
at java.sql.DriverManager.getDriver(DriverManager.java:253)
at coldfusion.sql.JdbcImpl.getConnection(JdbcImpl.java:79)
at coldfusion.sql.JdbcImpl.getConnection(JdbcImpl.java:59)
at coldfusion.sql.SqlImpl.execute(SqlImpl.java:363)
at coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:1059)
at coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:688)
at cfPortcullis2ecfc1039057496$funcISBLOCKED.runFunction(/webapp/virtual /mura/requirements/mura/Portcullis.cfc:247)
Our initial investigation shows that these threads were trying to load the DriverManager once again for every requests which is not supposed to be the case.