|
5 months ago ::
Dec 29, 2011 - 3:20PM
#1
|
Yasmin
WebSphere Support
IBM
|
Hi all I have this problem: I have a server that it suddenly hangs and it stopped responding requests. When I tried to stop the server with stopServer.sh, didn't work. Only shows "...has issued an order to stop the server..." but it freezes there.When I tried to kill the process with kill -9, it didn't work either 'cause it kills the process, but it restarts the process with a new pid. (why's that?) After that, when I ran stopServer again finally it works! what could it be? I generated a javacores and shows this: waiting threads:
| Servlet Dispatching |
com/ibm/ws/wswebcontainer/WebContainer.handleRequest() line 1433
|
| HTTP Client Communication |
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination() line 458
com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewInformation() line 387
|
| Async I/O Dispatching+ |
com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted() line 165
com/ibm/io/async/ResultHandler$2.run() line 881
|
| WebSphere Workers |
com/ibm/ws/util/ThreadPool$Worker.run() line 1497
|
Owning Stack
| Waiting on Condition Variable |
java/lang/Object.wait()native method
java/lang/Object.wait() line 199
|
| Socket Communication |
I com/ibm/ws/tcp/channel/impl/NBAccept.removePort()
line 169
I com/ibm/ws/tcp/channel/impl/TCPChannel.stop() line 413 |
| WebSphere Internals+ |
I com/ibm/ws/channel/framework/impl/ChannelFrameworkImpl.stopChainInternal() line
3030
I com/ibm/ws/runtime/component/ChannelFrameworkServiceImpl.stopChainsBy
AcceptorID() line 875 |
| Servlet Container |
I com/ibm/ws/wswebcontainer/WebContainer.
stopChains() line 854
I com/ibm/ws/webcontainer/WebContainerWorkloadRegulator.quiesceComplete()
line 155 |
| WebSphere Internals+ |
I com/ibm/ws/runtime/workloadcontroller/WorkloadController.quiesceCompleteWorkloads()
line 428
I com/ibm/ws/runtime/component/ServerCollaborator$ShutdownHook$1.run() line 687 |
| Security |
I com/ibm/ws/security/auth/ContextManagerImpl.runAs()
line 4157
I com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem() line 4254 |
| WebSphere Internals |
I com/ibm/ws/runtime/component/ServerCollaborator$ShutdownHook.run() line 677
I com/ibm/ws/runtime/component/ServerCollaborator$ShutdownHook.run() line 654 |
I don't understand what should I look.... Please help!!! :P
Moderated by
Celia Hamilton
on Jan 03, 2012 - 02:19PM
|
|
5 months ago ::
Dec 29, 2011 - 3:46PM
#2
|
Pankaj
Technical Associate
Tech Mahindra Ltd
|
Please check systemout.log and ffdc logs for that server and post it here.
|
|
5 months ago ::
Jan 02, 2012 - 3:43AM
#3
|
Christoph
IT-Systemprogrammierer
KKH-Allianz
|
you wrote:
When I tried to kill the process with kill -9, it didn't work either 'cause it kills the process, but it restarts the process with a new pid. (why's that?)
After that, when I ran stopServer again finally it works!
Node Agent detects the server failure/crash (kill -9) and then restarts the crashed server. This results in a new pid.
As the server restarted ("fresh instance" of the server), no threads are hung and therefore the server can be stopped normally via stopServer.
This behaviour can be configured in Application servers --> --> Java and process management --> Monitoring policy.
|
|
5 months ago ::
Jan 09, 2012 - 12:05AM
#4
|
Yasmin
WebSphere Support
IBM
|
Hi,
Thank you so much for your replies.
Christoph, good to know. ;)
I didn't know that node agent restarts automatically the crashed server, when is stopped with "kill-9".
Pankaj,
- In FFDC logs appears serveral times this exception:
java.lang.OutOfMemoryError com.ibm.ws.drs.message.DRSBuffWrapper.storeObject
mmm Is it related to Data Replication Service? Is it related with the information contained in javacore (see first post)? :S :S :S
- In the logs SystemOut and SystemError, messages appear only until the server failed. It did not print anything until I restarted the server. (I lost these logs because they were overwritten. :( )
Hope you can help me with something. ;)
Have a nice day!
|
|
5 months ago ::
Jan 09, 2012 - 1:54AM
#5
|
|
|
Hi Christoph,
You said,
This behaviour can be configured in Application servers --> --> Java and process management --> Monitoring policy.
I am a rookie with WAS, What kind of the behaviour as you mentioned? Could you show me? Thanks in advance.
Best wishes!
David
|
|
5 months ago ::
Jan 09, 2012 - 2:24PM
#6
|
Alexei
BPM Consultant
IBM Brazil
|
David, I believe he referred to the nodeagent restarting the server if there is a crash. Yasmin, you got some memory issue in your(s) application(s): memory is being depleted, thus you JVM crashes and become pretty much non-responsive. Look for heapdump files, usually sitting in the WAS profile where the server crashed, and try to analyze those with IBM HeapAnalyzer - java.sys-con.com/node/1229281- www-01.ibm.com/support/docview.wss?uid=s...That should provide you with a good starting point Cheers, Alexei
|
|
5 months ago ::
Jan 10, 2012 - 4:42PM
#7
|
Joseph
WebSphere Platform Technology Consultant
Consulting Ltd
|
Hi,
Thank you so much for your replies.
Christoph, good to know. ;)
I didn't know that node agent restarts automatically the crashed server, when is stopped with "kill-9".
Pankaj,
- In FFDC logs appears serveral times this exception:
java.lang.OutOfMemoryError com.ibm.ws.drs.message.DRSBuffWrapper.storeObject
mmm Is it related to Data Replication Service? Is it related with the information contained in javacore (see first post)? :S :S :S
- In the logs SystemOut and SystemError, messages appear only until the server failed. It did not print anything until I restarted the server. (I lost these logs because they were overwritten. :( )
Hope you can help me with something. ;)
Have a nice day!
use tivoli performance viewer to see the heap utilization under good amount of load.
Then if the JVM is operating near to the maximum heap... consider rising the max heap size.
If the heap utilization increased with time and never reduce... you need to suspect app and take help of heap analyzer to findout the issue.
|
