Script:
Owner:
Subdir:
Blog ID: 75809171
Group ID: User ID:

Top Point Earners

 
WebSphere Product WebSphere Applicat.. CPU Starvation error on the application Server...
Select Forum or Category:
Post Reply
Page 1 of 2  •  1 2 Next
CPU Starvation error on the application Server after installing ITCAM data
4 years ago  ::  Aug 11, 2010 - 8:37AM #1
GWCForum
Posts: 849
CPU Starvation error on the application Server after installing ITCAM data

--------------------------------

Hi,
I have installed the ITCAM data colector on one of my application server and have also sets the below Generic JVM paprameters,

/" -server -Xbootclasspath/p:${ITCAM61HOME}/toolkit/lib/bcm-bootstrap.jar:${ITCAM61HOME}/itcamdc/lib/ppe.probe-bootstrap.jar -XX: NeverTenure -XX:MaxLiveObjectEvacuationRatio=70 -XX:SurvivorRatio=8 -XX:NewSize=4g -XX: UseParallelOldGC -XX: PrintGCDetails -XX: PrintGCTimeStamps -XX: PrintHeapAtGC -Djavax.management.builder.initial= -Dcom.sun.management.jmxremote -agentlib:am_sun_15=/opt/IBM/itcam/WebSphere/DC/runtime/was61.feapp31Node01.feappsrv31/ -XX:MaxPermSize=1024m -verbosegc -XX:-DisableExplicitGC -XX: UseParallelGC -XX:ParallelGCThreads=32 -Dsun.rmi.dgc.server.gcInterval=300000 -Dsun.rmi.dgc.client.gcInterval=300000 -Dsun.rmi.transport.tcp.connectionPool=true/"

When i restarted the application server, i get the below error messages under the SystemOut.log file

/"HMGR0152W: CPU Starvation detected. Current thread scheduling delay is
> 15 second
> HMGR0152W: CPU Starvation detected. Current thread scheduling delay is
> 28 second
> HMGR0152W: CPU Starvation detected. Current thread scheduling delay is
> 35 second
> HMGR0152W: CPU Starvation detected. Current thread scheduling delay is
> 12 second/"
The application server has a total 32 GB of memory and 1.4 GHZ(8 core) processors.
Also the platform is soalris 10 sparc based and the application server version is 6.1.0.23 where as the data collector version is 6.1.0.3 with FP5.
I have also set the initial heapsize to 8000(MB) and maximum heap Size to 22000(MB)
the hardware i am uaing is as follow:
Sun Blade T6320
8-core UltraSPARC T2 processor
Each core supports 8 threads
memory 32 GB
OS: Solaris 10 Sparc Based

Any help wil be greatly appreciated.

With Regards,

--------------------------------

Posted By: devil786 at Jul 31 2009  7:18AM
0 (0 Ratings)
4 years ago  ::  Aug 11, 2010 - 8:45AM #2
GWCForum
Posts: 849
HMGR0152W: CPU Starvation detected messages in SystemOut.log

--------------------------------

Cause :
The HMGR0152W message is an indication that JVM thread scheduling delays are occurring for this process.
The WebSphere® Application Server high availability manager component contains thread scheduling delay detection logic, that periodically schedules a thread to run and tracks whether the thread was dispatched and run as scheduled. By default a delay detection thread is scheduled to run every 30 seconds, and will log a HMGR0152W message if it is not run within 5 seconds of the expected schedule. The message will indicate the delay time or time differential between when the thread was expected to get the CPU, and when the thread actually got CPU cycles.

Example:
[10/25/05 16:42:27:635 EDT] 0000047a CoordinatorCo W HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 9 seconds.

The HMGR0152W message can occur even when plenty of CPU resource is available. There are a number of reasons why the scheduled thread may not have been able to get the CPU in a timely fashion. Some common causes include the following:
The physical memory is overcommitted and paging is occurring.
The heap size for the process is too small causing garbage collection to run too frequently and/or too long, blocking execution of other threads.
There may simply be too many threads running in the system, and too much load placed on the machine, which may be indicated by high CPU utilization.

 

Resolving the problem
The HMGR0152W message is attempting to warn the customer that a condition is occurring that MAY lead to instability if it is not corrected. Analysis should be performed to understand why the thread scheduling delays are occurring, and what action(s) should be taken. Some common solutions include the following:
Add more physical memory to prevent paging.
Tune the JVM memory (heap size) for optimal garbage collection.
Reduce overall system load to an acceptable value.

If the HMGR0152W messages do not occur very often and indicate that the thread scheduling delay is relatively short (e.g. < 20 seconds), it is likely that no other errors will occur and the message can safely be ignored.

The high availability manager thread scheduling delay detection is configurable by setting either of the following two custom properties.

The custom property IBM_CS_THREAD_SCHED_DETECT_PERIOD determines how often a delay detection thread is scheduled to run. The default value of this parameter is 30 (seconds).
The custom property IBM_CS_THREAD_SCHED_DETECT_ERROR determines how long of a delay should be tolerated before a warning message is logged. By default this value is 5 (seconds)

These properties are scoped to a core group and can be configured as follows:

In the administrative console, click Servers > Core groups > Core groups settings, and then select the -> {core group name}.
Under Additional Properties, click Custom properties > New
Enter the property name and desired value.
Save the changes.
Restart the server for these changes to take effect.

While it is possible to use the custom properties mentioned above to increase the thread-scheduling-detect-period until the HMGR0152W warning messages no longer occur, this is not recommended. The proper solution is to tune the system to eliminate the thread scheduling delays. 


Source : <www-01.ibm.com/support/docview.wss?uid=s...
0 (0 Ratings)
4 years ago  ::  Aug 11, 2010 - 8:55AM #3
GWCForum
Posts: 849
Hi AnandRaj,
  Thanks for the prompt and detail reply.
   
As you have told that /" There are a number of reasons why the scheduled thread may not have been able to get the CPU in a timely fashion. Some common causes include the following:
The physical memory is overcommitted and paging is occurring.
The heap size for the process is too small causing garbage collection to run too frequently and/or too long, blocking execution of other threads.
There may simply be too many threads running in the system, and too much load placed on the machine, which may be indicated by high CPU utilization/"
let me tell you that I have set 8 GB for initial heap and 22 GB for Maximum Heap,
Also the server have a total of 32 GB of memory.therefore in my opinion this is not due to memory.
According to your suggestion i want to tune  the system to eliminate the thread scheduling delays.
As i have already submit the server current JVM parameters. Can you help
me in this regard?

With regards,

--------------------------------

Posted By: devil786 at Jul 31 2009 10:47AM
0 (0 Ratings)
4 years ago  ::  Aug 11, 2010 - 8:55AM #4
GWCForum
Posts: 849
Hi

As per my understanding I suggest these tuning parameters.

1) use -XX:SurvivorRatio=16 (Best Practice)

2) -XX:ParallelGCThreads = 32
Reduces the number of garbage threads. On the Chip multithreading processor based system, set the threads no higher than one quarter of the hardware threads.
First try to set to 16 if you feel that better thats ok otherwise reduce 8 and then to 4

3)keep JIT enabled

4)-Xmn use a value which should be less than half of the overall heapsize

Friends Please correct me If I'm missing anything here.

Thanks
AnandRaj

--------------------------------

Posted By: a.pusarla at Aug  1 2009  3:21PM
0 (0 Ratings)
4 years ago  ::  Aug 11, 2010 - 9:02AM #5
GWCForum
Posts: 849
Thanks for the reply.
I will apply and test them and will `test`.`post` the results.


With regards,

--------------------------------

Posted By: devil786 at Aug  2 2009  3:15PM
0 (0 Ratings)
4 years ago  ::  Aug 11, 2010 - 9:03AM #6
GWCForum
Posts: 849
Ok thanks Guys.
I think i have got the problem solved by using the foloowing JVM parameters:

a) -XX: DisableExplicitGC
b) -XX: AggressiveOpts

Also i have set the -XX:ParallelGCThreads=16 (previous 32) and -XX:SurvivorRatio=16.

Till Now i have not get any CPU Starvation error under the SystemOut.log .
So Hope for the best.
In case of any problem i will bother you people again.

With regards,

--------------------------------

Posted By: devil786 at Aug 13 2009  6:40AM
0 (0 Ratings)
4 years ago  ::  Aug 11, 2010 - 9:03AM #7
GWCForum
Posts: 849
Can you please let me know how to set these properties.

--------------------------------

Posted By: chandoo.ws at Aug 21 2009 10:52AM
0 (0 Ratings)
4 years ago  ::  Aug 11, 2010 - 9:05AM #8
GWCForum
Posts: 849
you can add the parameters under the Websphere Application Server generic JVM parameters.

i.e.
1.Access the Java Virtual Machine page.
a.Click Servers > Application Servers in the console navigation tree.
b.On the Application Server page, click on the name of the server whose JVM settings you want to configure.
c.On the settings page for the selected application server, click Process Definition.
d.On the Process Definition page, click Java Virtual Machine.
2.On the Java Virtual Machine page, specify (post_id, thread_id, post_subject, post_timestamp, post_content) VALUES for the JVM settings as needed and click OK.
3.Click Save on the console taskbar.
4.Restart the application server.


if any help required please let me know.
with regards,

--------------------------------

Posted By: devil786 at Aug 25 2009  3:50AM
0 (0 Ratings)
4 years ago  ::  Aug 11, 2010 - 9:06AM #9
GWCForum
Posts: 849
Thanks a lot i will set it and then i will let us know.

--------------------------------

Posted By: chandoo.ws at Oct 20 2009  7:11AM
0 (0 Ratings)
4 years ago  ::  Aug 11, 2010 - 9:06AM #10
GWCForum
Posts: 849
Can you please explain the following

-XX:SurvivorRatio=16
-XX: DisableExplicitGC
-XX: AggressiveOpts
-Xmn

Please let us know if there is any resoucre by which i can learn such kind of JVM Properties and their usage.

Many Thanks.

--------------------------------

Posted By: chandoo.ws at Oct 20 2009  7:20AM
0 (0 Ratings)
Page 1 of 2  •  1 2 Next
    Viewing this thread :: 0 registered and 1 guest
    No registered users viewing

Connect with Others