• Digital accessories
  • Server
  • Digital life
  • Privacy policy
  • Contact us
  1. Home
  2. Article
  3. WAS 6 PD

WAS 6 PD

Rsdaa 09/11/2021 5373

Collect and analyze the data

Find a resolution

ibm.com/redbooksInternational Technical Support Organization

WebSphere Application Server V6 Problem

Determination for Distributed Platforms

November 2005

SG24-6798-00 Note: Before using this information and the product it supports, read the information in "Notices" on page ix.

First Edition (November 2005)

This edition applies to WebSphere Application Server V6 on distributed platforms.

© Copyright International Business Machines Corporation 2005. All rights reserved.

Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADPSchedule Contract with IBM Corp.Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

Chapter 1. Approach to problem determination . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Introduction to problem determination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.1 Causes of problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.2 Types of problem symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Preparing for and preventing problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.1 Applying WebSphere maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.2 Checking the prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.3 Testing the application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.4 Setting up a test environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.5 Establishing safe operational procedures . . . . . . . . . . . . . . . . . . . . . 12 1.2.6 High availability and failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.2.7 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.2.8 System documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.2.9 Diagnostic data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.3 What to do when a problem occurs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.1 Revert to safe conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.2 Identify problem symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.3.3 Investigate and research the problem . . . . . . . . . . . . . . . . . . . . . . . . 27 1.3.4 Problem determination strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 1.4 Classify the problem and determine the root cause . . . . . . . . . . . . . . . . . 31 1.4.1 Installation or migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 1.4.2 Application packaging and deployment . . . . . . . . . . . . . . . . . . . . 35 1.4.3 System management and configuration . . . . . . . . . . . . . . . . . . . . . . 37 1.4.4 Runtime. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 1.5 Contacting IBM for support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 1.5.1 IBM support structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 1.5.2 Research the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 1.5.3 Collect MustGather files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 1.5.4 Determine the severity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 1.5.5 Create a PMR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

© Copyright IBM Corp. 2005. All rights reserved. iii

1.5.6 Send data to IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Chapter 2. Diagnostic data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

2.1 JVM logs (SystemOut and SystemErr) . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 2.2 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 2.3 Collector tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 2.4 First Failure Data Capture (FFDC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 2.5 Other logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 2.5.1 Process (native) logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 2.5.2 Service log (activity.log) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 2.5.3 Installation logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 2.6 IBM HTTP Server and plug-in logs and traces . . . . . . . . . . . . . . . . . . . . . 90 2.6.1 IBM HTTP Server logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 2.6.2 Web server plug-in logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 2.6.3 Web server plug-in trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 2.6.4 Network trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 2.7 System management logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.7.1 Output from wsadmin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.7.2 Management scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.7.3 Profile management logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.8 WebSphere Rapid Deployment logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 2.9 Summary of logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Chapter 3. Installation problem determination . . . . . . . . . . . . . . . . . . . . . . 97

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 3.2.1 Symptom: Launchpad or installation wizard will not start or fails . . 101 3.2.2 Symptom: Installation wizard hangs . . . . . . . . . . . . . . . . . . . . . . . . 103 3.2.3 Symptom: Profile creation failure . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.2.4 Symptom: IVT fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 3.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 3.3.1 Web browser requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 3.3.2 Application server startup problems . . . . . . . . . . . . . . . . . . . . . . . . 111 3.3.3 Profile creation problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 3.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Chapter 4. System management problem determination . . . . . . . . . . . . 115

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 4.1.1 Collecting data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 4.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 4.2.1 High-level symptom analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 4.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 4.3.1 Problem: Unable to access the administrative console. . . . . . . . . . 127 4.3.2 Problem: wsadmin or management scripts can't access server . . . 130

iv WebSphere Application Server V6 Problem Determination for Distributed Platforms

4.3.3 Problem: Unable to stop a server process . . . . . . . . . . . . . . . . . . . 133 4.3.4 Problem: Unable to start a server process . . . . . . . . . . . . . . . . . . . 135 4.3.5 Problem: Unable to access a node agent . . . . . . . . . . . . . . . . . . . . 137 4.3.6 Problem: Unable to manage a Web server . . . . . . . . . . . . . . . . . . . 138 4.3.7 Problem: Unable to manage applications . . . . . . . . . . . . . . . . . . . . 141 4.3.8 Problem: Failure adding a node to a deployment manager . . . . . . 143 4.3.9 Problem: Repository synchronization . . . . . . . . . . . . . . . . . . . . . . . 145 4.3.10 Problem: Save conflicts in the administrative console . . . . . . . . . 150 4.3.11 Problem: enterprise applications missing . . . . . . . . . . . . . . . . . . . 151 4.3.12 Problem: Invalid or expired certificates . . . . . . . . . . . . . . . . . . . . . 153 4.3.13 Problem: WebSphere Rapid Deployment . . . . . . . . . . . . . . . . . . . 1584.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

Chapter 5. Web container problem determination . . . . . . . . . . . . . . . . . . 165

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 5.1.1 Initial symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1685.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 5.2.1 Data to collect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 5.2.2 High-level symptom analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 5.2.3 Symptom: HTTP 404 error - The page cannot be displayed. . . . . . 172 5.2.4 Symptom: HTTP 404 error - Failed to find resource . . . . . . . . . . . . 175 5.2.5 Symptom: HTTP 404 error - WebGroup/virtual host not defined . . 177 5.2.6 Symptom: HTTP 500 error - JSP processing error . . . . . . . . . . . . . 179 5.2.7 Symptom: HTTP 500 error - IllegalStateException . . . . . . . . . . . . . 1815.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 5.3.1 Application URL specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 5.3.2 Static resources not displayed . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 5.3.3 Web resources not reloading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 5.3.4 Encoding and internationalization issues . . . . . . . . . . . . . . . . . . . . 195 5.3.5 HTTP session management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2025.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

Chapter 6. JCA connection problem determination. . . . . . . . . . . . . . . . . 211

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 6.1.1 JCA technical overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2136.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 6.2.1 Symptom: A JDBC call returns incorrect data . . . . . . . . . . . . . . . . . 217 6.2.2 Symptom: Failure to connect to a new data source . . . . . . . . . . . . 222 6.2.3 Symptom: Failure to connect to an existing data source . . . . . . . . 224 6.2.4 Symptom: Failure to access a resource through JDBC . . . . . . . . . 225 6.2.5 Symptom: Failure to access a non-relational resource . . . . . . . . . . 2266.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 6.3.1 Configuration problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

Chapter 7. Web server plug-in problem determination . . . . . . . . . . . . . . 251

7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 7.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 7.2.1 Collect the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 7.2.2 Analyze the high-level symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . 259 7.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 7.3.1 Problem: Web server will not start . . . . . . . . . . . . . . . . . . . . . . . . . 261 7.3.2 Problem: Failure between the Web server and plug-in . . . . . . . . . . 263 7.3.3 Problem: Sessions are being lost . . . . . . . . . . . . . . . . . . . . . . . . . . 272 7.3.4 Problem: The application works intermittently. . . . . . . . . . . . . . . . . 277 7.3.5 Problem: Application load is not being evenly distributed . . . . . . . . 280 7.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

Chapter 8. Application server crash problem determination . . . . . . . . . 285

8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 8.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 8.2.1 High-level symptom analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 8.2.2 Data to collect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 8.2.3 Analyze the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 8.2.4 Analyze the javacore file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 8.2.5 Finding a workaround for JIT problems. . . . . . . . . . . . . . . . . . . . . . 293 8.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 8.3.1 Stack overflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 8.3.2 Out of memory error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 8.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 8.4.1 Sun Solaris . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 8.4.2 HP-UX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300

Chapter 9. Default messaging provider problem determination. . . . . . . 301

9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 9.1.1 Problem categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 9.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 9.2.1 Collect the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 9.2.2 Analyze the high-level symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . 307 9.2.3 What to do if your symptom is not listed here . . . . . . . . . . . . . . . . . 310

vi WebSphere Application Server V6 Problem Determination for Distributed Platforms

9.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 9.3.1 Messaging engine startup problems . . . . . . . . . . . . . . . . . . . . . . . . 310 9.3.2 Message flow problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 9.3.3 Application configuration and resource problems . . . . . . . . . . . . . . 344 9.3.4 Product errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3499.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

Contents viiviii WebSphere Application Server V6 Problem Determination for Distributed PlatformsNotices

This information was developed for products and services offered in the U.S.A.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in anymanner serve as an endorsement of those Web sites. The materials at those Web sites are not part of thematerials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate withoutincurring any obligation to you.

© Copyright IBM Corp. 2005. All rights reserved. ix

TrademarksThe following terms are trademarks of the International Business Machines Corporation in the United States,other countries, or both:

Eserver® Domino® OS/390®

Eserver® DB2 Connect™ OS/400® Redbooks (logo) ™ DB2® Rational® developerWorks® Informix® Redbooks™ z/OS® IBM® SecureWay® AIX® MQSeries® Tivoli® ClearCase® Netfinity Manager™ WebSphere® Cloudscape™ Netfinity® CICS® OS/2®

The following terms are trademarks of other companies:

iPlanet, Enterprise JavaBeans, EJB, Java, Java Naming and Directory Interface, JavaBeans, JavaScript,JavaServer, JavaServer Pages, JDBC, JDK, JMX, JSP, JVM, J2EE, Solaris, Sun, and all Java-basedtrademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Windows, Win32, and the Windows logo are trademarks of Microsoft Corporation in the UnitedStates, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, and service names may be trademarks or service marks of others.

x WebSphere Application Server V6 Problem Determination for Distributed Platforms

This IBM® Redbook is intended to help customers through the process of

identifying and resolving problems in WebSphere Application Server V6 on distributed platforms. Previously published as a series of IBM Redpapers, each chapter addresses a specific aspect of the problem determination process. Links to the original papers are provided throughout the book.

The subsequent papers address selected topics that have been identified as some of the more common problem types that customers usually need to address.

The team that wrote this redbook

This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Raleigh Center.

Simon Davitt works in the WebSphere MQ Java™ Messaging level 3 service.

Previously he was an independent/free-lance programmer in the UK. He has over 20 years experience working in the IT field. Has been permanently employed by IBM since 2000 working in the WebSphere MQSeries® service area specializing in Java and JMS.

Rana Katikitala is an Advisory Software Specialist for Tivoli® Security in IBM

Software Labs, India. He has over seven years of experience in IT industry in the ares of development, support and test of operating systems, systems management software and e-business solutions. He holds a masters degree in

© Copyright IBM Corp. 2005. All rights reserved. xi

Structural Engineering from Regional Engineering College (REC) Warangal, India. His areas of expertise include IBM OS/2®, Windows® 2K, Netfinity® Manager™, IBM Director, Healthcare domain solutions of HIPAA(Health Insurance Portability and Accountability Act) and HCN(Healthcare Collaborative Network).

Thu-Giang Pham Thu-Giang Pham is an Advisory Software Engineer working

as a team leader for IBM WebSphere Application Server L2 support group in RTP, North Carolina.

Thu-Giang has more than seventeen years of experience in the IT industry

where she spent the last thirteen years providing technical support and consulting in the fields of databases, development tools and application servers. Thu-Giang also had four years of experience as a software engineer involved in the development of an application code generator and various financial applications. Before joining IBM in RTP five years ago, Thu-Giang worked for DataDirect Technology (formerly known as MERANT, Intersolv), Oracle and Software AG of Canada. Thu-Giang holds a Bachelor of Mathematics degree, joint honors in the fields of Computer Science, Combinatorics and Optimization from the University of Waterloo in Waterloo, Ontario, Canada.

Craig Scott is a Software Support specialist in Australia. He has 16 years of

experience in IT and over five years experience in WebSphere Application Server. He holds a degree in Computer Science from the University of Canberra. His areas of expertise includes WebSphere Application Server, WebSphere Edge Server, IBM DB2® and IBM Content Manager.

David Titzler is a Level 2 Support analyst for WebSphere Application Server

based in Raleigh, North Carolina. He joined IBM in 2001 and has been in his current position for four years. In this position, he works directly with clients to resolve WebSphere Application Server problems. He specializes in problems relating to connection pooling, messaging, EJBs, transactions, Web services, and more. He is an IBM Certified Advanced System Administrator for WebSphere Application Server V5. He holds a degree in Computer Science from the Pennsylvania State University.

xii WebSphere Application Server V6 Problem Determination for Distributed Platforms

helping customers to design, implement, and tune J2EE™ applications. For thepast four years, he has been working as project leader for a J2EE developmentframework based on J2EE patterns in the Clinical Engineering R&D Center,Argentina. He holds a degree in Information Technology Engineering from theU.T.N. Córdoba University, Argentina. His areas of expertise include J2EEarchitecture design, WebSphere Application Server consulting services andlegacy mainframe systems integration.

With special thanks to Ron Verbruggen for his guidance in designing anddeveloping the content of this book.

Thanks to the following people for their contributions to this project:

Gustavo BustosGrupo Leviminond

Thanks to the following IBMers:

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html

Comments welcome Your comments are important to us!

We want our Redbooks™ to be as helpful as possible. Send us your comments

about this or other Redbooks in one of the following ways: 򐂰 Use the online Contact us review redbook form found at: ibm.com/redbooks 򐂰 Send your comments in an email to: redbook@us.ibm.com 򐂰 Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HZ8 Building 662 P.O. Box 12195 Research Triangle Park, NC 27709-2195

xiv WebSphere Application Server V6 Problem Determination for Distributed Platforms

Chapter 1. Approach to problem

determination This paper introduces problem determination strategies for WebSphere Application Server V6. It discusses how to prevent problems, how to plan and prepare for problems that can occur, and what to do when a problem does occur so that it is resolved as quickly as possible. It then guides you to the more detailed information that can help you diagnose the cause of the type of problem that you are experiencing.

© Copyright IBM Corp. 2005. All rights reserved. 1

1.1 Introduction to problem determination Keeping your enterprise applications highly available to your customers is crucial in today's on demand business environment. WebSphere Application Server V6 has many new features and tools that are designed to minimize problem occurrences. However, if a problem does occur that might negatively impact your business, you need to be able to respond quickly and effectively.

2 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 IBM Education Assistant: WebSphere Application Server problem determination http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg27005460

1.1.1 Causes of problems

When a problem occurs, your first inclination might be to call IBM Support so that they can provide a fix that resolves the problem. In some cases, contacting IBM is necessary. However, the experience of the WebSphere Application Server Support team has shown that a small percentage of client-reported problems are actually due to defects with WebSphere Application Server code. Most problems are caused by configuration issues, environment issues, application code defects, or a misunderstanding of WebSphere Application Server. Many of these problems can be resolved easily without calling IBM Support to open a problem management record (PMR). In addition, many issues can be resolved by following the problem determination procedures that are discussed in this book.

1.1.2 Types of problem symptoms

When a user of an application that is running on WebSphere Application Server first notices a problem, a problem symptom is observed. Sometimes the problem symptom provides clues about the cause of the problem. Other times, a significant amount of problem determination is needed to determine the problem's root cause.

Chapter 1. Approach to problem determination 3

򐂰 An application does not respond to incoming requests. 򐂰 An application produces unexpected results (possibly errors or exceptions). 򐂰 An application cannot connect to an external system or resource. 򐂰 An application performs slowly or its performance degrades over time.

1.2 Preparing for and preventing problems

In any enterprise computing system, you can expect that some problems - large or small - will occur at times. In a best case scenario, the problems that you encounter will not be severe and will not result in critical business impact. However, it is a best practice to prepare and plan for the worst.

1.2.1 Applying WebSphere maintenance

Although many problems are caused by factors other than WebSphere Application Server code defects, the WebSphere Application Server Support team does find product defects when working through PMRs with clients. When a defect is found, the support team opens an authorized program analysis report (APAR). Each APAR has a unique identifier, a string that contains two letters (either PQ or PK) and five numbers. You can search for a particular APAR or a problem symptom reported in an APAR on the WebSphere Application Server Support site (Figure 1-1 on page 5). The site is available at: http://www-306.ibm.com/software/webservers/appserv/was/support

4 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Figure 1-1 Searching for an APAR on the WebSphere Application Server Support site

Figure 1-2 Changes in terminology for WebSphere maintenance between V5 and V6

Fix packs do not contain upgrades to the Java Software Development Kit (SDK).They are tested with the latest Java SDK service release, but the upgrades to the

Chapter 1. Approach to problem determination 5

Java SDK are delivered as separate fixes. You can also download the SDK fixes from the WebSphere Application Server Support site.

Proactively installing fix packs as soon as they become available is an effective

way to prevent problems from occurring. When you install a fix pack, you can be assured that you will not encounter any of the WebSphere Application Server code defects that are fixed in the fix pack. This saves the time and frustration of seeing one of these problems occur on your system.

For more information about the WebSphere Application Server V6 Update

Strategy, you can review the Update Strategy document, which is available on the Support site: http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21191989

6 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Figure 1-3 Checking the refresh pack and fix pack level from the administrative console

1.2.2 Checking the prerequisites

Another strategy for preventing problems is to ensure that all software and hardware in your environment meets the prerequisites of WebSphere Application Server V6. WebSphere Application Server has been tested with specific software and hardware configurations. It is known to work successfully in these configurations and to integrate well with the products with which it has been tested. You can find the software and hardware with which WebSphere Application Server has been tested and that it supports at: http://www-306.ibm.com/software/webservers/appserv/doc/latest/prereq.html

1.2.3 Testing the application

The best strategy to prevent problems from occurring when running WebSphere Application Server in production is thorough testing. You should develop a detailed testing strategy for your application and make sure that the strategy is followed every time that you install a new version of the application or that you upgrade WebSphere Application Server.

Chapter 1. Approach to problem determination 7

There are countless methods of software testing that you can employ, and we could fill an entire book discussing every method. Instead, we discuss the basic types of tests that every client should perform. In addition, we provide links to other documentation that might be useful in developing your testing strategy. Testing methods include the following: 򐂰 Unit testing Unit testing ensures that each method in each class of the application provides the expected output for all possible input. There are several unit testing frameworks that can be used to make unit testing easier. One popular framework is JUnit, which is open-source software. 򐂰 Functional testing Functional testing ensures that the application performs as the user expects it to. It tests the entire application to make sure that every component works together correctly. Functional testing is sometimes called integration testing. There are several functional testing frameworks available. JFunc is an extension to JUnit for functional testing. 򐂰 Performance testing Performance testing ensures that the application's performance is acceptable to users. WebSphere Application Server V6 includes an enhanced Tivoli Performance Viewer, which is accessible within the administrative console. In Version 5 and 5.1, the Tivoli Performance Viewer is run in a separate graphical user interface (GUI). To use the Tivoli Performance Viewer, enable the Performance Monitoring Infrastructure (PMI) metrics that you want to view in the administrative console. Log onto the administrative console, select your application server, and then select Performance Monitoring Infrastructure (PMI). PMI monitoring can be done in a production environment using the Basic set (default configuration) or the Extended set with minimal impact. After selecting a statistic set, you can then use the Tivoli Performance Viewer to monitor the current performance of the system, log the performance data, or view data that had been previously logged. You can access the Tivoli Performance Viewer by expanding Monitoring and Tuning in the administrative console, then expanding Performance Viewer, and selecting to view the current activity or previously logged performance data. The charts and graphics in the Tivoli Performance Viewer can give the administrator clues about where performance bottlenecks exist. They can then tune the appropriate WebSphere Application Server properties to relieve the problem. Another feature included in WebSphere Application Server V6 is the Tivoli Performance Advisor. The Advisor analyzes the performance data from your system and provides suggestions on which WebSphere configuration

8 WebSphere Application Server V6 Problem Determination for Distributed Platforms

properties to change to improve the performance. To enable the Advisor, select your application server in the administrative console and then select Runtime Performance Advisor Configuration. For more information about the Tivoli Performance Viewer and Advisor, review the following WebSphere Information Center sections: - Why use Tivoli Performance Viewer? http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/cprf_tpv.html - Monitoring performance with Tivoli Performance Viewer (TPV) http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/tprf_tpvmonitor.html - PMI data organization http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/rprf_datacounter6.html In addition to the Tivoli Performance Viewer and Advisor, there are several other tools, both open-source and proprietary, that are available to make performance testing easier. You can read an overview of the performance tools that are available in the technical article Comment lines from Ruth Willenborg: Selecting WebSphere performance tools, which is available at: http://www-128.ibm.com/developerworks/websphere/techjournal/ 0410_col_willenborg/0410_col_willenborg.html򐂰 Load and scalability testing Load testing involves testing your application with a simulated workload that corresponds to the amount of load that you expect your application to be able to handle in production. Scalability testing involves testing your application with increasingly higher amounts of load to determine if the application is scalable for future growth. There are several popular tools for simulating load during load and scalability testing. One such tool is Apache JMeter. You should coordinate your load and scalability testing with the tools that you use for performance testing so that you can tune WebSphere Application Server to provide better performance with higher levels of load.

We recommend using the WebSphere Application Server V6 Test Environment

within Rational® Application Developer, Version 6 to develop and test all of yourapplication code before installing it on your WebSphere Application Serverenvironment.

Chapter 1. Approach to problem determination 9

As you design your tests, consider the following: 򐂰 Your test scenarios should focus on the most used code path, but you should comprehensively test all possible code paths. 򐂰 Testing should be done with multiple users (not just the same one, over and over). 򐂰 Tests should be done with multiple functions in parallel. In production, several functions are executed at the same time. Some problems might only occur when the functions are run in parallel. It is important to test your application's functions together and not in isolation.

1.2.4 Setting up a test environment

We strongly recommend that you maintain a test environment that is configured exactly the same as your production environment. The WebSphere Application Server maintenance level (including refresh packs and fix packs), the versions of your applications, and your configuration should all be the same on both systems. There are many benefits to doing this: 򐂰 When you successfully test an application in your test environment, it gives you an accurate reflection of how the application will perform in production. 򐂰 When you need to make a change to an application or apply maintenance to WebSphere Application Server, you can fully test these changes in the test environment to be sure that there are no problems before you make any changes to your production environment. 򐂰 When a problem does occur in production, you can reproduce the problem in the test environment and perform problem determination in the test environment. This ensures that your users will not experience down time with the production environment. 򐂰 You can collect any diagnostic data that is needed to determine the root cause of a problem in the test environment. Because some diagnostic methods can impact performance or necessitate restarting one or more WebSphere Application Server processes, it is usually beneficial to collect the data in your test environment.

10 WebSphere Application Server V6 Problem Determination for Distributed Platforms

The key to this strategy is ensuring that the test environment and productionenvironment are configured exactly the same in every way. This includes:򐂰 The hardware and network configuration.򐂰 The operating system level and operating system patches.򐂰 The other software that is used in conjunction with WebSphere Application Server. This might include Web servers, databases, and messaging systems.򐂰 The WebSphere Application Server level, including refresh packs, fix packs, Java SDK fixes, and any individual APAR fixes that you might have downloaded or obtained from the WebSphere Application Server Support team.򐂰 The versions of all applications that are installed.򐂰 The WebSphere Application Server edition (Base, Network Deployment, or Express).򐂰 The WebSphere Application Server configuration. WebSphere Application Server V6 includes functionality to create configuration archive files with a .car extension. You can export the configuration from one machine to a .car file and then import that .car file to another system. Any configuration information that is specific to one system (for example, the host name) is removed in the configuration archive. It is a good practice to use configuration archive files to replicate the same WebSphere Application Server configuration on your test environment and production environment. Configuration archives are exported and imported with the wsadmin tool. To export the configuration of a WebSphere Application Server V6 profile or application server, use these wsadmin commands: $AdminTask exportWasprofile {-archive c:\myDirectory\myCell.car} $AdminTask exportServer {-archive c:\myDirectory\myServer.car -nodeName node1 -serverName server1} Use the target directory, node name, and server name that is appropriate for your system. To import the configuration of a profile or application server into a WebSphere Application Server V6 environment, use these commands: $AdminTask importWasprofile {-archive c:\myDirectory\myCell.car} $AdminTask importServer {-archive c:\myDirectory\myServer.car [-nodeInArchive node1][-serverInArchive server1][-nodeName node1][-serverName server1]} Again, use the target directory, node names, and server names that are appropriate for your system.

You can also automate changes to your test and production environments withscripts that run in wsadmin. Automation is advantageous if changes must be

Chapter 1. Approach to problem determination 11

done during non-peak hours (typically late nights or on weekends). The scripts can be scheduled to run at those times so that no one has to run the scripts manually during those hours. You might also want to run scripts to update the configuration of other software on your system at the same time. You can also use the scripts as part of your change log (see "Establishing safe operational procedures" on page 12).

Any time you make a change to either the test environment or the production environment, you should synchronize the two environments so that they remain identical.

It is a good idea to establish a baseline configuration, meaning a configuration

that has been tested successfully and found to be stable. When you make any type of change, you can test it. If the change is successful and does not cause any problems, you can then add the change to the baseline. If the change does cause a new problem, you can revert to the safe baseline configuration.

1.2.5 Establishing safe operational procedures

Another important facet of problem prevention and preparation is establishing a set of safe operational procedures for your organization. These procedures should outline the proper processes for making any types of changes to your test and production environments. The WebSphere Application Server Support team has found that many problems occur as a result of configuration or code changes made by one person in an organization of which other people in the organization were not aware. A strategy to eliminate these occurrences greatly reduces the chance of unexpected problems.

It is important to define security roles for people in your organization and to

ensure that only people who are authorized or part of the appropriate role can make changes to your configuration, upgrade your software, install applications, or do anything else that could potentially introduce problems. WebSphere Application Server V6 provides a comprehensive security infrastructure that you can use to define roles, authenticate, and authorize users. You should also take advantage of the security features of your operating system. This book does not discuss security. However, you can learn more about security in the security section of the WebSphere Information Center at: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/welc_concepts_csec.html

12 WebSphere Application Server V6 Problem Determination for Distributed Platforms

In addition to restricting the configuration of your environment, you should alsoimplement a change control system for your application code. There are severalchange control software products available. However, we recommend RationalClearCase®, which integrates seamlessly with Rational Application Developer.You can get more information about Rational ClearCase at its product Web site:http://www-306.ibm.com/software/awdtools/clearcase

A key to any strategy for configuration or code changes is documentation. It is

vitally important to document any and all changes that are made to yourproduction and test environments. We call this documentation a change log.Many problems surface after an application has been running successfully inproduction for a long time. When you carefully document all changes that weremade to your environment in one location, such as the change log, it is mucheasier to determine why a problem might have occurred at a certain time or date.Ensure that each person in your organization is aware of the process forupdating the change log when any change is made to the environment and thatthey follow the process. They should include the precise time and date of anychanges when they update the change log. Maintaining the change log andensuring strict compliance with this process can save many hours ofinvestigation and frustration when a problem occurs.

Establishing safe operational procedures is an essential part of the planning

process. A good set of procedures that are agreed upon and acted upon byeveryone in your organization can prevent problems from occurring and makeyou more prepared when problems do occur.

Chapter 1. Approach to problem determination 13

1.2.6 High availability and failover As discussed earlier, even the most sophisticated problem prevention techniques cannot guarantee that problems will never occur in your WebSphere Application Server V6 environment. Given this, it is logical to develop a contingency plan for when a WebSphere Application Server process or service becomes unavailable. Fortunately, new features in the area of high availability and failover are included with Version 6.

New in WebSphere Application Server V6 is the high availability manager feature

to provide failover for all applications and WebSphere Application Server services. It is configured automatically when you install WebSphere Application Server Network Deployment Version 6. The high availability manager runs critical WebSphere services (such as WLM and the transaction manager) on any available WebSphere Application Server processes.

To make the WebSphere Application Server transaction manager highly

available in Version 6, you must store the transaction logs (located by default in the /profiles//tranlog directory) on a network attached storage (NAS) system that is accessible to all of the WebSphere processes to which your transactions could failover (members of the core group). You also must select Enable high availability for persistent services for your cluster in the administrative console. When this is done, and a cluster member fails, its in-flight transactions are recovered on another cluster member.

A core group is a group of WebSphere Application Server processes

(deployment managers, node agents, application servers, and cluster members) within a cell that can participate in high availability together. That is, a service running on one member of the core group can failover to another member of the core group. By default, all processes in a cell are part of one core group (called DefaultCoreGroup). This is also the recommended scenario for most production environments, although it is possible to have multiple core groups within one cell.

14 WebSphere Application Server V6 Problem Determination for Distributed Platforms

A process can only be a member of one core group. System services, such as the WLM service, PMI, and the high availability coordinator itself, can failover to any member of the core group. Other services, such as the transaction manager and messaging engine, must failover another cluster member in the same cluster in which they were running when they failed. This is because the transaction manager and messaging engine need the same applications to be installed on the process that they failover to in order to recover.

The WLM service in Version 6 provides tload balancing functionality that is

comparable to Version 5 and 5.1. The addition of the high availability manager significantly enhances the failover capabilities in Version 6, making it no longer necessary to run third-party high availability tools with WebSphere Application Server.

Chapter 1. Approach to problem determination 15

and possibly dissatisfied with your company when these types of situations occur.

A good monitoring strategy can help you to identify problems before your customers experience them.

You notice several system messages in the SystemOut log. WebSphere

Application Server V6 includes a wide variety of system messages that are designed to provide you with information, warnings, and notifications of errors. You also find many system messages in WebSphere Application Server traces. Each system message has a unique message identifier that is nine characters in length and is in the form CCCC1234X. The first four characters (CCCC) indicate the WebSphere Application Server component that issued the message. The next four characters (1234) indicate the specific message that is being issued by the component. The last character (X) indicates the severity of the message. Its value is either I (informational), W (warning), or E (error).

Here are examples of the different types of system messages:

SECJ0231I: The Security component's FFDC Diagnostic Module com.ibm.ws.security.core.SecurityDM registered successfully: true. ADMN0001W: The service is unable to parse the MBean descriptor file com/ibm/ws/management/descriptor/xml/mbeans.xml. SRVE0068E: Could not invoke the service() method on servlet /com.ibm.ws.console.probdetermination/loggingSettingsGroups.jsp. Exception thrown : java.lang.NullPointerException

These system messages are useful for monitoring purposes. Informational

messages usually are not indicative of a problem. However, if an informational message is unexpected, it might alert you to an unusual occurrence that can result in a problem. Warnings and error messages are definitely signs that a problem has occurred.

16 WebSphere Application Server V6 Problem Determination for Distributed Platforms

The SystemErr log does not contain WebSphere system messages, but it doesshow exceptions that are thrown by WebSphere Application Server or by anapplication. It is a good idea to monitor the SystemErr log entries in addition tothe system messages in the SystemOut log.

By monitoring the SystemOut and SystemErr logs and adding your ownapplication logging, you can discover proactively a majority of the problems thatare covered in this book.

You can view errors, warnings, and informational messages directly in theadministrative console (Figure 1-4).

Chapter 1. Approach to problem determination 17

Figure 1-4 Viewing Runtime Events in the administrative console

You can select each message that you see to get more details about the message, the source of the message, and the reason why it occurred (Figure 1-5 on page 18).

Figure 1-5 Viewing message details in the administrative console

Another way to monitor your application is by using the Tivoli Performance

Viewer, discussed in "Testing the application" on page 7. You can observe

18 WebSphere Application Server V6 Problem Determination for Distributed Platforms

performance metrics, such as average response time, number of requests, thread pool sizes, connection pool sizes, JVM memory, CPU, I/O, and system paging, to monitor the health of WebSphere Application Server and your applications. You can log the performance data from a time when everything is running normally and then compare that data to the current performance data to see if there are major discrepancies. Unexpected discrepancies are an indication of a problem. When you see signs that a problem might be occurring, you can perform the specific problem diagnostic steps that are described within this book.

A well-planned monitoring strategy, especially when combined with a good high

availability and failover strategy, should enable you to diagnose and resolve problems that occur, possibly before your users and customers know about them.

1.2.8 System documentation

In the event that a problem occurs in your environment, it is possible that you will need to enlist the help of other people, either internal or external to your organization, to determine the root cause of the problem. When that happens, you will want everyone involved to understand thoroughly the details of the systems that are involved in your environment.

To this end, it is important to document the details of your configuration. In

"Establishing safe operational procedures" on page 12, we discussed documenting all of the changes that have been made to your environment. In addition to that, you should maintain a high-level description of your basic topology. We call this system documentation. System document is useful in the following circumstances: 򐂰 A problem occurs and you need to get assistance from others who might not be as familiar with your application and topology as you are. The system documentation allows you to bring them up to speed as quickly as possible. 򐂰 A problem occurs and you want to identify from which parts of your environment you should collect diagnostic data or monitor. Your system documentation shows the software components that are involved and the flow of your application, that is how different software components are used when your application processes a request.

Chapter 1. Approach to problem determination 19

Your system documentation should consist of written documents and diagrams. Which information is included in the written documents and which is included in diagrams is a matter of preference. Overall, the information should be detailed and should show the specific versions and maintenance levels of the operating system and all software products involved, the hardware and network configuration, and specific host names and IP addresses of the systems that are involved.

A common and important component of system documentation is the topology

diagram. It gives a quick overview of your system topology and application flow. Figure 1-6 on page 20 illustrates an example.

host5.mycompany.com host2.mycompany.com 1.12.34.60 1.12.34.57

Web

Firewall Server Cluster MQ Member 2

Figure 1-6 Example of a topology diagram

In this scenario, the accompanying written documentation should include the

specific software and hardware levels that are involved.

20 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Detailed system documentation is an integral part of your problem planning strategy that should not be overlooked.

1.2.9 Diagnostic data collection

Finally, to prepare for a problem occurrence, you should plan what diagnostic data to collect for various problem scenarios. In "Types of problem symptoms" on page 3, we discussed several broad categories of problem symptoms. In "What to do when a problem occurs" on page 24 and other papers in this series, we elaborate on how to determine the cause of a problem for each of the problem symptoms. We also include information about what data to collect for different types of problems. It is a good idea to identify the most common problems that have occurred in the past in your environment and those that you believe might occur the most in the future. Then, you can form a diagnostic data collection plan so that you are prepared to collect the necessary data if a problem does occur.

Chapter 1. Approach to problem determination 21

several times to collect the necessary data. To avoid this situation, it is recommended that you have between 2 GB and 5 GB of extra disk space available on each system. 򐂰 After you resolve a problem, either delete or archive the diagnostic data that you collected for the problem. This will prevent the old diagnostic data from being confused with the new diagnostic data the next time that a problem occurs. 򐂰 Configure the thread monitor for hang detection. WebSphere Application Server V6 includes a thread monitor feature. This is also included in Version 5.1.1. The thread monitor is notified when the Web container, ORB, or asynchronous bean thread pools give work to a thread. By default, the thread monitor checks the status of all active threads every three minutes. If it finds a thread that has been active for more than ten minutes, it outputs a warning to the SystemOut log, similar to the following: WSVR0605W: Thread threadname has been active for hangtime and may be hung. There are totalthreads threads in total in the server that may be hung. The thread monitor makes it easier to determine that a problem has occurred. If you see a WSVR0605W warning in your SystemOut log, you out know that a thread has stopped responding. You can then perform further diagnostic steps to determine the cause of the hung thread. The thread monitor does not take any action to fix the problem beyond notifying you of the problem. In preparing for diagnostic data collection, you might want to change the default thread monitor behavior. You can change the interval that the thread monitor checks the status of threads (this is three minutes by default) and the amount of time that a thread can be active before it is reported by the thread monitor (this is ten minutes by default). To alter these properties: a. Log onto the administrative console. b. Select your application server. c. Under Server Infrastructure, select Administration. d. Select Custom Properties and then click New. e. Create these properties and provide the desired values for them: * com.ibm.websphere.threadmonitor.interval (the interval that the thread monitor checks the status of threads) * com.ibm.websphere.threadmonitor.threshold (the amount of time that a thread can be active before it is reported by the thread monitor) For more information about the thread monitor, see the following sections in the WebSphere Information Center:

22 WebSphere Application Server V6 Problem Determination for Distributed Platforms

- Detecting hung threads in J2EE applications http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/ctrb_hangdetection.html - Configuring the hang detection policy http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/ttrb_confighangdet.html򐂰 Consider enabling verbose garbage collection on each application server. The performance impact of enabling verbose garbage collection is minimal, and the data is often useful when performance problems occur. To enable verbose garbage collection: a. Log onto the administrative console. b. Select your application server. c. Under Server Infrastructure, expand Java and Process Management, and then select Process Definition. d. On the resulting screen, select Java Virtual Machine under Additional Properties. e. Select Verbose garbage collection. When verbose garbage collection is enabled, the output appears in the native_stderr.log file for your application server.򐂰 Become familiar with the WebSphere Application Server collector tool. The collector tool is run as an executable in the /bin directory. It produces a Java archive (jar) file that contains all of the logs and XML configuration files from your system. The resulting jar file is very useful to the WebSphere Application Server Support team and any others who are involved in the problem determination process. It allows them to view quickly your WebSphere Application Server configuration and to see any errors or exceptions that have occurred. You can find more information about the collector tool in WebSphere Application Server V6: Diagnostic Data at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4085.pdf

The WebSphere Application Server Support team has compiled a

comprehensive list of MustGather documents for different types of WebSphereApplication Server problems. For more information, see IBM - MustGather: Readfirst for all WebSphere Application Server products at:http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21145599

The files listed in the MustGather documents are useful in determining the causeof your problem. You can use these documents to as part of your diagnostic datacollection plan.

Chapter 1. Approach to problem determination 23

Consider all the types of problems that you might be likely encounter in your environment while reading this book. Based on the types of problems that have occurred in your environment in the past, you might be able to predict what types of problems can occur in the future. Take note of the diagnostic data that is needed to determine the root cause of each problem type. Make sure that this is documented so that everyone in your organization knows what data to collect. This will make the problem determination process simpler and less stressful.

1.3 What to do when a problem occurs

You have prepared, and you have planned. Your application is now running in your production environment, and it is being used successfully by your customers. Just as you are ready to celebrate, you get a call that a problem has occurred. What do you do now?

1.3.1 Revert to safe conditions

When any problem occurs, your first action should be to consider the business impact of the problem. Depending on the business impact, it might be necessary to take steps to limit the business impact before beginning your problem determination efforts. If the problem occurred in your test environment, the business impact is relatively low when compared to the potential impact of a production outage. In that case, making an effort to work around the problem while looking for a permanent solution is probably not necessary. You can use the test environment to execute all of the necessary problem determination steps.

However, if the problem occurred in production, you will probably want to

consider how to quickly alleviate the problem symptoms so that customers and users will experience the least possible negative effects. In this section, we refer to this process as reverting to safe conditions. You will want to do this in parallel with your problem determination efforts.

Techniques for reverting to safe conditions include:

򐂰 Making your test environment or another similar environment where the problem is not occurring your temporary production environment. Configure

24 WebSphere Application Server V6 Problem Determination for Distributed Platforms

your systems so that incoming requests are processed by an environment where the problem does not occur. 򐂰 Installing an older version of application code where the problem does not occur. If the problem started to occur after an application code change was introduced, it might be a good idea to go back to an earlier working version of the application. 򐂰 Changing any recent configuration changes back to your baseline configuration. As discussed in "Setting up a test environment" on page 10, the baseline configuration should be a configuration that is fully tested and is known to be stable. 򐂰 Removing any WebSphere Application Server maintenance that was recently installed before the problem occurred. This would temporarily resolve the problem if it is caused by a WebSphere Application Server code defect introduced by the latest maintenance package. 򐂰 Making the application function that produced the problem inaccessible to customers and users. You can post a notification on your Web site that the function is temporarily unavailable or under maintenance. You can provide an estimated time for when it will be available again.

1.3.2 Identify problem symptoms

When you are first notified about the problem, you might only receive a vague, non-specific set of problem symptoms. You might be told that users cannot access your application at all or possibly that a specific action taken by your users is resulting in an error message.

Chapter 1. Approach to problem determination 25

򐂰 How do you know when this particular problem occurs? Is there a something specific to watch for in order to recognize if the problem occurs again? 򐂰 How would you know that the problem was resolved? Would it be that an error message no longer occurred? Would the application behave differently? What specifically would confirm a problem resolution? 򐂰 Where did the problem occur? Did the problem occur only in your test environment, only in your production environment, or on both? Did it only occur on one system in your environment? Did it occur on multiple systems? Did it occur on every cluster member or only one? 򐂰 When did the problem occur? What was the time stamp of the error or unexpected behavior? Did the problem occur only once or many times? How often did the problem occur? Did it occur at certain intervals, or did it seem to occur at random times? Was there some event that occurred that might have triggered the problem? For example, did the user attempt a specific application function? 򐂰 Why might the problem have occurred? You might not be able to answer this right away. Was this the first time something specific was tried? Was there a recent change to the application code or the configuration of your environment? Does it happen in all of your environments or only one? If it only happens in one environment, how is this environment different from the others? 򐂰 Has the diagnostic data that is identified in your diagnostic data collection plan been collected? Does the data provide any other details about the problem or offer immediate clues as to why the problem occurred?

26 WebSphere Application Server V6 Problem Determination for Distributed Platforms

On the other hand, the answer might require more investigation. If this is the case, you have developed a solid and complete problem description to use as the basis for more problem determination efforts.

1.3.3 Investigate and research the problem

While investigating the problem, you should form a list of all of the problem symptoms that occurred within your problem log. Sometimes, there is only one symptom. If this is the case, your job is easier. However, often several symptoms occur, and it can be difficult to determine which symptoms characterize the problem and which symptoms are simply the result of the problem. It is a good idea to organize the symptoms into a time line. You can also include details about what was happening in your environment in your time line. For example, your time line might look similar to the following: 10:00 - Peak workload reached 10:07:53 - ConnectionWaitTimeoutException in SystemOut.log for server1 10:08:24 - ConnectionWaitTimeoutException in SystemOut.log for server1 10:14:09 - ConnectionWaitTimeoutException in SystemOut.log for server2 10:20:46 - Users cannot log onto Web site, application is not responding

Because the ConnectionWaitTimeoutException was the first symptom that was

observed, it is a good starting point for your investigation. However, it is possible that your investigation might reveal that a symptom is actually a different problem that is unrelated to the problem that you are investigating.

WebSphere system messages are useful in your problem determination efforts.

Informational messages and warnings provide context. They show you what was occurring immediately before or immediately after a problem. Error messages are significant problem symptoms. The message identifier for an error message is useful for entering into search engines.

When you choose a symptom to investigate, you can begin researching it. A good place to start is your organization's internal documentation. It is possible

Chapter 1. Approach to problem determination 27

that someone else has already seen the same problem or a similar problem in your environment. If they have, it is likely that the root cause of the problem is the same. If this is the first time that a problem similar to this has been seen in your organization, the next place to research is online documentation that is available from IBM. We recommend using the following resources to research your symptoms: 򐂰 WebSphere Information Center http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp The WebSphere Information Center includes detailed information about the features and configuration of the product. It also includes descriptions of every WebSphere Application Server system message identifier, and it contains a troubleshooting guide that might aid your problem determination efforts: http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/welc_concepts_ctrb.html 򐂰 WebSphere Application Server Support site http://www-306.ibm.com/software/webservers/appserv/was/support You can search the support site for APARs and technotes that might be related to your problem symptom. As discussed in "Applying WebSphere maintenance" on page 4, APARs are reports of known WebSphere Application Server code defects. If an APAR description matches your problem symptom, you should install the WebSphere Application Server fix pack that contains the fix for the APAR. The APAR description explains which fix pack includes the fix. Technotes are documents that are created and maintained by the WebSphere Application Server Support team and knowledge engineering team. They document known problems and the solutions to those problems. Technotes are usually created for problems that are solved by a configuration change rather than a code change. 򐂰 WebSphere developerWorks® http://www-130.ibm.com/developerworks/websphere The developerWorks site contains many articles written by IBM developers and other technical staff members. Many articles discuss application programming best practices and provide tips for avoiding and resolving problems with WebSphere Application Server. 򐂰 IBM Support Assistant http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21192593 The IBM Support Assistant is a downloadable tool that is used to simplify the problem determination process for many IBM software products. There are several product plug-ins for the Support Assistant, including one for WebSphere Application Server V6. It includes a federated search interface

28 WebSphere Application Server V6 Problem Determination for Distributed Platforms

that allows you to search for your problem symptoms at multiple IBM Web sites simultaneously. It also provides easy access to product education modules, including the IBM Education Assistant. If you need to open a PMR with IBM Support, the tool collects pertinent information that is needed by the support team. You can even open a PMR within the tool.

This book also provides valuable information as you research your problem.

1.3.4 Problem determination strategies

At this point, you have identified specific and detailed problem symptoms, reverted to safe conditions so that you have minimized the business impact of the problem, and begun your investigation of the problem. The next step is to start determining the root cause of the problem, which is the main focus of this book.

For any problem that occurs, there are two major strategies that you can use to determine the root cause: the analysis strategy and the isolation strategy.

The analysis strategy involves analyzing the diagnostic data, possibly through several iterations, until the cause of the problem is found, as illustrated in Figure 1-7 on page 30. To be successful with this strategy, you must have a good understanding of the diagnostic data. This is the strategy used most often by the WebSphere Application Server Support team. There are many diagnostic tools that are available to help you. For example, the ThreadAnalyzer helps you to analyze Java thread dumps, and the HeapRoots tool is useful in analyzing out of memory issues. You can review and download the most current diagnostic tools at: 򐂰 WebSphere Application Server Support site http://www-306.ibm.com/software/webservers/appserv/was/support 򐂰 WebSphere developerWorks http://www-130.ibm.com/developerworks/websphere

Chapter 1. Approach to problem determination 29

Analyze data

Collect diagnostic data

Form conclusions Find root cause

Figure 1-7 The analysis strategy

The isolation strategy is often employed when there is an error or exception. In

the SystemOut or SystemErr log for the application server, you see the Java stack trace of the exception. The stack trace is read from the bottom to the top. At the bottom, you see the thread that is allocated by the thread pool. Reading the stack trace upwards, you see every method that is called (including WebSphere Application Server code, application code, and possibly other third-party utilities) until the exception is thrown. You can use the stack trace to identify the different variables in the problem scenario and then remove them.

It might also help to insert print statements into the application to print debugging information as the application executes.

Similar to the analysis strategy, the isolation strategy is an iterative process. You keep removing variables until you have isolated the variable that is causing the problem.

30 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Analyze results

Reproduce the problem

Remove variables Find root cause

Figure 1-8 The isolation strategy

Now that we have discussed the general strategies for problem determination, we focus on the specific strategies for each problem classification.

1.4 Classify the problem and determine the root cause

The next step in the problem determination process, and a very important one, is classifying the problem. This book refers to many different classifications of problems. You need to identify which classification the problem fits into.

Chapter 1. Approach to problem determination 31

performance monitoring. See "System management and configuration" on page 37 for more information. 򐂰 An application or WebSphere Application server process (for example, an application server, node agent, or deployment manager) is unable to start. When running the startServer, startNode, or startManager commands to start a WebSphere Application Server process or when using the administrative console or wsadmin to start a process or an application, a problem occurs that causes the process or application to fail to start. See "Application or WebSphere process is unable to start" on page 42 for more information. 򐂰 An application does not respond to incoming requests. This could occur due to a Web server, Edge component, or Web server plug-in problem, application server crash, hang, out of memory condition, or 100% CPU utilization condition. See "Application does not respond to incoming requests" on page 43 for more information about determining the root cause of the problem. 򐂰 An application produces unexpected results (possibly errors or exceptions). An error is seen or an exception is thrown when certain application code is executed or when certain conditions (for example, heavy load) are met. The application could also behave differently than expected, but produce no error or exception. There are several components of WebSphere Application Server in which this type of problem could occur. See "Unexpected results from an application" on page 49 for more information. 򐂰 An application cannot connect to an external system or resource. The application might need to access an external system or resource. This could be a database, a messaging system, an enterprise information system accessed through the Java Connector Architecture (JCA), an Enterprise JavaBeans™ (EJB™) running on a remote system, or a Web service. There might be a problem establishing a connection to the external system or resource, or an error might occur when the application interacts with it. See "Application cannot connect to an external system or resource" on page 63 for more information. 򐂰 An application performs slowly or its performance degrades over time. Although performance problems fall outside the scope of this book, we do provide some external resources for performance problems. In general, performance problems can be corrected by tuning WebSphere Application Server, the other software products with which it interacts, and the operating system. Thorough performance testing and tuning should be completed before an application is put into production. See "Application is slow or its performance degrades over time" on page 69 for more information.

32 WebSphere Application Server V6 Problem Determination for Distributed Platforms

The WebSphere Application Server Support team has found that the majority ofproblems encountered by clients fit into one of these problem classifications. Inthis book, we provide comprehensive information about problem determinationstrategies for some of these common problem areas. For the problemclassifications that are not covered in this book, we provide links to externalresources that discuss problem determination for each problem type.

If you have already examined the WebSphere logs and have determined that aparticular error message holds the key to the problem, use Table 1-1 as a quickreference to find information based on the message prefix.

Table 1-1 Error message prefix to topic mapping

Message prefixes Topic

INST, ADMU (during profile creation) or "Installation" on page 34

WSVRT (during IVT)

MIGR "Migration" on page 35

ADMA "Application packaging and deployment"

SECJ, SECG, JSAS, JSSL, WSEC, or WSSK "Security" on page 39

PLGN, PLGC, and PLPR "Web server plug-in" on page 45

SRVE (Web container), JSPG (JSPs), or "Web container" on page 50

CNTR, PMGR, and ACIN "EJB container" on page 51

SESN "Session management" on page 52

DYNA "Dynamic cache" on page 54

WTRN, WLTC "Transaction manager" on page 55

WWLM "Workload management" on page 56

HMGR, CWRCB, CWWCW "High availability manager" on page 57

CWWDR "Data replication service" on page 58

WACS, APPR, ASYN, OBPL, SCHD, STUP, "Program model extensions" on page 59 ACWA

Chapter 1. Approach to problem determination 33

Message prefixes Topic

I18N, LTXT "Internationalization/Double byte

character set" on page 61

CHFW, DCSV, HTPC, SSLC, TCPC, WSSC, "Transport channel service" on page 62 and XMEM

J2CA, WSCL (WebSphere client) "JCA connection manager" on page 63

DSRA or CONM "Database connections" on page 64

CWSIA, CWSIB, CWSIC, CWSID, CWSIE, "Messaging" on page 64

CWSIF, CWSIH, CWSII, CWSIJ, CWSIK, CWSIL, CWSIM, CWSIN, CWSIO, CWSIP, CWSIQ, CWSIR, CWSIS, CWSIT, CWSIU, CWSIV, CWSIW, CWSIX, CWSIY, CWSIZ, CWSJA, CWSJB, CWSJC, CWSJD, CWSJO, CWSJQ, CWSJR, CWSJU, CWSJW, CWSWS, WMSG

NMSV "JNDI naming" on page 65

CORBA COMM_FAILURE "ORB" on page 66

WSWS, SOAP, WSIF, CWWSG "Web services" on page 67

PMON "PMI and Tivoli Performance Viewer" on

1.4.1 Installation or migration

This section addresses problems related to the installation of WebSphere Application Server or the migration from previous releases.

34 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Note: For problem determination strategies for installation problems, see WebSphere Application Server V6: Installation Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4068.pdf

1.4.2 Application packaging and deployment

Symptom: You are having problems deploying an application to WebSphere Application Server using the administrative console or wsadmin. You might receive WebSphere system messages that begin with ADMA.

For problem determination strategies for application installation and deployment

problems, review the following resources: 򐂰 WebSphere Information Center: Developing and deploying applications http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/welc6topdeveloping.html

Chapter 1. Approach to problem determination 35

򐂰 WebSphere Information Center: Troubleshooting deployment http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/ttrb_deploy.html 򐂰 WebSphere Information Center: Explanation of ADMA system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ADMA.html 򐂰 MustGather: Problems during Deployment of EAR/JAR/WAR files http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199344 򐂰 MustGather: Enhanced EAR file problems for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199181

WebSphere Rapid Deployment

Symptom: You are trying to use WebSphere Rapid Deployment to develop and are trying to test an application. You cannot connect to an application server or WebSphere Rapid Deployment does not create or update the applications.

Note: For issues with WebSphere Rapid Deployment, see WebSphere

Application Server V6: System Management Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4067.pdf

Application Server Toolkit

Symptom: You have a problem when using the Application Server Toolkit (AST) to assemble your applications. This encompasses all possible problems with the AST, including starting it, any problems that you experience with application assembly, and any errors that occur.

36 WebSphere Application Server V6 Problem Determination for Distributed Platforms

1.4.3 System management and configuration Symptom: You experience difficulties in WebSphere Application Server system management or configuration. This includes problems with the following system management tools or functions: 򐂰 Configuration and management using administration tools 򐂰 Security 򐂰 JMX clients 򐂰 PMI and Tivoli Performance Viewer

Configuration and management using administration tools

Symptom: You have a problem accessing or using the administrative console, wsadmin scripting tool, or command line scripts. This would include the following symptoms: 򐂰 You are not able to access the administrative console. 򐂰 You cannot access server processes using wsadmin or the management scripts such as stopServer. 򐂰 You are getting errors performing system management functions, for example managing application servers, node agents, Web servers, or applications. 򐂰 You cannot federate a node with a deployment manager. 򐂰 You are getting save conflict messages in the administrative console. 򐂰 Your enterprise applications no longer appear in the administrative console.

Note: For problem determination strategies for these system management

problems, see WebSphere Application Server V6: System Management Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4067.pdf

Resources for problems with scripting tools

You can find problem determination strategies for issues with administrative scripting tools in the following resources: 򐂰 WebSphere Information Center: Using Ant to automate tasks http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/tovr_ant.html 򐂰 WebSphere Information Center: Using command line tools http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/txml_command.html

Chapter 1. Approach to problem determination 37

򐂰 WebSphere Information Center: Troubleshooting administration http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/ttrb_admin.html 򐂰 WebSphere Information Center: Explanation of WASX system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WASX.html 򐂰 MustGather: ws_ant problems on V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196231 򐂰 MustGather: WebSphere configuration archive functionality http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21200348

Resources for other system management problems

You can find problem determination strategies for other system management issues in the following resources: 򐂰 WebSphere Information Center: Setting up the administrative architecture http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/tagt_admin.html 򐂰 WebSphere Information Center: Administering application servers http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/trun_svr_conf.html 򐂰 WebSphere Information Center: Troubleshooting administration http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/ttrb_admin.html 򐂰 WebSphere Information Center: Explanations of ADFS, ADMB, ADMC, ADMD, ADME, ADMF, ADMG, ADMK, ADML, ADMN, ADMR, ADMS, ADMU, BNDE, CHKC, CHKP, CHKS, CHKW, ECNS, ODCF, PROC, WACT, and WSVM system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.express.doc/info/exp/ae/welc_ref_trb_msg.html 򐂰 MustGather: System management functionality for V5.0, V5.1 and V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199596 򐂰 MustGather: Synchronization problems in V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196219 򐂰 MustGather: Federation or Removal of a Node Issues for Version V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196227 򐂰 MustGather: Profile Creation/Removal Issues for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196228

38 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 MustGather: Usage and creation of templates fail on V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21195439򐂰 MustGather: Node agent and Deployment Manager discovery problems for all releases and editions of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196220򐂰 MustGather: Port Management for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196226򐂰 WebSphere Application Server V6 System Management & Configuration Handbook, SG24-6451.

Note: For information about problems with SSL configuration see WebSphere Application Server V6: System Management Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4067.pdf

Chapter 1. Approach to problem determination 39

򐂰 WebSphere Information Center: Explanation of SECG system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SECG.html 򐂰 WebSphere Information Center: Explanation of JSAS system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/JSAS.html 򐂰 WebSphere Information Center: Explanation of JSSL system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/JSSL.html 򐂰 WebSphere Information Center: Explanation of WSEC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSEC.html 򐂰 WebSphere Information Center: Explanation of WSSK system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSSK.html 򐂰 MustGather: Problems using Global Security for all Releases of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199336 򐂰 MustGather: Problems using JAAS Web logins for all Releases of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199332 򐂰 MustGather: Problems using Web services Security for all Releases of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199335 򐂰 MustGather: Java Secure Socket Extension (JSSE), SSL or Java Cryptography Extensions (JCE) problems http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21162961 򐂰 WebSphere Application Server V6 Security Handbook, SG24-6316.

40 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 WebSphere Information Center: Troubleshooting administration http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/ttrb_admin.html 򐂰 MustGather: JMX API client for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196218

PMI and Tivoli Performance Viewer

Symptom: You have a problem with the Performance Monitoring Infrastructure (PMI) or the Tivoli Performance Viewer. You might receive WebSphere system messages that begin with PMON. You might also have problems configuring PMI or performing any operation with the Tivoli Performance Viewer in the administrative console.

The PMI is instrumented by WebSphere Application Server V6 to monitor

performance through several counters and statistics. You can view the performance data using the Tivoli Performance Viewer within the administrative console. For information about these features, see "Testing the application" on page 7.

Chapter 1. Approach to problem determination 41

򐂰 Unexpected results from an application 򐂰 Application cannot connect to an external system or resource 򐂰 Application is slow or its performance degrades over time

Application or WebSphere process is unable to start

Symptom: An application or WebSphere Application Server process (for example an application server, node agent, or deployment manager) is unable to start.

Note: For problems with the startServer, stopServer, startNode, stopNode,

startManager, and stopManager commands, see WebSphere Application Server V6: System Management Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4067.pdf

42 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Application does not respond to incoming requestsSymptom: An application does not respond to incoming requests. The fault couldlie with any of the following components or conditions:򐂰 IBM HTTP Server򐂰 Edge components򐂰 Web server plug-in򐂰 Application server crash򐂰 Application server hang򐂰 100% CPU utilization򐂰 Out of memory

IBM HTTP Server

Symptom: An application server does not respond to incoming requests andstatic HTML pages are not being served by the IBM HTTP Server.

Chapter 1. Approach to problem determination 43

򐂰 IBM HTTP Server, Version 6 Information Center http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.ihs.doc/info/welcome_ihs.html 򐂰 IBM HTTP Server, Version 6 Information Center, Troubleshooting IBM HTTP Server http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.ihs.doc/info/aes/ae/welc_troub.html 򐂰 MustGather: Read first for IBM HTTP Server http://www-1.ibm.com/support/docview.wss?rs=177&uid=swg21192683

44 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 Load Balancer, Version 6 Troubleshooting http://www-306.ibm.com/software/webservers/appserv/doc/v60/ec/infocenter/ edge/LBguide.htm#HDRTRB

Web server plug-in

The Web server plug-in enables the Web server to send requests for dynamiccontent to Web applications (servlets and JSPs) that are installed in WebSphereApplication Server.

Symptom: An application does not respond to incoming requests. The application

responds when accessed directly through the application server but not whenaccessed through the Web server. High-level symptoms of a Web server plug-inproblem include:򐂰 Users cannot access an application through the Web server򐂰 Load balancing and failover not working properly򐂰 Session data is being lost򐂰 Slow or intermittent application response򐂰 The Web server will not start after plug-in installation or configuration

When your application fails to respond to requests, first check to see if your Webserver is responding to requests for static HTML pages. If it is, you can narrowyour focus to the plug-in and the application server.

To determine if it is a plug-in problem, try to access a servlet or JSP™ directly on

the application server, bypassing the plug-in. You can do this by using theapplication server's HTTP transport port (port 9080 by default) in the URL toaccess the servlet or JSP. For example, you can try to access the snoop servlet,which is one of the WebSphere Application Server samples:http://localhost:9080/snoop

Chapter 1. Approach to problem determination 45

Application server crash Symptom: An application is not responding to incoming requests. The application server process is no longer running.

Application server hang

Symptom: An application is not responding to incoming requests but the application server process is still running.

46 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 Detecting hung threads in J2EE applications http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/ctrb_hangdetection.html򐂰 Web module or application server dies or hangs http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rtrb_appdies.html򐂰 IBM Education Assistant: WebSphere Application Server problem determination http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg27005460򐂰 You can find a list of MustGather documents for hangs with the following search argument http://www-1.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc1=SSCMPB9&q=Must GatherDocument

100% CPU utilization

Symptom: You observe that a WebSphere Application Server process (anapplication server, cluster member, node agent, or deployment manager)reaches 100% or an unusually high percentage of CPU utilization.

You might notice this by using your operating system utilities to check the CPUuse of each process on the system, or you might have an application server hangcaused by the high CPU use.

You can find problem determination strategies for 100% CPU usage issues in thefollowing resources:򐂰 MustGather: 100% CPU Usage on AIX® Platforms http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21116458򐂰 MustGather: 100% CPU Usage on HP-UX http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21166017򐂰 MustGather: 100% CPU usage on Linux® http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21132699򐂰 MustGather: 100% CPU Usage on Solaris™ platforms http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21115625򐂰 MustGather: 100% CPU Usage on Windows Platforms http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21137447

Out of memorySymptom: You observe that a WebSphere Application Server process (anapplication server, cluster member, node agent, or deployment manager)

Chapter 1. Approach to problem determination 47

consumes all of the available memory on your system, or you see a java.lang.OutOfMemoryError in the SystemOut or SystemErr log for the process.

You might notice this by checking the memory usage that is using your operating system utilities, by checking the logs, or by having an application server hang that is caused by an out of memory problem.

48 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Unexpected results from an applicationAn application produces unexpected results (possibly errors or exceptions). Thisis a broad category that includes problems that can occur in any of the followingruntime components or services:򐂰 Just-In-Time (JIT) compiler򐂰 Web container򐂰 EJB container򐂰 Classloader򐂰 Session management򐂰 Dynamic cache򐂰 Transaction manager򐂰 Workload management򐂰 High availability manager򐂰 Data replication service򐂰 Program model extensions򐂰 Internationalization/Double byte character set򐂰 Transport channel service

Just-In-Time (JIT) compiler

Symptom: You receive intermittent unexpected results from an application.Disabling the JIT compiler resolves the problem.

Chapter 1. Approach to problem determination 49

For more information about mixed mode threshold, see: http://www.ibm.com/developerworks/java/jdk/diagnosis

If the problem does still occur when JIT is disabled, review the other problem classifications described here, determine into which classification the problem fits, and follow the problem determination steps for that classification.

50 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Note: For problem determination strategies for Web container problems, see WebSphere Application Server V6: Web Container Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4058.pdf

EJB containerThe EJB container is the runtime environment for EJBs in WebSphereApplication Server and it handles all EJB requests from clients.

Symptom: You receive intermittent unexpected results from an application and

the problem appears to originate with an EJB (entity beans, session beans, andmessage-driven beans). High-level symptoms can include any of the following:򐂰 Unexpected behavior when an EJB runs򐂰 Problems with the EJB life cycle and caching provided by the EJB container򐂰 Any errors or exceptions that occur when running an EJB򐂰 Error or warning messages with the following prefixes: CNTR, PMGR, and ACIN

Many errors might be related to data access and transactions.

Chapter 1. Approach to problem determination 51

򐂰 WebSphere Information Center: Explanation of PMGR system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/PMGR.html 򐂰 WebSphere Information Center: Explanation of ACIN system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ACIN.html 򐂰 MustGather: EJB container for releases of V4.0, V5.0, V5.1 and V6 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21153218 򐂰 MustGather: Persistence Manager http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21200344

You can find problem determination strategies for classloader problems in the following resources: 򐂰 WebSphere Information Center: Class loading http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/trun_classload.html 򐂰 WebSphere Information Center: Class loading: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rrun_classload_rlinks.html 򐂰 WebSphere Information Center: Troubleshooting class loaders http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/ttrb_classload_viewer.html 򐂰 MustGather: Classloader Issues in V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196187

Session management A session (sometimes called an HTTP session) is a series of HTTP requests to a servlet from the same user using the same Web browser. WebSphere

52 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Application Server provides session management functionality to keep track ofeach user and enable applications to provide personalized content.

Symptom: You experience a problem relating to the management of HTTP

sessions by WebSphere Application Server. This can include any of thefollowing:򐂰 Unexpected session behavior򐂰 Session time outs򐂰 Problems with session storage (database persistence or memory-to-memory sessions)򐂰 Session data being lost򐂰 Errors or exceptions from the WebSphere Application Server session manager򐂰 Problems with personalized Web content򐂰 Error or warning messages with the prefix SESN

Note: For information about session management problems, see WebSphere

Application Server V6: Web Container Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4058.pdf

Note: When there is a session management problem in a clustered

environment, there might be an underlying Web server plug-in problem. For more information about this, see WebSphere Application Server V6: Web Server Plug-in Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4045.pdf

Chapter 1. Approach to problem determination 53

򐂰 WebSphere Information Center: Problems creating or using HTTP sessions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rtrb_httpsessprobs.html 򐂰 WebSphere Information Center: Managing HTTP sessions: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rprs_r4ln.html 򐂰 WebSphere Information Center: Explanation of SESN system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SESN.html 򐂰 MustGather: Sessions and session management problems in V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21192604

Note that problems with the data replication service can cause session management problems (see "Data replication service" on page 58).

Dynamic cache The dynamic cache feature caches the output of servlets, JSPs, and external Web services that are called by clients within WebSphere Application Server.

Symptom: You experience a problem with the dynamic cache feature of

WebSphere Application Server. Potential problems can include any of the following: 򐂰 Unexpected or incorrect cache behavior 򐂰 Errors and exceptions from the dynamic cache component 򐂰 Dynamic cache configuration issues 򐂰 Slow performance 򐂰 Error or warning messages with the prefix DYNA

54 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 WebSphere Information Center: Troubleshooting tips for the dynamic cache service http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rdyn_trb.html򐂰 WebSphere Information Center: Explanation of DYNA system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/DYNA.html򐂰 MustGather: Dynamic cache problems in V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21193837

Note that problems with the data replication service can cause dynamic cacheproblems (see "Data replication service" on page 58).

Symptom: You experience a problem relating to the management of transactions

by WebSphere Application Server. This includes any of the following symptoms:򐂰 Unexpected transaction behavior򐂰 Transaction time outs򐂰 Unexpected transaction rollbacks򐂰 Problems with transaction recovery򐂰 Any errors or exceptions from the transaction manager򐂰 Error or warning messages that have the prefixes WTRN or WLTC

Transaction problems might also be related to database, messaging, or

enterprise information system connection problems, which are described in"Application cannot connect to an external system or resource" on page 63.

Chapter 1. Approach to problem determination 55

򐂰 WebSphere Information Center: Tips for troubleshooting transactions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rjta_prob0.html 򐂰 WebSphere Information Center: Transaction service exceptions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rjta_except.html 򐂰 WebSphere Information Center: Explanation of WTRN system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WTRN.html 򐂰 WebSphere Information Center: Explanation of WLTC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WLTC.html 򐂰 MustGather: Java Transaction Service (JTS) problems in V4.0, V5.0, V5.1 and V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21153216

Symptom: You experience a problem with the WLM features of WebSphere

Application Server, including load balancing and failover of an EJB request or HTTP request. Potential problems include: 򐂰 Workload between cluster members not being balanced correctly 򐂰 A cluster member that is down does not failover 򐂰 Workload is not routed to a cluster member that is running 򐂰 An EJB client cannot reach any cluster members (a CORBA NO_IMPLEMENT error is issued) 򐂰 Any errors or exceptions relating to WLM 򐂰 Error or warning messages with the prefix WWLM

56 WebSphere Application Server V6 Problem Determination for Distributed Platforms

You can find problem determination strategies for problems with WLM of EJBrequests in the following resources:򐂰 WebSphere Information Center: Workload management (WLM) for distributed platforms http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/crun_wlm.html򐂰 WebSphere Information Center: Workload management component troubleshooting tips http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/rtrb_wlmcomp.html򐂰 WebSphere Information Center: Workload is not getting distributed http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/topic/com.ibm.websphe re.nd.doc/info/ae/ae/rtrb_wlmprobs.html򐂰 WebSphere Information Center: Workload management run-time exceptions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/rrun_wlm_exceptions.html򐂰 WebSphere Information Center: Clustering and workload management: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/rrun_wlm_rlinks.html򐂰 WebSphere Information Center: Explanation of WWLM system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WWLM.html򐂰 MustGather: Enterprise JavaBeans (EJB) workload management http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21052165

High availability manager

The high availability manager in WebSphere Application Server V6 eliminatessingle points of failure and provides failover for all applications and WebSphereApplication Server services. It manages the availability of your applications andservices. It is discussed in detail in "High availability and failover" on page 14.

Symptom: You experience a problem with the high availability manager in

WebSphere Application Server, including򐂰 One process in your core group fails and its services are not started on another process as expected.򐂰 Errors or exceptions from the high availability manager.򐂰 Error or warning messages that begin with HMGR, CWRCB, or CWWCW.

Chapter 1. Approach to problem determination 57

You can find problem determination strategies for high availability manager problems in the following resources: 򐂰 WebSphere Information Center: Setting up a high availability environment http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/trun_ha_environment.html 򐂰 WebSphere Information Center: High availability manager http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/crun_ha_hamanager.html 򐂰 WebSphere Information Center: Troubleshooting high availability environment problems http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/rtrb_ha_env_trbl.html 򐂰 WebSphere Information Center: Learning about high availability http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.pmc.nd.doc/tasks/tjt0014_.html 򐂰 WebSphere Information Center: Explanation of HMGR system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/HMGR.html 򐂰 WebSphere Information Center: Explanation of CWRCB system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/CWRCB.html 򐂰 WebSphere Information Center: Explanation of CWWCW system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/CWWCW.html 򐂰 MustGather: High Availability and the High Availability (HA) Manager http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21201016

Data replication service

The data replication service transfers information between cluster members, enabling memory-to-memory session replication, dynamic cache, and stateful session bean failover.

Symptom: You experience a problem with the data replication service in

WebSphere Application Server. Potential problems include: 򐂰 Unexpected data replication behavior (might appear as workload management, dynamic cache, or session management problems). 򐂰 Any errors or exceptions from the data replication service. 򐂰 Error or warning messages that begin with CWWDR.

58 WebSphere Application Server V6 Problem Determination for Distributed Platforms

You can find problem determination strategies for data replication serviceproblems in the following resources:򐂰 WebSphere Information Center: Replicating data across application servers in a cluster http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/trun_drs_replication.html򐂰 WebSphere Information Center: Replication http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/crun_drs_replication.html򐂰 WebSphere Information Center: Explanation of CWWDR system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/CWWDR.html

Program model extensions

Programming model extensions (PMEs) enhance application capability andperformance, and make programming and deployment faster and moreproductive.

Chapter 1. Approach to problem determination 59

򐂰 WebSphere Information Center: Troubleshooting ActivitySessions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ass/tasks/tas_probd.html 򐂰 WebSphere Information Center: ActivitySession service: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ass/ref/ras_rlinks.html 򐂰 WebSphere Information Center: Explanation of WACS system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WACS.html 򐂰 WebSphere Information Center: Application profiling http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_appprof.html 򐂰 WebSphere Information Center: Explanation of APPR system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/APPR.html 򐂰 WebSphere Information Center: Asynchronous beans http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_asb.html 򐂰 WebSphere Information Center: Explanation of ASYN system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ASYN.html 򐂰 WebSphere Information Center: Object pools http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_objp.html 򐂰 WebSphere Information Center: Object pools: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/asyncbns/ref/rasb_objpoolsrlinks.html 򐂰 WebSphere Information Center: Explanation of OBPL system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/OBPL.html 򐂰 WebSphere Information Center: Scheduler service http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_sch.html 򐂰 WebSphere Information Center: Explanation of SCHD system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SCHD.html

60 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 WebSphere Information Center: Startup beans http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_sub.html򐂰 WebSphere Information Center: Explanation of STUP system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/STUP.html򐂰 WebSphere Information Center: Work area http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_wa.html򐂰 WebSphere Information Center: Explanation of ACWA system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ACWA.html

Internationalization/Double byte character set

Symptom: You have any type of problem with character encoding or theinternationalization of your application (producing output in the language, timezone, currency, and cultural conventions for different regions or locales). Thiscan include:򐂰 Problems displaying double byte characters for certain languages.򐂰 Problems that only occur with internationalized applications.򐂰 Error or warning messages that begin with I18N and LTXT.

Note: Character encoding problems are discussed in WebSphere Application

Server V6: Web Container Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4058.pdf

You can find problem determination strategies for other internationalization

problems in the following resources:򐂰 WebSphere Information Center: Internationalization service http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_in.html򐂰 WebSphere Information Center: Internationalization service errors http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/i18n/ref/rin_troubleshoot.html򐂰 WebSphere Information Center: Internationalization: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rin_resources.html

Chapter 1. Approach to problem determination 61

򐂰 WebSphere Information Center: Explanation of I18N messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/I18N.html 򐂰 WebSphere Information Center: Explanation of LTXT system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/LTXT.html 򐂰 MustGather: i18n (Internationalization)/Double Byte Character Set (DBCS) http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21141732

Transport channel service

The transport channel service is a new feature in WebSphere Application Server V6. It manages client connections and I/O processing for HTTP and JMS requests based on the new non-blocking I/O features in Java 1.4.

62 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 WebSphere Information Center: Explanation of HTPC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/HTPC.html򐂰 WebSphere Information Center: Explanation of SSLC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SSLC.html򐂰 WebSphere Information Center: Explanation of TCPC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/TCPC.html򐂰 WebSphere Information Center: Explanation of WSSC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSSC.html򐂰 WebSphere Information Center: Explanation of XMEM system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/XMEM.html

Application cannot connect to an external system or resource

These problems include:򐂰 JCA connection manager򐂰 Database connections򐂰 Messaging򐂰 JNDI naming򐂰 ORB򐂰 Web services

JCA connection manager

Symptom: You have any problem connecting to an external resource through theJava Connector Architecture (JCA) connection manager. This includes datasources (which are used to obtain JDBC™ database connections), JMSconnection factories, and connections to enterprise information systems with aninstalled JCA resource adapter. The JCA connection manager pools andmanages connections to these systems.

Potential problems include general connection pooling problems or configuration

issues, problems establishing a connection, problems with activationspecifications that get messages from your back-end systems. JCA connectionsymptoms are observed through WebSphere error messages with prefixes DSRA,WSCL, J2CA, WTRN, CONM, SQLException, or database error codes.

Problems caused by JCA components or JCA connection configuration errors

can appear as one or more of the following initial symptoms:

Chapter 1. Approach to problem determination 63

򐂰 A connection hangs or incorrectly returns data to application 򐂰 An application connect to or access a database or EIS

Note: For problem determination strategies for JCA connection manager

problems, see WebSphere Application Server V6: JCA Connection Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4080.pdf

Note: For problem determination strategies for database connection

problems, see WebSphere Application Server V6: JCA Connection Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4080.pdf

64 WebSphere Application Server V6 Problem Determination for Distributed Platforms

CWSIA, CWSIB, CWSIC, CWSID, CWSIE, CWSIF, CWSIH, CWSII, CWSIJ, CWSIK, CWSIL, CWSIM, CWSIN, CWSIO, CWSIP, CWSIQ, CWSIR, CWSIS, CWSIT, CWSIU, CWSIV, CWSIW, CWSIX, CWSIY, CWSIZ, CWSJA, CWSJB, CWSJC, CWSJD, CWSJO, CWSJQ, CWSJR, CWSJU, CWSJW, and CWSWS. Note that these messages contain ten characters instead of nine.򐂰 Problems with the WebSphere MQ JMS provider or a generic JMS provider might include JMS connection factory or destination configuration issues, message listener service and message-driven bean problems, and WebSphere system messages that begin with WMSG.

Note: For problem determination strategies for messaging problems, see

WebSphere Application Server V6: Default Messaging Provider Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4076.pdf

Symptom: Your application experiences a problem when accessing any resource

through a JNDI lookup. Indications of a JNDI problem include:򐂰 JNDI naming error or exception in the SystemOut or SystemErr log.򐂰 Problems using the dumpNameSpace tool.򐂰 Error or warning messages that begin with NMSV.

Chapter 1. Approach to problem determination 65

򐂰 WebSphere Information Center: Naming and directories: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/topic/com.ibm.websphe re.base.doc/info/aes/ae/rnam_r4ln.html 򐂰 WebSphere Information Center: Explanation of NMSV system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/NMSV.html 򐂰 MustGather: Java Naming and Directory Interface (JNDI) and naming problems for all releases http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21143296

Symptom: Your application experiences a problem when accessing an EJB

through the Object Request Broker (ORB). Indications of an ORB problem include: 򐂰 ORB errors or exceptions in SystemOut or SystemErr log. Most likely, these would be CORBA error codes (such as a CORBA COMM_FAILURE). 򐂰 Error or warning messages that begin with ORBX.

66 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 WebSphere Information Center: Explanation of ORBX system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ORBX.html򐂰 MustGather: Object Request Broker (ORB) for all releases http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21157005

Symptom: Your application experiences a problem when acting as a Web

services client to a remote Web service or when acting as a Web service that isaccessed by external clients. This would include:򐂰 Web services errors and exceptions that appear in the SystemOut and SystemErr logs򐂰 Unexpected Web services behavior򐂰 Configuration issues򐂰 Problems with Web services tooling, such as the Java2WSDL and WSDL2Java scripts򐂰 Problems that occur when configuring Web services or Web services gateway instances with the service integration bus.򐂰 Error or warning messages that begin with WSWS, SOAP, WSIF (for Web services), or CWWSG (for Web services gateway)

You can find problem determination strategies for Web services problems in thefollowing resources:򐂰 WebSphere Information Center: Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_wbs.html򐂰 WebSphere Information Center: Troubleshooting Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twbs_troubleshootwbs.html򐂰 WebSphere Information Center: Troubleshooting Web services command-line tools http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_trbcommand.html

Chapter 1. Approach to problem determination 67

򐂰 WebSphere Information Center: Troubleshooting Web services compiled bindings http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_trbjavacompiler.html 򐂰 WebSphere Information Center: Troubleshooting the runtime for a Web services client http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_trbclientruntime.html 򐂰 WebSphere Information Center: Troubleshooting serialization and deserialization in Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_trbserialize.html 򐂰 WebSphere Information Center: Troubleshooting the Web Services Invocation Framework http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twsf_trouble.html 򐂰 WebSphere Information Center: UDDI Registry troubleshooting http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twsu_probdet.html 򐂰 WebSphere Information Center: Universal Discovery, Description, and Integration, Web Service, and SOAP component troubleshooting tips http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rtrb_svsccomp.html 򐂰 WebSphere Information Center: Errors returned to a client sending a SOAP request http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rtrb_soapprobs.html 򐂰 WebSphere Information Center: Tracing Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twbs_tracewbscomp.html 򐂰 WebSphere Information Center: Tracing SOAP messages with tcpmon http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twbs_tracewbs.html 򐂰 WebSphere Information Center: Frequently asked questions about Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_faq.html

68 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 WebSphere Information Center: Web services: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_resourceslearning2.html򐂰 WebSphere Information Center: Explanation of WSWS system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSWS.html򐂰 WebSphere Information Center: Explanation of SOAP system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SOAP.html򐂰 WebSphere Information Center: Explanation of WSIF system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSIF.html򐂰 WebSphere Information Center: Explanation of CWWSG system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/CWWSG.html򐂰 MustGather: Web Services Engine problems for all releases and editions of WebSphere Application Server V5.0.2, V5.1 and V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21198363򐂰 MustGather: Problems with the Web services gateway http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21159216򐂰 MustGather: Problems with the Web Services Invocation Framework (WSIF) http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21159155򐂰 MustGather: Problems with UDDI http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21164764

Application is slow or its performance degrades over time

This broad classification covers all problems with slow or degradingperformance.

Chapter 1. Approach to problem determination 69

򐂰 WebSphere Information Center: Performance: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rprf_resourceslearning.html 򐂰 WebSphere Application Server Performance information http://www-306.ibm.com/software/webservers/appserv/was/performance.html 򐂰 WebSphere Application Server V6 Scalability and Performance Handbook, SG24-6392.

1.5 Contacting IBM for support

As you might have realized by now, many potential problems with WebSphere Application Server V6 can be resolved without the help of the WebSphere Application Server Support team. By carefully preparing for and preventing problems and by following your problem determination strategy, you have the ability to resolve most problems on your own. However, there might be instances where invoking the Support team will be necessary, especially if a WebSphere Application Server code defect is the cause of a problem. This section explains the IBM support process and how you can work with the Support team most effectively.

1.5.1 IBM support structure

In order to get the most out of the IBM support process, it is important to understand how the support organization is structured. There are several groups of support personnel with whom you might work for each problem:

70 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 Front office (Problem entry) team If you report a problem by phone, the first person you speak with is a member of the front office (also called problem entry) team. The people on this team work with you in your national language. They are usually located in the same region of the world from where you are calling. The main goals of the front office team are to get a detailed problem description from you, confirm your contact information and availability, open a problem management record (PMR), and then route your PMR to the appropriate product support team.򐂰 Front end (Level 1) team The next step in the process is for the front end team (called the Level 1 support team in the United States) to work on your PMR. If you open your PMR electronically, this would be the first team with whom you would work. This team also works with you in your national language, and they are usually located in the same region of the world as you. The people on the front end team in many countries have broad skills in many IBM software products, and they work PMRs for several products. The Level 1 team in the United States only works WebSphere Application Server PMRs. They have broad skills in WebSphere Application Server, and they support the entire product. The support analysts obtain more details about the problem. They are likely to request files or logs that aid in determining the cause of the problem. They then conduct an investigation into the problem and provide possible solutions. If they are unable to provide a solution to the problem, they escalate the PMR to the next level of support.򐂰 Back end (Level 2) team Your PMR is escalated to the back end (Level 2) support team if it cannot be resolved by the front end or Level 1 team. The Level 2 team for WebSphere Application Server is located only in the United States, and they work only in English. The front end team asks you if you are able to work with the Level 2 team in English. If you would prefer to work in your own language, the front end team can translate the communication from the Level 2 team for you. The Level 2 team is composed of a group of specialized teams. Each team works only on certain areas of the product, and they develop expertise in those areas. They conduct a more extensive analysis of the problem and the files that you have sent, and they obtain more information about the problem if it is required. The Level 2 team might also ask for more detailed diagnostic data, such as traces for a specific component, to help in their investigation. After analyzing all of the information and conducting research, the Level 2 team either provides a solution to the problem or works with the next level of support if they believe that the problem is caused by a WebSphere

Chapter 1. Approach to problem determination 71

Application Server code defect. In that case, an authorized program analysis report (APAR) is opened. 򐂰 Change team (Level 3) If the Level 2 team's analysis shows that the problem is caused by a defect with WebSphere Application Server code, they send your PMR to the change team, also known as the Level 3 team. The change team is part of the WebSphere Application Server development team that is specifically focused on fixing defects. The members of the Level 3 team work only on one specific component of WebSphere Application Server. They have very deep knowledge of that particular component. The Level 3 team does not communicate directly with clients. Instead, they provide updates to the Level 2 team, and the Level 2 team communicates updates to the client.

1.5.2 Research the problem

Before you open a PMR, we recommend that you will follow all of the problem determination strategies discussed in this book. The most important thing to do is to research the problem. The WebSphere Application Server Support team has found that a large percentage of problems have already been reported by other clients. In these cases, you can save a great deal of time and effort by reviewing IBM documentation to make sure that the problem is not a known problem before you open a PMR. You can review the list of Web sites in "Investigate and research the problem" on page 27 to conduct your research.

1.5.3 Collect MustGather files

We first mentioned MustGather documents in "Diagnostic data collection" on page 21. These documents are useful guides when developing your diagnostic data collection plan, and they are also important when you open a PMR with the WebSphere Application Server Support team. The support team needs certain diagnostic data to investigate each type of WebSphere Application Server problem. In "Classify the problem and determine the root cause" on page 31, we listed each problem classification, and we included the links to the MustGather documents for each classification. You can review the specific document for the type of problem that you are experiencing, or you can start with the general MustGather document for all types of problems, IBM - MustGather: Read first for all WebSphere Application Server products, at: http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21145599

72 WebSphere Application Server V6 Problem Determination for Distributed Platforms

1.5.4 Determine the severity When you open a PMR, you provide a severity level for the PMR. The severity levels are in the range of 1 to 4, with 1 being the most severe and 4 being the least severe. The severity level reflects the business impact and, therefore, the urgency of the problem. Although it is tempting to open all PMRs at the Severity 1 level, it is very important to determine an accurate severity level for the situation. IBM support prioritizes PMRs according to the severity level. When you have a true Severity 1 issue, you do not want the support team to have to prioritize it at the same level as other issues that are not as critical. Table 1-2 provides the official descriptions of each severity level from the IBM Software Support Handbook.

Note: Although IBM support analysts might sometimes ask you if a severity level can be changed, it is ultimately your decision (or the decision of others in your company) as to what the severity level is.

Table 1-2 Guidelines of PMR severity levels

Severity level Description

Severity 1 Critical business impact: You are unable

to use the program, resulting in a critical impact on operations. This condition requires an immediate solution.

Severity 2 Significant business impact: The

program is usable but is severely limited.

Severity 3 Some business impact: The program is

usable with less significant features (not critical to operations) unavailable.

Severity 4 Minimal business impact: The problem

causes little impact on operations, or a reasonable circumvention to the problem has been implemented.

To put these guidelines in WebSphere Application Server terms:

򐂰 Severity 1 PMRs are typically situations where the availability of your production environment is affected. In a case where your production system is down, the support team's first priority is to help you restore your production functionality, as discussed in "Revert to safe conditions" on page 24. It can be any situation where there is a high level of business impact. For example, it could be a problem that causes a loss of revenue or productivity for your organization. For Severity 1

Chapter 1. Approach to problem determination 73

PMRs, there is no work around or solution that is currently available. The support team works around the clock on your PMR, as long as they are able to contact you at any time of day, until the problem is resolved. 򐂰 Severity 2 PMRs are the most common levels of PMRs. In this case, the problem has a significant impact, but the application is still running in your production environment. It can also include situations where your application is not yet in production. These include situations where you have a deadline for resolving the problem, and situations where the problem delays when your application goes into production. Severity 2 PMRs are considered severe and are prioritized highly by the support team but the problems are not considered as critical as Severity 1 problems. The support team works with you during the normal business hours in your time zone. 򐂰 Severity 3 PMRs indicate that a problem is not severely impacting your business. Almost all Severity 3 problems occur in your test environment and do not affect production. If it does impact production, it has only a minor impact, and you might have a work around for the problem. Although you expect the problem to be resolved in a timely manner, you do not have a deadline for resolving the problem and the problem is not delaying the production date for your application. The support team works with you during the normal business hours in your time zone. 򐂰 Severity 4 PMRs indicate that a problem has almost no business impact. This includes technical questions and requests to update documentation. It can also include problems that occur only in your test environment where a work around has already been provided. The severity is sometimes lowered to Severity 4 when a solution is provided, but you need some time to test the solution before closing the PMR. It is expected that Severity 4 PMRs are not impacting your production environment. You do not have a deadline for resolving the problem and you are able to wait a longer period of time to receive a solution. The support team will work with you during the normal business hours in your time zone and updates by the support team might not be frequent.

1.5.5 Create a PMR

Now that you are familiar with the IBM support structure, you have researched the problem, collected the MustGather data, and determined the severity of the problem, it is time to open a PMR.

74 WebSphere Application Server V6 Problem Determination for Distributed Platforms

You can currently open a PMR in two ways, electronically or by phone. We havefound that opening PMRs electronically is more convenient for most clients. Werecommend that you use that method. However, phone support is still available.

Open a PMR electronically

To open a PMR electronically on the Web, you can use the Electronic ServiceRequest (ESR) problem submission tool at:http://www-306.ibm.com/software/support/probsub.html

Open a PMR by phone

Your other option is to open a PMR by phone. The phone number that you call toopen a PMR varies by country and changes from time to time. To find the phonenumber for your country, check this link from the IBM Software SupportHandbook:http://techsupport.services.ibm.com/guides/contacts.html

Chapter 1. Approach to problem determination 75

1.5.6 Send data to IBM In the course of working a PMR, it might be necessary to send many different types of data to the WebSphere Application Server Support team. As discussed in "Create a PMR" on page 74, using the ESR tool to attach files to your PMR makes this process easy. However, if you do not use the ESR tool, you can still send data through e-mail or FTP. As a general rule, you should use e-mail if the total size of the data is less than 10 MB, and you should use FTP if the data is more than 10 MB in size. Also, when you FTP data, your PMR is updated automatically so that the support team is aware that you sent the data and they can begin analyzing it right away. However, if you send an e-mail, be sure to either update your PMR electronically or call the IBM support phone number for your country and ask to have the PMR updated, so that the support team knows that you sent the data.

We recommend that you send all e-mail related to your PMR to

weblev2@us.ibm.com. This is a shared e-mail address. The entire WebSphere Application Server Level 1 and Level 2 teams have access to this e-mail address. If you only send an e-mail to a specific support analyst, only that analyst has access to your e-mail. This can be disadvantageous, especially if the analyst is out of the office or if you need help outside of normal business hours.

76 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Chapter 2. Diagnostic data

This paper contains information about the diagnostic data that is available in WebSphere Application Server V6. It contains information about the location of the data, how it is collected, and configuration options.

It includes information about the following:

򐂰 JVM logs (SystemOut and SystemErr) 򐂰 Tracing 򐂰 Collector tool 򐂰 First Failure Data Capture (FFDC) 򐂰 Process (native) logs 򐂰 Service log (activity.log) 򐂰 Installation logs 򐂰 IBM HTTP Server and plug-in logs and traces 򐂰 System management logs 򐂰 WebSphere Rapid Deployment logs

© Copyright IBM Corp. 2005. All rights reserved. 77

2.1 JVM logs (SystemOut and SystemErr) SystemOut and SystemErr logs are created for every WebSphere Application Server process (application server, cluster member, node agent, and deployment manager). These logs are known as JVM logs. The System.Out and System.Err streams for each JVM are redirected to the SystemOut and SystemErr logs. WebSphere Application Server writes to these logs. Your applications can also write to them by using the print(), println(), and printStackTrace() methods.

You can find the logs in the following directory:

/profiles//logs/

Figure 2-1 Changing the log file rotation properties

78 WebSphere Application Server V6 Problem Determination for Distributed Platforms

In the Configuration tab, you can edit the following:򐂰 The file name (and the directory) of the SystemOut and SystemErr logs򐂰 The file formatting We recommend that you leave this at the default value of Basic to make the logs easier to read.򐂰 Log file rotation The SystemOut and SystemErr logs are self-managing. They write to the specified file until either the maximum file size or a certain time is reached. When that happens, the current log file is renamed as the current file name plus the current time stamp. Then a new SystemOut or SystemErr file is created for further logging. The older log files are called historical log files. For example, after this occurs you might have the following SystemOut files in the /profiles//logs/ directory: - SystemOut.log, the current log file - SystemOut_05.06.07_10.28.48.log, the historical log file Depending on your needs, you can choose to have the log files rotate (roll over) when they reach a specified size, a certain time interval, or both. If you choose a time, we recommend that you specify 24 hours as the Repeat Time. You can also set the Start Time to specify the time at which the logs rotate. If you specify a file size, we recommend that you increase the Maximum Size above its default of 1 MB. You will want to coordinate the value of Maximum Size with the Maximum Number of Historical Log Files, based on the available disk space on your system. With either method, make sure that the amount of log data that is saved is enough so that the relevant log data is there when you identify that a problem has occurred.򐂰 Maximum Number of Historical Log Files The value that is entered here is the number of historical log files that are kept. If the value is reached and another historical log file needs to be created, the oldest one is removed from your system.򐂰 Installed Application Output These properties affect how print and println statements from your applications are output. There are two options: - Show application print statements. This is enabled by default. If you deselect it, application print and println statements are not logged to the SystemOut and SystemErr log files. - Format print statements. This is also enabled by default. You can deselect it if you do not want your application print and println statements to be

Chapter 2. Diagnostic data 79

formatted similar to WebSphere Application Server messages in the log files.

All of these properties can be changed for both the SystemOut and the SystemErr logs. You can choose to use the same properties for both logs, which we recommend, or use different properties for them.

The entries in the output of the SystemOut.log are in the following format: [7/12/05 14:46:00:264 EDT] 0000001a ApplicationMg A WSVR0221I: Application started: adminconsole

Each entry can be deciphered as follows:

򐂰 Time stamp In the example, the time stamp is [7/12/05 14:46:00:264 EDT]. The time stamp is formatted using the locale of the process where it is formatted. It includes a fully qualified date (for example MM/DD/YY), 24-hour time with millisecond precision, and a time zone. 򐂰 Thread ID In the example, the thread ID is 0000001a. The thread ID is an eight-character hexadecimal value that is generated from the hash code of the thread that issued the message. 򐂰 Short name In the example, the short name is ApplicationMg. The short name is the abbreviated name of the component that issued the message. This name is typically the class name of a WebSphere Application Server component and would be some other identifier for applications.

80 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 Event type In the example, the event type is A. The event type is a one character field that indicates the type of the message. The possible values are: - F - fatal message - E - error message - W - warning message - A - audit message - I - informational message - C - configuration message - D - detail message - O - message that was written directly to System.out by an application or internal components - R - message that was written directly to System.err by the user application or internal components - Z - a placeholder to indicate that the type was not recognized򐂰 Message Identifier In the example, the message identifier is WSVR0221I. The message identifier is a string that is nine characters in length and is in the form CCCC1234X. The first four characters (CCCC) indicate the WebSphere Application Server component that issued the message. The next four characters (1234) indicate the specific message that the component is issuing. The last character (X) indicates the severity of the message. Its value is either I (informational), W (warning), or E (error). You can find descriptions of all WebSphere Application Server message identifiers in the WebSphere Information Center item Troubleshooter reference: Messages at: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.express.doc/info/exp/ae/welc_ref_trb_msg.html򐂰 Message In the example, the message is Application started: adminconsole. The message is the data that is logged to the SystemOut.log by the component. It is meant to provide useful output for informational purposes, debugging, and troubleshooting.

Chapter 2. Diagnostic data 81

2.2 Tracing Trace logs can also be configured in a manner similar to the JVM logs. Traces must be explicitly enabled. They are disabled by default. Trace output gives very detailed information about the execution of WebSphere Application Server code. It provides time stamps, details about which WebSphere methods were called, and special diagnostic data that is included to make troubleshooting easier.

To configure the properties of the traces from the administrative console:

1. Select Troubleshooting → Logs and Trace. 2. Select the process whose trace logs you want to configure. 3. Click Diagnostic Trace. The General Properties window opens, as shown in Figure 2-2.

Figure 2-2 Changing the trace properties

82 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Trace files cannot be rolled over based on time. You must specify a MaximumFile Size in conjunction with the Maximum Number of Historical Files. You shouldset these values appropriately, depending on how long it might take to reproducethe problem with trace enabled and how much disk space is available. Withthese properties set, the trace files roll over in the same manner as the JVM logs.You can also specify a File Name and specify a directory for the trace files.

As with the JVM logs, we strongly recommend selecting Basic (the default value)for the Trace Output Format. This makes the trace easier to read, and it is thepreferred format of the WebSphere Application Server support team.

Chapter 2. Diagnostic data 83

Figure 2-3 Changing the log detail level in the administrative console

Components can be any WebSphere Application Server packages or classes.

Groups are predefined sets of packages and classes that are useful for troubleshooting a particular component.

84 WebSphere Application Server V6 Problem Determination for Distributed Platforms

The MustGather documents for WebSphere Application Server componentsdiscuss which specific log detail level to set for different types of problems. It is agood idea to record the log detail level for different types of problems in yourdiagnostic data collection plan. When setting the log detail level, you should setthe level to all in almost all cases. You should also include *=info in thebeginning of the log detail level so that informational logging is enabled forcomponents that are not being traced. For example, in order to enable a trace forthe J2C connection manager component, you would set the log detail level to:*=info:WAS.j2c=all

Each entry can be deciphered as follows:

򐂰 Time stamp In the example, the time stamp is [7/12/05 16:13:10:379 EDT]. The time stamp is formatted using the locale of the process where it is formatted. It includes a fully qualified date (for example MM/DD/YY), 24-hour time with millisecond precision, and a time zone.򐂰 Thread ID In the example, the thread ID is 00000032. The thread ID is an eight-character hexadecimal value generated from the hash code of the thread that issued the trace event.

Chapter 2. Diagnostic data 85

򐂰 Short name In the example, the short name is DSConfigurati. The short name is the abbreviated name of the component that issued the trace event. This is typically the class name of a WebSphere Application Server component, and would be some other identifier for applications. 򐂰 Event type In the example, the event type is a greater than symbol (>). The event type is a one character field that indicates the type of the trace event. The possible values are: - > - indicates the entry of the specified method name - < - indicates the exit of the specified method name - 1 - a trace entry of type fine or event - 2 - a trace entry of type finer - 3 - a trace entry of type finest, debug, or dump - Z - a placeholder to indicate that the trace type was not recognized The example indicates that the getPooledConnection method is entered. 򐂰 Class name The class name is an optional part of the trace entry. It indicates the class that generated the trace event. In the example, the class name does not appear. 򐂰 Method name The method name is another optional part of the trace entry. It indicates the method that generated the trace event. In the example, the method name is getPooledConnection. 򐂰 Text message In the example, the message is Entry.

86 WebSphere Application Server V6 Problem Determination for Distributed Platforms

2.3 Collector tool The WebSphere Application Server collector tool is a script that can be found in the /bin directory (collector.bat or collector.sh). Running the script produces a Java archive (jar) file that contains all of the logs and XML configuration files from your WebSphere Application Server installation, as well as operating system information, Java information, and data on whether the software prerequisites were met and their levels.

2.4 First Failure Data Capture (FFDC)

WebSphere Application Server V6 includes a feature called First Failure Data Capture (FFDC). The FFDC feature runs in the background and collects events and errors that occur during WebSphere Application Server runtime. The information that it collects are written to log files in the /profiles//logs/ffdc directory.

FFDC does not affect the performance of WebSphere Application Server and should not be disabled. The FFDC logs will not, most likely, be useful in your

Chapter 2. Diagnostic data 87

problem determination efforts. However, they might be useful to the WebSphere Application Server support team if you open a PMR.

There are three FFDC configuration files in the /properties

directory. The only file that you should modify is the ffdcRun.properties file. You can add the ExceptionFileMaximumAge property to the file. This property specifies the number of days that an FFDC log remains in the /profiles//logs/ffdc directory before it is deleted. As part of your diagnostic data collection plan, you might want to modify the ExceptionFileMaximumAge property to ensure that the FFDC files remain on your system for a certain time period. You should not modify any other properties unless you are asked to do so by the WebSphere Application Server support team.

2.5 Other logs

The following logs are not always useful for problem determination, but you might find that on occasion they will be required.

2.5.1 Process (native) logs

Native code running in a WebSphere Application Server process can write data to the process logs (also called native logs). Native code is non-Java code, typically found in files with .dll, .exe, and .so extensions. The process logs are named native_stdout.log and native_stderr.log. They are located in the /profiles//logs/ directory.

88 WebSphere Application Server V6 Problem Determination for Distributed Platforms

You can find more information about the process logs in the WebSphere Information Center item Process logs at: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/ctrb_stdlogs.html

2.5.2 Service log (activity.log)

The service log is more commonly known as the activity.log and is found in the /profiles//logs directory. There is only one activity.log for each node. WebSphere Application Server runtime events are logged to the activity.log. It is written in binary format, so it cannot be viewed in a text editor. The main purpose of the activity.log is that it can be viewed with the Log Analyzer tool, is a graphical user interface that displays the events from the activity.log and uses a symptom database to analyze the events and diagnose problems.

It is also possible to view the events in the activity.log outside of the Log Analyzer by using the showlog script in the /bin directory.

You can configure properties of the activity.log in the administrative console:

1. Select Troubleshooting → Logs and Trace. 2. Select the WebSphere Application Server process. 3. Select IBM Service Logs. You can select whether to enable or disable the activity.log, choose the directory location and file name, set the maximum file size, and select what types of messages will be logged.

Chapter 2. Diagnostic data 89

2.5.3 Installation logs When you have a problem installing WebSphere Application Server V6, you might need to view the following logs to determine the failure causes. 򐂰 /logs/log.txt This log file records the installation status 򐂰 /profiles//logs/pctlog.txt This log file records the profile creation status 򐂰 /profiles//logs/ivtClient.log This log file records the events of install verification test

2.6 IBM HTTP Server and plug-in logs and traces

When you have a problem relating to the IBM HTTP Server or the Web server plug-in, you might need to view the logs or enable a trace. This section discusses the details about these logs and traces.

2.6.1 IBM HTTP Server logs

The IBM HTTP Server writes two log files: an access log that contains details of all accesses to the Web server and an error log that contains details of any errors. The default location of the logs is as follows: 򐂰 Windows - Access log: \logs\access.log - Error log: \logs\error.log 򐂰 UNIX® - Access log: /logs/access_log - Error log: /logs/error_log

2.6.2 Web server plug-in logs

The plug-in also writes its own log, which you can find in the Web server plug-in install directory path. The log file that you are looking for is under another directory structure, named for the logical Web server as defined in the WebSphere configuration.

You can find the location of the log file by first looking at the Web server configuration. This refers to the plug-in configuration file as shown in Example 2-1 on page 91. The plug-in configuration file then tells you where the

90 WebSphere Application Server V6 Problem Determination for Distributed Platforms

log file is as shown in Example 2-1. This example also shows you where you set the amount of detail that is logged.

Example 2-1 Location of plug-in log file

The default setting for LogLevel is Error, but you can set it to Trace to collect significantly more information. Should you need to raise this problem with IBM Support, they will request a plug-in trace.

2.6.3 Web server plug-in trace

To get an effective trace, you need to enable as much logging as possible on the Web server. For example, you can set the logging level to capture verbose output in the IBM HTTP Server by modifying the LogLevel directive in the configuration file as shown: LogLevel debug

You need to restart the IBM HTTP Server for this to take effect.

You do not need to restart the IBM HTTP Server for this change to take effect.

Tip: The plug-in trace generates significant amounts of data. Make your test as specific as possible, and run it in isolation to reduce the number of lines generated.

2.6.4 Network trace

In rare cases, you might need to use a network protocol analyzer that allows you to capture an iptrace. This tool can help you to determine where the problem lies. WebSphere Application Server does not supply such a tool. However, there are third-party tools available (for example, Ethereal from http://www.ethereal.com/).

Chapter 2. Diagnostic data 91

2.7 System management logs When you have a system management problem, you might need to view certain logs or enable a trace. This section discusses the details about these logs and traces.

2.7.1 Output from wsadmin

Messages from wsadmin are written to the wsadmin.traceout log file: /profiles//logs/wsadmin.traceout

You can also increase the amount of data that is logged to this file by tracing the wsadmin utility. To do so, update the following file: /properties/wsadmin.properties

Uncomment the following line:

com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled

Note that the information that is logged is of limited use because wsadmin calls MBeans in the application server that is running the administrative console application. So, you usually need to trace that application server as well.

2.7.2 Management scripts

You can manage WebSphere Application Server services using the supplied management scripts. For example, each WebSphere Application Server installation has a script to start an application server, a script to stop an application server, and a script to show you the status of all application servers defined in a profile. Each of these scripts writes its own log file into the server's logs directory. For example, the stopServer script writes stopServer.log into the logs directory: /profiles//logs//stopServer.log

2.7.3 Profile management logs

The profile creation and management tool wasprofile writes messages to the profile independent logs directory, that is: /logs/wasprofile/.log

This log file is in XML format.

92 WebSphere Application Server V6 Problem Determination for Distributed Platforms

The Java graphical interface that is used to create a profile simply calls the wasprofile command after collecting the information needed. By default, it does not write a log, but you can pass it a log parameter as shown: pctWindows -is:log c:\temp\pct.log

2.8 WebSphere Rapid Deployment logs

The WebSphere Rapid Deployment tool works on a directory that you create and pass to WebSphere Rapid Deployment in the WORKSPACE environment variable. It logs Eclipse messages into two separate files within this directory: 򐂰 /.metadata/.log 򐂰 /project/.metadata/.log

In a manner similar to other WebSphere Application Server utilities, WebSphere

Rapid Deployment calls MBeans on the application server. The application server logs can help you resolve a problem with WebSphere Rapid Deployment. There is no way to trace the WebSphere Rapid Deployment utility. However, you can trace the application server as described in "Tracing" on page 82.

2.9 Summary of logs

Table 2-1 on page 94 shows a summary of the WebSphere logs.

Note: In this table:

򐂰represents the installation root for WebSphere Application Server, for example: c:\WebSphere\Appserver 򐂰represents the root directory for a specific WebSphere Application Server profile, for example: /profiles/ 򐂰represents the installation directory for the IBM HTTP Server, for example: c:\IBM HTTP Server

Chapter 2. Diagnostic data 93

Table 2-1 Log file summary Tasks Logs Format / tools

Installation tasks

WebSphere Installation /logs/log.txt text

IBM HTTP Server installation / text

򐂰 ihsv6_install.log 򐂰 gskitInstall.log

Sample and IVT application /logs/ text

installation: 򐂰 DefaultApplication 򐂰 defaultapp_config.log 򐂰 ivtApp 򐂰 defaultapp_deploy.log 򐂰 query 򐂰 ivt_config.log 򐂰 PlantsByWebSphere 򐂰 query_config.log 򐂰 SamplesGallery 򐂰 samples_config.log 򐂰 samples_install.log

WebSphere system application /logs/ text

installation: 򐂰 filetransfer 򐂰 filetransfer_config.log 򐂰 filetransferSecured 򐂰 mejb.log 򐂰 ManagementEJB 򐂰 scheduler.cal_config.log 򐂰 SchedulerCalenders 򐂰 webui_config.log 򐂰 adminconsole

Profile tasks

Profile creation wizard /logs/pctLog.txt text

/logs/waspro XML file/wasprofile_create_.l og.

Application and system tasks

Application print() and println() 2 /logs//: Text.

򐂰 SystemOut.log1 JVM System.out and System.err 򐂰 SystemErr.log1 administrative console: streams Troubleshooting → Logs and Trace →→JVM Logs 1 Configurable 2 can represent the location of the profile for an application server, node agent, or deployment manager. If the profile is for a node agent,is "nodeagent". If the profile is for a deployment manager,is "dmgr"

94 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Tasks Logs Format / tools

WebSphere processes Process logs at Text (the server must be stopped

2 /logs//: to view with a text editor) 򐂰 native_stderr.log1 򐂰 native_stdout.log1 administrative console: Troubleshooting → Logs and Trace →→Process Logs

Operational tasks

Start / stop an application server. /logs//: text

򐂰 SystemOut.log1 򐂰 SystemErr.log1

If using startServer

-trace, see /logs//: 򐂰 startServer.log 򐂰 stopServer.log

Start / stop a deployment /logs/dmgr/: text

manager 򐂰 SystemOut.log1 򐂰 SystemErr.log1

/logs/dmgr/: 򐂰 startServer.log 򐂰 stopServer.log

Start / stop a node agent /logs/nodeagent/ text

If using startNode -trace, see

/logs/nodeagent/ : 򐂰 startServer.log 򐂰 stopServer.log1 Configurable2 can represent the location of the profile for an application server, node agent, ordeployment manager. If the profile is for a node agent,is "nodeagent". If the profile is for adeployment manager,is "dmgr"

Chapter 2. Diagnostic data 95

Tasks Logs Format / tools

Start / stop a cluster When you start or stop a cluster,

that action is taken on each server. The logging is the same as though you started or stopped each server.

Configuration tasks

Adding a node to a cell /logs/: text

򐂰 addNode.log 򐂰 runAddNode.log 1 Configurable 2 can represent the location of the profile for an application server, node agent, or deployment manager. If the profile is for a node agent,is "nodeagent". If the profile is for a deployment manager,is "dmgr"

96 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Chapter 3. Installation problem

determination This paper discusses how to diagnose issues associated with WebSphere Application Server V6 installation. It covers actions and resolutions related to the following installation issues: 򐂰 Launchpad or installation wizard will not start 򐂰 Installation failure or hang 򐂰 Profile creation failure 򐂰 Install Verification Test (IVT) failures

The installation routine for other products, such as IBM HTTP Server and Web Server plug-ins, is separate from the application server products. This paper only covers issues that are related to WebSphere Application Server V6 installation.

Important: We recommend that you start your problem determination process

by reading Approach to Problem Determination in WebSphere Application Server V6 at http://www.redbooks.ibm.com/redpapers/pdfs/redp4073.pdf.

© Copyright IBM Corp. 2005. All rights reserved. 97

3.1 Introduction The installation process for WebSphere Application Server V6 has changed from previous versions. The main change is that only one copy of the core product files is installed. Profiles are used to define multiple server runtime environments.

The entry point into WebSphere Application Server installation is the launchpad (Figure 3-1). From here, you can choose to launch the installation wizard for WebSphere Application Server.

Figure 3-1 Launchpad for WebSphere Application Server installation

98 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Figure 3-2 provides an overview of the installation events and common issuesthat are associated with installation.

LaunchPad Displays tasks to perform

Common Problems No supported browser, Browser configuration Insufficient disk-space

Common problems Low resources like disk space, virtual memory Problems with JVM

Profile creation wizard

Create a new application server, deployment manager, custom profile

Installation verification tool (IVT)

Verify the installation

Figure 3-2 Overview of installation events, common issues, and logs

Chapter 3. Installation problem determination 99

Note: In Figure 3-2 on page 99, the Installation wizard and Profile creation wizard are shown as different entities. This is what you will see if you are installing the Network Deployment version. In a WebSphere Application Server base or Express installation, profile creation for the default profile occurs in the Installation wizard itself.

3.2 Work the problem

You begin the problem determination process by evaluating the high-level symptoms to determine if one of these symptoms describes your problem. If it does not, you need to collect the appropriate data that is required to diagnose the problem.

If you do not find your symptom listed here, go to "The next step" on page 113.

Recovering from a failed or hung installation:

Depending on the state that the system is in when an installation fails or

hangs, you might need to uninstall WebSphere Application Server manually before you retry the process. The WebSphere Information Center has detailed, platform-specific instructions for uninstalling at: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.nd.doc/info/ae/ae/tins_uninstman.html

100 WebSphere Application Server V6 Problem Determination for Distributed Platforms

3.2.1 Symptom: Launchpad or installation wizard will not start or fails This symptom covers the following situations: 򐂰 Executing the launchpad.bat command in windows or the launchpad.sh script in a UNIX environment does not start the launchpad. 򐂰 Selecting Launch the installation wizard for WebSphere Application Server from the launchpad fails to start the installation wizard or the wizard exits with or without an error message.

Possible causes for type of problem include Web browser requirements and disk space or permission requirements.

If the installation wizard starts and runs for a while but appears to hang, go to "Symptom: Installation wizard hangs" on page 103.

Chapter 3. Installation problem determination 101

If this file does not exist, then run the installer from a command window using -log option to create a log of all events. - UNIX ./install -log !logfile @ALL - Windows install -log !logfile @ALL Where logfile is a fully qualified file name for writing the log events.

As you collect the log files, copy them to a place where you can view them. If you need to recreate the problem in order to collect this documentation, be sure to prepare (schedule off-shift hours, review prerequisites, and so forth).

What to look for

Problems starting the launchpad or installation wizard can usually be traced back to missing prerequisite system or application levels. The first step in diagnosing this problem is to ensure that you have the proper prerequisites for installation. You can find this information at: http://www-306.ibm.com/software/webservers/appserv/doc/latest/prereq.html

Problems starting the launchpad or Installation wizard

If the problem you are having is in starting the launchpad, look at the IBM_WebSphere_LaunchPad_log.txt log. This log normally contains minimal messages that indicate when the launchpad was run. In the event of an error during launch, it might also contain error messages. If the launchpad does not start at all, then no log file is created.

Problems in the installation wizard

Installation events are installed in log.txt. If the installation wizard starts but fails during the installation process, look in log.txt for messages that contain INSTCONFSUCCESS, INSTCONFPARTIALSUCCESS, or INSTCONFFAILED, which might indicate the current status of the installation.

If you see the INSTCONFPARTIALSUCCESS or INSTCONFFAILED messages, then there

should be error or warning messages preceding them (for example, messages that indicate problems with resources, such as not enough disk space, exceptions in the JVM, segmentation faults, and so forth).

102 WebSphere Application Server V6 Problem Determination for Distributed Platforms

An error that resembles the following indicates a problem with disk space: A suitable JVM could not be found. Please run the program again using the option -is:javahome < JAVA HOME DIR> No space left on device

This error indicates there is not enough free space for the installer to run on. You can get this error even if enough space exists where you plan to install WebSphere Application Server (for example, drive D: or /usr) .

Verify that the location of C:\Temp or %TEMP% in Windows or /tmp directory in

UNIX has enough free space for the installer to run. You need to check the install document to determine the exact amount of temporary disk space required, typically a minimum of 100 MB.

Another option is to use -is:tempdir with the installation wizard, where tempdir is the location of a temporary directory on a partition with enough free space.

If you meet the installation prerequisites and you still have not resolved your problem, go to "The next step" on page 113 for information about how to search for known problems with installation.

3.2.2 Symptom: Installation wizard hangs

During installation, a progress indicator is displayed to show you how far the install has progressed. If there is no change in the progress indicator for a very log time, the installation process could be hung.

In a WebSphere Application Server or WebSphere Application Server - Express

installation, the installation wizard creates a default server profile. The installer first starts the file copy process and then the profile creation process. In WebSphere Application Server Network Deployment, this is a two-step process. The install completes, then you are asked if you want to create a profile.

Chapter 3. Installation problem determination 103

Data to collect If the installation hangs, check the following: 򐂰 /logs/log.txt If the installer fails at a very early stage, then this log file might not be created or it might be exist in the system temporary area, which is %TEMP%\log.txt in Windows or /tmp/log.txt in UNIX. To ensure that you get a full listing of messages in this log, run the installer from a command window using the -log option to create a log of all events. - UNIX ./install -log !logfile @ALL - Windows install -log !logfile @ALL Where logfile is a fully qualified file name for writing the log events. 򐂰 /profiles//logs/pctLog.txt This log file is created only when the profile creation wizard is executed. This log is not created when using the wasprofile command directly or during installation of the product.

What to look for

If you think the install process is hung, check the log.txt file periodically to see if progress is being made. Messages such as INSTCONFSUCCESS, INSTCONFPARTIALSUCCESS, or INSTCONFFAILED in log.txt indicate the current status of the installation. If you see the INSTCONFPARTIALSUCCESS or INSTCONFFAILED messages, then there should be error or warning messages preceding them.

Also, check other system activities, such as CPU utilization, hard disk usage, or any network activity (if installing remotely), to make sure there are not external factors that are affecting the install.

If the installation does appear to be hung, look for the last recorded message in the log file. This message gives you an idea of what the installer was doing before it hung.

104 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Determine if errors occurred in the file copy process Look in log.txt for an entry such as the following: (), Install, com.ibm.ws.install.ni.ismp.actions.ISMPConfigManagerLaunchAction, msg1, INSTCONFSUCCESS: Post-installation configuration is successful

If you see this message, the file copy process has completed successfully. If not, inspect the messages in the log for an indication of the error.

Determine if the errors occurred in the profile creation process

If the file copy was completed successfully, any error messages after this indicate problems in profile creation or with other subsequent steps, including sample application and administrative console application deployment. Look for the following entry in the log (log.txt for WebSphere Application Server and WebSphere Application Server - Express, pctLog.txt for WebSphere Application Server Network Deployment): (), Install, com.ibm.ws.install.ni.ismp.actions.ISMPWSProfileLaunchAction, msg1, INSTCONFSUCCESS: Post-installation configuration is successful

If you do not see this message, then there are problems with profile creation. To diagnose problems with profile creation, see "Symptom: Profile creation failure" on page 105.

On the other hand, if you see this message, the profile creation was successful, and it is time to re-evaluate your symptoms. Go to "The next step" on page 113.

3.2.3 Symptom: Profile creation failure

Profiles allow you to define multiple runtime environments, each with its own administrative interface while sharing the same code base. Problems with profile creation might be due to long directory paths, file permissions, problems with the host name, and so forth.

A default server profile is created as part of the installation process in a

WebSphere Application Server or WebSphere Application Server - Express installation. The Network Deployment installation wizard gives you the option of creating a profile. Profiles can also be created at any time after installation.

Chapter 3. Installation problem determination 105

Data to collect If the profile creation fails, check the following: 򐂰 /logs/wasprofile/wasprofile_create_.log This log file is created when the installation phase has completed the file copy process and starts creating a default profile. This log file is also created whenever the profile creation wizard or wasprofile command is executed. This log file traces all the events that occur during profile creation. It is an XML log file and is best viewed by a viewer that can format XML, for example a Web browser or WordPad in Windows. The entries in this log file consist ofentries. A sample log entry that indicates an error would look something similar to the following:2005-07-19T14:18:53 1121762933500 2984 com.ibm.ws.install.configmanager.ConfigManager WARNING com.ibm.ws.install.configmanager.ConfigManager executeAllActionsFound 11 Fatal configuration action failed: com.ibm.ws.install.configmanager.actionengine.ANTAction -C:\IBM\WAS\profileTemplates\managed\actions\ executeManagedProfileSetup.ant򐂰 /logs/log.txt

The following log files are created during profile creation. They are located in the /profiles//logs directory. Each log might or might not exist depending on the type of profile that is created. 򐂰 pctLog.txt Created only when the profile creation wizard is executed. This log is not created when using the wasprofile command directly or during installation of the product. 򐂰 amjrte_config.log Tivoli Access Manager configuration log for its Java Runtime Environment. 򐂰 collect_metadata.log Collects metadata information about managed objects in the system to evaluate and prevent potential installation conflicts.

106 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 createDefaultServer.log A log from wsadmin recording the creation of the server1 process in the default profile.򐂰 createshortcutforprofile.log Windows tool log for creating menu entries and shortcuts.򐂰 defaultapp_config.log JACL script log from configuring default application resources.򐂰 Service.log Start and stop events for server1.򐂰 Application installation and configuration logs for system and sample applications: - filetransfer_config.log for the file transfer application - ivt_config.log for the ivtAPP application - mejb_config.log for the ManagementEJB application - hamanager_config.log for the high availability application - query_config.log for the Query application - samples_config.log for the PlantsByWebSphere sample application - samples_install.log for the SamplesGallery and PlantsByWebSphere sample applications - scheduler.cal_config.log for the SchedulerCalendars application - webui_config.log for the administrative console application - defaultapp_deploy.log for the DefaultApplication application򐂰 SIBDefineChains.log Creation log for service integration bus endpoints, inbound channels and channel chains, outbound thread pool, and outbound channel and channel chains.򐂰 SIBDeployRA.log Deployment log for the service integration bus function.򐂰 winservice_config.log Service log for the Windows service created for server1.򐂰 addNode.log Log for federating a node to a cell (Custom profile).

What to look for

Messages such as INSTCONFSUCCESS, INSTCONFPARTIALSUCCESS, orINSTCONFFAILED in log.txt or pctLog.txt indicate the current status of the

Chapter 3. Installation problem determination 107

installation. If you see the INSTCONFPARTIALSUCCESS or INSTCONFFAILED messages, then there should be error or warning messages preceding them.

Look for an entry such as the following:

(), Install, com.ibm.ws.install.ni.ismp.actions.ISMPWSProfileLaunchAction, err, INSTCONFFAILED: Cannot complete required configuration actions after installation. The configuration failed. The installation is not successful. Refer to \install_root\logs\wasprofile\wasprofile_create_profilename.log for more details

If you see this entry, look at the wasprofile_create_.log to try to

determine what task was being performed when the profile creation failed. Most tasks, such as system or sample application installation, are logged to individual log files in the /profiles//logs directory. If you can determine which task the profile creation was doing, collect the file for that task. For example, if you had problems creating a custom node then look at the log file /profiles//logs/addNode.log for any errors.

For information about common problems, see "Profile creation problems" on

3.2.4 Symptom: IVT fails

The IVT looks at the profile configuration for the server and looks for a server running on the server port number. Note that if a server is up and running on that port, the IVT runs against that server, even if it is not the one you just installed. If no server is running on that port, the IVT attempts to start the server. When the server has started successfully, the IVT accesses the server and runs various tests, including servlet engine verification, JSP verification, EJB verification, and so forth.

The option to run the IVT is displayed in the First Steps console after the installation completes and after every profile creation, with the exception of a custom profile creation because no server is actually created.

You should run the IVT before making any configuration changes to WebSphere Application Server. This acts as a checkpoint to see if any problems exist as the result of the installation. If the IVT runs clean and problems show up later, they

108 WebSphere Application Server V6 Problem Determination for Distributed Platforms

are most likely due to configuration changes done after the installation. It isrecommended that all instances of WebSphere Application Server be stoppedbefore running the IVT.

A failure in the IVT is usually because the application server fails to start.Common reasons for this include port conflicts, not enough memory, and soforth.

What to look for

The first file to look at in case IVT has failed is ivtClient.log. Messages with IVTLand ADMU prefixes provide information about what the IVT application is doingand the status of each action.

Look for the following message in ivtClient.log:

ADMUXXXXX: Server servername open for e-business; process id is xxxx

If you find this message, then the server has started successfully, and the IVTfailed while executing one of the tests. Examine the error messages inivtClient.log to locate the failing process.

If the server does not start, look at the Server Port number is: entry. This entrycontains the server port number of the profile instance. Make sure that this port isnot in use (see "Application server startup problems" on page 111).

Chapter 3. Installation problem determination 109

If the port does not seem to be the problem, look at startServer.log, SystemErr.log, and SystemOut.log for information related to server startup, such as the following: 򐂰 Look for any error messages starting with WSVR (server runtime) or ADMU (management utility) in the startServer.log and SystemOut.log. 򐂰 Look for error messages in SystemErr.log. These messages will start with: [Date and time stamp] 0000000a SystemErr

If you cannot identify the problem, go to "The next step" on page 113.

3.3 Analyzing problem areas

Your analysis of the data that you gathered will most likely lead you to one of the following areas. If not, see "The next step" on page 113.

3.3.1 Web browser requirements

If you are having problems in starting the launchpad, it is possible that you do not have a supported Web browser installed or configured properly. If this is the case: 򐂰 Verify that the latest version of a supported Web browser is installed. Mozilla and Internet Explorer are supported for configuration and installation activities. 򐂰 On UNIX platforms, ensure that the location of the supported browser is exported. For example if the Mozilla executable is located in the /usr/bin directory, then export its location as follows: export BROWSER=/usr/bin/mozilla 򐂰 Ensure that JavaScript™ is enabled in the browser options or preferences. For example: - In Mozilla, select Edit → Preferences → Advanced → Scripts & Plugins - Enable JavaScript for: Navigator, Allow Scripts to: (select all boxes) - In Internet Explorer, select Tools → Internet Options → Security → Custom Level for Internet Scripting → Active scripting → Enable

The following link gives specific information about system readiness for installing WebSphere Application Server on RedHat Linux Enterprise Edition 4: http://www-1.ibm.com/support/docview.wss?uid=swg21201306

110 WebSphere Application Server V6 Problem Determination for Distributed Platforms

3.3.2 Application server startup problems During profile creation, it is important that the port numbers that are specified for each of the WebSphere Application Server services are unique. The Profile creation wizard, by default, tries to define port numbers that do not conflict with other profiles in the installation. However, it is also possible that there are other applications using the same port numbers (for example, older versions of WebSphere Application Server in a co-existence environment or the unit test environment of Rational Application Developer).

If an application server does not start during the IVT process, or later using the startServer command, look for the following message in startServer.log: ADMYUXXXXX: an instance of the server may already be running: server1

This message indicate that another instance of the server is already running or another process is using the same port number.

To resolve the problem, do the following:

򐂰 Use the serverStatus command to determine if the server instance that you are trying to start is already started. 򐂰 Review the ports that are required by the application server process. You can see these in: /profiles//config/cells//nodes// serverindex.xml 򐂰 List the ports that are currently being used by the system. One way of doing this is by issuing a netstat -a command in a command prompt window. If you have a port conflict, update the serverindex.xml file to use a non-conflicting port number.

Chapter 3. Installation problem determination 111

3.3.3 Profile creation problems The following are common problems with profile creation.

File path length errors

Windows 2000 has a length restriction of 258 characters for a command. A problem can occur that prevents the successful creation of a profile when a path is too long. The maximum length for thedirectory is 60 characters. The maximum length for the profiles installation root directory is 80 characters.

If your log files have errors similar to Input line is too long then the length of the file path and node name in the command string has caused the entire command to exceed the operating system limit for command length.

This error can show up in a message box during the wizard or if using wasprofile, in /profiles//logs/collect_metadata.log.

Create the profile again using shorter directory paths and node names. If you are still in the installation process, consider re-installing using a shorter path for the installation root.

Host name error

If you see errors similar to localhost is not a valid host name for remote access, then you have entered localhost as the value in the host name field of the Profile creation wizard. Other machines in the network cannot reach your node using localhost, so you must provide a host name that can be resolved by other systems to your system's IP address.

Template path error

If in a UNIX environment you get an error similar to the following then verify the profile templates location and permissions are correct: Incoming command line is: { "-create" ,"-help" ,"-templatesPath" ,"/opt/WebSphere/AppServer/crso/profileTemplates/managed" } Could not resolve templatePath from command line

Custom profile error

If an error occurred creating a custom profile, look at the addNode.log file for any additional errors. Look for an error message similar to the following: [Date and Time stamp] 0000000a AbstractNodeC E ADMU0006E: Exception creating Deployment Manager connection: com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host hostname at port 8879

112 WebSphere Application Server V6 Problem Determination for Distributed Platforms

If you see this message, it is likely that the deployment manager profile is not running. Ensure that the deployment manager profile is created and running on the specified port and host before creation of custom profile.

If neither of these messages appears and you do not see any other obvious problems, go to "The next step" on page 113.

3.4 The next step

The symptoms and problem areas included in this paper are some that you are more likely to experience. However, there are other things that can go wrong during installation.

Review the problem classifications to see if there are any other components that might be causing the problem.

Look also at the WebSphere Information Center documentation for additional

resources for diagnosing and fixing issues: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/rtrb_allrfl.html

Chapter 3. Installation problem determination 113

114 WebSphere Application Server V6 Problem Determination for Distributed Platforms 4

Chapter 4. System management

problem determination This paper discusses techniques for determining the cause of problems with WebSphere Application Server system management. The symptoms that this paper discusses are: 򐂰 You are not able to access the administrative console. 򐂰 You cannot access server processes using wsadmin or the management scripts such as stopServer. 򐂰 You are getting errors performing system management functions (for example, managing application servers, node agents, Web servers, or applications). 򐂰 You are not able to federate a node with a deployment manager. 򐂰 You are getting save conflict messages in the administrative console. 򐂰 Your enterprise applications no longer appear in the administrative console. 򐂰 You have problems with WebSphere Rapid Deployment. 򐂰 You are having trouble communicating using SSL and are getting messages that indicate your certificates have expired.

Important: We recommend that you start your problem determination process

by reading Approach to Problem Determination in WebSphere Application Server V6 at http://www.redbooks.ibm.com/redpapers/pdfs/redp4073.pdf.

© Copyright IBM Corp. 2005. All rights reserved. 115

4.1 Introduction In this paper, we look at resolving problems that occur during system management activities. WebSphere Application Server configuration is maintained in XML files. The XML files are maintained in a directory structure that reflects the topology of your WebSphere Application Server installation (Figure 4-1).

config repository

Admin console

wsadmin

WebSphere WebSphere server server

Figure 4-1 Overview of system administration

The WebSphere administrative console and the wsadmin command line

administration tool read, maintain, and modify the contents of the XML files.

We strongly recommend that you back up the config repository on a regular

basis. You can either back up the repository manually via a file system backup, or you can use the WebSphere Application Server supplied tool backupConfig. This script creates a zipped file that contains all the configuration files that you might need to restore the WebSphere Application Server configuration.

Tip: Editing the XML files in the repository directly is not supported and can lead to unexpected results.

116 WebSphere Application Server V6 Problem Determination for Distributed Platforms

At the most basic level, that is a stand-alone installation of a single WebSphereApplication Server, the config directory is as shown in Figure 4-2. The XML filesshown contain the data that gets formatted by the administrative console anddisplayed in the various boxes and text fields in the browser.

Figure 4-2 The config directories and files

The administrative console and other management tools call management beans(MBeans) in the WebSphere Application Server process. MBeans are used byWebSphere Application Server to perform system management functions.

In a Network Deployment environment, the deployment manager maintains the

master repository for all of the WebSphere Application Server nodes and serversthat it manages in the cell. Copies of the files that each node needs arereplicated to that node by a process known as synchronization.

Chapter 4. System management problem determination 117

D e p lo ym e n t M a s te r M anager R e p o s ito ry

C e ll

Node Agent Node Agent

Node01 Node02

Node01 Node02 R e p o s ito ry R e p o s ito ry

Figure 4-3 Replicated admin repository

Only changed files are synchronized, so even if your system is performing

automatic synchronization every minute (the default interval), this will have a minimal performance impact because the node agent will connect to the deployment manager and be told that nothing has changed.

You can manually initiate synchronization using the administrative console by

selecting System administration → Nodes, putting a check by the node that you wish to synchronize, and then clicking either Synchronize or Full Resynchronize.

Choosing Synchronize only synchronizes changes that the administrative

console is aware of using the normal synchronization mechanism. Choosing Full Resynchronize synchronizes all changes that have been made to the master repository by whatever means. For example, if you were to edit the master copy of the server.xml file with a text editor, clicking Synchronize does not replicate this file to the node. If you click Full Resynchronize, then your updates are replicated.

118 WebSphere Application Server V6 Problem Determination for Distributed Platforms

You can also perform synchronization from the node agent using thesyncNode.bat|sh script. You must stop the node agent to use this tool. You canpass parameters to the script to log messages to a specified file or to trace thesynchronization.

Note: You can also perform synchronization using the wsadmin tool. See the WebSphere Information Center for further details.

Tip: Synchronization does not protect against losing your entire replicated repository. The node agent configuration files must exist for synchronization to work. It is not a replacement for a comprehensive backup policy.

Application developers can quickly create a new clean profile to test an

application and then remove that profile when testing is complete.

Profiles are created either by the graphical profile creation wizard or thewasprofile command and are managed via the wasprofile command.

A user registry contains the names, groups memberships, roles, and passwordsof users in an organization. At the most basic level, most operating systems

Chapter 4. System management problem determination 119

contain a user registry so that users can logon with a user name and password and have access to operating system resources. At the more complex level, a large distributed organization might maintain a complete list of users including their passwords, personal information, group memberships, and job roles in a distributed LDAP repository.

If you have multiple installations of WebSphere Application Server on multiple

servers that you need to secure, then you need to use an LDAP registry so that the same registry is available to all servers.

You can encode the passwords in these files to prevent them from being read easily by using the password encoder utility: /bin/PropFilePasswordEncoder.bat|sh

When security is enabled, WebSphere Application Server uses SSL for

communication from the browser to the administrative console and for other administrative communications. SSL communication relies on a key database file and password. WebSphere Application Server comes with a dummy key database file that is secured with the password WebAS. We highly recommend

120 WebSphere Application Server V6 Problem Determination for Distributed Platforms

that you create your own key database file for production servers. Refer to theWebSphere Information Center for details on customizing the key database:http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/tsec_preparetruststorefile.html

Again, you can encode the passwords in these files.

WebSphere Rapid Deployment

WebSphere Rapid Deployment is a new feature in WebSphere ApplicationServer V6 that simplifies and speeds up the process of deploying enterpriseapplications or their components for development purposes. WebSphere RapidDeployment is a cut-down version of the Eclipse IDE and runs in batch mode.

WebSphere Rapid Deployment is a foreground process that monitors a chosen

directory. If the developer copies an enterprise application EAR file to thatlocation, WebSphere Rapid Deployment detects the new EAR and automaticallydeploys it to the WebSphere Application Server. Figure 4-4 shows theWebSphere Rapid Deployment components.

Figure 4-4 WebSphere Rapid Deployment components

Chapter 4. System management problem determination 121

WebSphere Rapid Deployment is only intended for development environments to simplify the process of testing updates to applications. It is not intended for production environments where you do not want production code updated easily. For that reason, the WebSphere Rapid Deployment process only runs in the foreground.

4.1.1 Collecting data

You need to collect the following log files and data for system management problem determination: 򐂰 WebSphere Application Server logs 򐂰 wsadmin output 򐂰 Management script logs 򐂰 Profile management logs 򐂰 WebSphere Rapid Development logs

If the problem is difficult to recreate or disruptive to business operations, see

"The next step" on page 162 for a complete list of documentation to collect before continuing. In particular, you should review the MustGather documents for a complete list of documentation that is required by IBM support.

WebSphere Application Server logs

You need to look at the WebSphere Application Server log files when diagnosing system management problems.

122 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Application serverWith some problem types, you might also need to look at the logs for theapplication server that you are trying to manage. These log files are:/profiles//logs//SystemOut.log/profiles//logs//SystemErr.log

System management trace

Most system management problems can be traced by setting the following tracestring:Admin=all

If you cannot connect to the WebSphere administrative console or wsadmin, you

can set tracing by updating the server.xml file for the application server that youneed to trace. You can find the server.xml file under the server's entry in theconfiguration file directory structure:/profiles//config/cells//nodes//servers//server.xml

Chapter 4. System management problem determination 123

Open this file in a text editor and update the services tag, changing the startupTraceSpecification to the trace string that you require, as shown in Example 4-1.

Example 4-1 Setting the trace string in server.xml

You need to restart the server for this to take effect.

wsadmin output Messages from wsadmin are written to the wsadmin.traceout log file: /profiles//logs/wsadmin.traceout

You can also increase the amount of data that is logged to this file by tracing the wsadmin utility. Update the following file: /properties/wsadmin.properties

Uncomment the line:

com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled

Note that the information that is logged is of limited use because wsadmin calls MBeans in the application server that is running the administrative console application. You usually need to trace the application server as well.

Management script logs

WebSphere Application Server services can be managed using the supplied management scripts. For example, each WebSphere Application Server installation has a script to start an application server, a script to stop an application server, and a script to show you the status of all application servers that are defined in a profile. Each of these scripts writes its own log file into the server's logs directory. For example, the stopServer script writes stopServer.log into the logs directory:

/profiles//logs//stopServer.log

Profile management logs

The profile creation and management tool wasprofile writes messages to the profile independent logs directory, that is: /logs/wasprofile/.log

This log file is in XML format.

124 WebSphere Application Server V6 Problem Determination for Distributed Platforms

The Java graphical interface that is used to create a profile simply calls the wasprofile command after collecting the information that is needed. By default, it does not write a log. However, you can pass it a log parameter as shown: pctWindows -is:log c:\temp\pct.log

WebSphere Rapid Development logs

The WebSphere Rapid Deployment tool works on a directory that you create and pass to WebSphere Rapid Deployment in the WORKSPACE environment variable. It logs Eclipse messages into two separate files within this directory: 򐂰 /.metadata/.log 򐂰 /project/.metadata/.log

Similar to other WebSphere Application Server utilities, WebSphere Rapid

Deployment calls MBeans on the application server. The application server logs can help you resolve a problem with WebSphere Rapid Deployment. There is no way to trace the WebSphere Rapid Deployment utility. However, you can trace the application server as described in "WebSphere Application Server logs" on page 122.

Note: In a UNIX environment, files and directories starting with a period (.) are hidden. You need to pass the -a parameter to the ls command to see them.

4.2 Work the problem

You begin the problem determination process by evaluating the high level symptoms to determine if one of these describes your problem. If it does not, you collect the appropriate data that is required to diagnose the problem.

4.2.1 High-level symptom analysis

Let's start working the problem: 򐂰 If you are not able to access the WebSphere administrative console, go to "Problem: Unable to access the administrative console" on page 127. 򐂰 If you are using wsadmin or the management scripts such as stopServer, but these admin tools cannot access the server process, go to "Problem: wsadmin or management scripts can't access server" on page 130.

Chapter 4. System management problem determination 125

򐂰 If you are trying to perform administrative tasks against WebSphere Application Server processes (such as starting or stopping a server), consider the following situations: - If you are unable to stop an application server, go to "Problem: Unable to stop a server process" on page 133. - If you are unable to start an application server, go to "Problem: Unable to start a server process" on page 135. - If the deployment manager is not communicating with a node agent, go to "Problem: Unable to access a node agent" on page 137. - If you are unable to manage a Web server from the WebSphere administrative tools, go to "Problem: Unable to manage a Web server" on page 138. - If you are unable to manage (start, stop, install) an application, go to "Problem: Unable to manage applications" on page 141. 򐂰 If you are unable to federate a node with a deployment manager, go to "Problem: Failure adding a node to a deployment manager" on page 143. 򐂰 If you are making changes in the administrative console and getting messages warning you of a save conflict, go to "Problem: Save conflicts in the administrative console" on page 150. 򐂰 If enterprise applications no longer appear in the administrative console go to "Problem: enterprise applications missing" on page 151. 򐂰 If you are having a problem communicating with your system via SSL or are receiving messages that your certificates have expired, go to "Problem: Invalid or expired certificates" on page 153 򐂰 If you are having problems with WebSphere Rapid Deployment, go to "Problem: WebSphere Rapid Deployment" on page 158. 򐂰 If you cannot create, delete, or update WebSphere Application Server profiles, see WebSphere Application Server V6: Installation Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4068.pdf

Where to go from here: If, after finding your symptom and going to the appropriate section, or if none of these symptoms resemble your problem, go to "The next step" on page 162.

126 WebSphere Application Server V6 Problem Determination for Distributed Platforms

4.3 Analyzing problem areas Your analysis of the data that you gathered will most likely lead you to one of the following problem areas. If not, see "The next step" on page 162.

4.3.1 Problem: Unable to access the administrative console

You are trying to access the WebSphere administrative console and the login page is not being displayed. Instead, you see a blank page or an HTTP 404 error, page not found error.

You experience this problem immediately after installing WebSphere Application

Server, after enabling security, after adding a node to the deployment manager, or after restarting the WebSphere Application Server process.

In a Network Deployment environment, the process will run in the deployment

manager. In a stand-alone server environment, the process will run in the server (server1).

You might also need to take an admin trace and a security trace, depending on the problem. See "Collecting data" on page 122 for details.

What to look for

This problem can occur because the server that hosts the administrative console is not running, or you are trying to connect to the wrong port number. It can also occur if the administrative console files have been accidently deleted or security is incorrectly configured.

Verify the port number and URL

By default, access to the administrative console is via the following URL: http://:9060/ibm/console

However, the port is configurable, and if multiple profiles have been installed on a machine, the port might be different from the default.

Review the WebSphere Application Server SystemOut.log to ensure that the

administrative console has started. You will see messages similar to those in Example 4-2 on page 128, showing you that the administrative console has started and on which port the administrative console is listening.

Chapter 4. System management problem determination 127

Example 4-2 Administrative console startup messages [6/22/05 9:58:31:342 EDT] 0000001b ApplicationMg A WSVR0221I: Application started: adminconsole [6/22/05 9:58:31:492 EDT] 0000000a TCPChannel A TCPC0001I: TCP Channel TCP_1 is listening on host * (IPv4) port 9060.

Figure 4-5 Administrative console port number

Ensure that you are connecting to the correct server on the correct port number, for example: http://localhost:9060/ibm/console

Verify that the adminconsole application has started

If the administrative console fails to start, you will see errors similar to those shown in Example 4-3.

Example 4-3 Administrative console failure messages

[7/3/05 15:10:29:359 EDT] 0000000a DeployedAppli E WSVR0100W: An error occurred initializing,adminconsolejava.io.FileNotFoundException: C:\IBM\WAS6\AppServer\systemApps\adminconsole.ear\deployment.xml(The system cannot find the path specified)at java.io.FileInputStream.open(Native Method)

128 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Note: In WebSphere Application Server V6, the adminconsole.ear and filetransfer.ear applications no longer appear as installed applications and cannot be managed in the same way as other applications. This is to prevent them from being uninstalled accidently.

Verify the security configuration

You might not be able to access the administrative console application if securityis enabled due to a problem with the digital certificates that are used to setup anSSL communication between your browser and WebSphere Application Server.If this is the case, you would be able to access other applications on theapplication server that did not use SSL without any problems. Refer to "Problem:Invalid or expired certificates" on page 153 for details.

If your problem is not resolved by fixing invalid certificates, you can confirm that itis related to security by turning security off and retesting.

You can find the security.xml file in the configuration directory structure under: /profiles//config/cells//security.xml

You need to restart the server for this change to take effect.

Chapter 4. System management problem determination 129

4.3.2 Problem: wsadmin or management scripts can't access server You are trying to use wsadmin to manage your system, but you are unable to connect to an application server process or are getting no response from wsadmin.

You have this problem when trying to use the supplied management scripts, such as stopServer.bat or serverStatus.bat.

You experience this problem immediately after installing WebSphere Application

Server, after enabling security, after adding a node to the deployment manager, or after restarting the WebSphere Application Server server.

If you are trying to manage a service using a management script, such as

stopServer.bat, then you need the log file from that script (stopServer.log) and possibly a trace of that script.

What to look for

This problem can occur because the application server to which you are trying to connect is not running, or you are trying to connect to the wrong port number.

The wsadmin utility connects to an application server in a stand-alone

environment or deployment manager in a distributed environment. The following error shows that no connection could be established: WASX7023E: Error creating "SOAP" connection to host "hostname"

Make sure that the application server is running and that you are using the correct port number to connect to the application server or deployment manager.

Verify the port number

You can determine to which server wsadmin is trying to connect and on what port by looking at the wsadmin.properties file. This file is in the profile directory structure: /profiles//properties/wsadmin.properties

130 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Example 4-4 shows the host name and port number that wsadmin uses bydefault.

Example 4-4 SOAP host and port for a stand-alone server

com.ibm.ws.scripting.host=localhostcom.ibm.ws.scripting.port=8879

You can compare this to the entry in the SystemOut.log from the applicationserver that tells you on which port the application server is actually listening forSOAP connections, as shown in Example 4-5.

Example 4-5 SOAP connector port number from the SystemOut.log

[7/8/05 14:19:55:445 EDT] 0000000a JMXSoapAdapte A ADMC0013I: The SOAPconnector is available at port 8880

Alternatively, you can pass the port number to wsadmin with the -portparameter:wsadmin -host kll6571 -port 8880

In a Network Deployment installation, wsadmin attempts to send requests to the

deployment manager. Again, the wsadmin.properties file contains the host nameand port number of the deployment manager process. Using the -host and -portparameters, you can use wsadmin to connect directly to a process other than thedeployment manager. For example you could connect directly to a node agent orremote application server. Example 4-6 shows a connection to a node agent andthen issuing a stopServer command.

Example 4-6 Connecting directly to a node agent using wsadmin

$ ./wsadmin.bat -host m23vnx60.itso.ral.ibm.com -port 8881WASX7209I: Connected to process "nodeagent" on node m23vnx60Node01 using SOAPconnector; The type of process is: NodeAgentWASX7029I: For help, enter: "$Help help"wsadmin>$AdminControl stopServer nodeagentWASX7337I: Invoked stop for server "nodeagent" Waiting for stop completion.

Chapter 4. System management problem determination 131

Note: In Example 4-6, you will not receive any further response because the server shuts down. Use Ctrl+c to exit the utility.

Verify that the application server is running

You can use the netstat command to ensure that a process is listening on the port. Example 4-7 shows the output of the netstat command that has been filtered with the grep tool to only show lines containing the string 8880.

Example 4-7 netstat output

C:\IBM\WAS6\AppServer\profiles\StandAlone\bin>netstat -an|grep 8880 TCP 0.0.0.0:8880 0.0.0.0:0 LISTENING TCP 9.42.171.145:8880 9.42.171.145:1114 CLOSE_WAIT TCP 127.0.0.1:8880 127.0.0.1:1122 CLOSE_WAIT TCP 127.0.0.1:8880 127.0.0.1:1161 CLOSE_WAIT

Note: The netstat command is available on both UNIX and Windows. The grep command is available on UNIX by default, and you can download free copies of grep for Windows.

You can also use the telnet command to try and get a connection to the host name and port as shown in Example 4-8.

Example 4-8 Using to telnet to test a port

C:\IBM\WAS6\AppServer\profiles\StandAlone\bin>telnet localhost 8880 Connecting To localhost...Could not open a connection to host on port 8880 : Connect failed

Note that even if you can connect, you will not be able to do anything because WebSphere Application Server does not respond to telnet commands. You can close your telnet session by pressing Ctrl+c.

If telnet is not able to connect, then the server is not started or not accepting any form of connection. Try restarting the application server. You probably need to kill the Java process because it will not accept the stopServer command.

132 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Verify the security configuration If security is enabled, then you need to pass a user name and password to wsadmin in order to connect. If you do not pass a user name or password, or get them wrong, you get an error connecting, as shown in Example 4-9

Example 4-9 Incorrect user name or password

WASX7246E: Cannot establish "SOAP" connection to host "localhost" because of an authentication failure. Ensure that user and password are correct on the command line or in a properties file. Exception message (if any): "ADMN0022E: Access is denied for the getProcessType operation on Server MBean because of insufficient or empty credentials."

You might not be able to connect to the application server or deployment

manager if security is enabled, due to a problem with the digital certificates that are used to setup an SSL communication. See "Problem: Invalid or expired certificates" on page 153 for details.

4.3.3 Problem: Unable to stop a server process

You are trying to stop an application server through the administrative console but are getting failure messages.

If you are trying to manage a server using a management script, such as

stopServer.bat, then you need the log file from that script (stopServer.log) and possibly a trace of that script.

Chapter 4. System management problem determination 133

What to look for Figure 4-6 shows the result that you would see in the administrative console if you tried to stop an application server that had hung. It takes some time for the stop request to time out.

Figure 4-6 Administrative console error stopping an application server

In this case, the SystemOut.log for the deployment manager will most likely not contain any relevant messages. However, in the SystemOut.log for the node agent, you might see messages similar to those that are shown in Example 4-10.

Example 4-10 Node agent error stopping an application server

[7/11/05 13:51:54:207 EDT] 0000000d ThreadMonitor W WSVR0605W: Thread"SoapConnectorThreadPool : 6" (00000032) has been active for 607027 milliseconds and may behung. There is/are 1 thread(s) in total in the server that may be hung.

In this example, the problem is that the application server is hung and cannot respond to requests. You might also see a situation in which the node agent is hung, as shown in Example 4-11.

Example 4-11 Hung node agent error messages

[7/11/05 14:19:00:918 EDT] 0000000f ThreadMonitor W WSVR0605W: Thread "WebContainer : 1"(00000038) has been active for 619290 milliseconds and may be hung.There is/are 1 thread(s) in total in the server that may be hung.

In the case where the node agent is hung, the administrative console will eventually time out after at least 10 minutes. You will see a message in the deployment manager's SystemOut.log before the time out.

Note: In the Windows Task Manager, the process ID is not shown by default. However you can add the process ID (PID) using the View → Select Columns option.

134 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Stopping a hung process is simply a work-around. The real problem is the hang.

4.3.4 Problem: Unable to start a server process

If you are unable to start an application server, the problem is likely to be either a configuration or an application error. Both the administrative console and the startServer script output will return an error telling you where to look.

Note: The steps outlined here are also true for a deployment manager or node agent.

Data to collect If you are trying to start the server through the administrative console, open a command window and try to start the server using the startServer command.

If these logs do not make clear the cause of the problem, you might also need to trace the script.

What to look for

When a server fails to start, you need to review the output from the startServer command and the logs for that application server for messages that tell you at what point the server failed.

Example 4-12 shows the output from a startServer command that indicates why a server will not start.

Example 4-12 Unable to start a server from the command line

C:\IBM\WAS6\AppServer\profiles\StandAlone\bin>startserver server1 ADMU7701I: Because server1 is registered to run as a Windows Service, the request to start this server will be completed by starting the associated Windows Service. ADMU0116I: Tool information is being logged in file

Chapter 4. System management problem determination 135

the server server1 is already running b) some other process is using port 8880 ADMU3027E: An instance of the server may already be running: server1 ADMU0111E: Program exiting with error: com.ibm.websphere.management.exception.AdminException: ADMU3027E: An instance of the server may already be running: server1 ADMU1211I: To obtain a full trace of the failure, use the -trace option. ADMU0211I: Error details may be seen in the file:

C:\IBM\WAS6\AppServer/profiles/StandAlone\logs\server1\startServer.log

In Example 4-12, the output tells us that some other process is using one of the ports that this server needs in order to start. You can confirm that the port is in use with the netstat command.

Example 4-13 shows that some process is listening on port 8880. In UNIX, you can determine what process is listening on that port using the lsof command.

Example 4-13 Output from the netstat command

TCP 9.42.171.145:8880 9.42.171.145:2698 TIME_WAIT TCP 9.42.171.145:8880 9.42.171.145:2739 TIME_WAIT

Note: The netstat command is available on both UNIX and Windows.

Example 4-14 shows the output from the lsof command. This output tells you that a Java process is listening on TCP port 8880. The process ID is 12968.

Example 4-14 Using lsof to check for open ports

[root@m23vnx60 root]# lsof -i|grep 8880 java 12968 wasadmin 135u IPv4 655014 TCP *:8880 (LISTEN)

Note: There are Windows equivalents of the lsof command available for download from the Internet.

136 WebSphere Application Server V6 Problem Determination for Distributed Platforms

By checking the server logs, you can determine if the server has actually started.

Example 4-15 Starting a server loses contact

C:\IBM\WAS6\AppServer\profiles\StandAlone\bin>startserver server1 ADMU7701I: Because server1 is registered to run as a Windows Service, the request to start this server will be completed by starting the associated Windows Service. ADMU0116I: Tool information is being logged in file

You might be able to resolve this by setting the SOAP timeout parameter in the following file: /profiles//properties/soap.client.props

4.3.5 Problem: Unable to access a node agent

You have recently restarted the deployment manager and discovered that one of your node agents does not appear to be available.

Chapter 4. System management problem determination 137

However, depending on the cause of the problem, you might or might not see any helpful messages in the logs.

What to look for

Check the node agent to ensure that it is actually running by looking for its process and checking its SystemOut.log. You could also ensure that it is running by trying to connect to the node agent directly with wsadmin as shown in Example 4-6 on page 131.

If the node agent is running and the deployment manager is not reporting any errors but is still saying that the node agent is down, it is possible that a DNS error occurred as the deployment manager was starting.

4.3.6 Problem: Unable to manage a Web server

WebSphere Application Server V6 has the ability to manage the IBM HTTP Server V6 from within the administrative console.

You are having problems starting, stopping, or getting the status of the Web server.

What to look for

You can start and stop any of the supported Web servers using the WebSphere administrative console. The Web server must be installed on a managed node. A WebSphere Application Server node agent executes the appropriate start or stop command.

138 WebSphere Application Server V6 Problem Determination for Distributed Platforms

When you are using IBM HTTP Server V6.0, you do not need to install the Webserver on a managed node because administration can be performed throughthe IBM HTTP Administration Server. In addition to starting and stopping theIBM HTTP Server V6.0, you can also manage its configuration file and view theWeb server logs.

IBM HTTP Server on an unmanaged node

If you cannot administer the IBM HTTP Server from the WebSphereadministrative console, verify that the IBM HTTP Server administration serverhas been properly set up.

Automatically using MD5 format.

New password: ****** Re-type new password: ****** Adding password for user webadmin

Chapter 4. System management problem determination 139

When you are managing an IBM HTTP Server using the WebSphere administrative console, you must ensure that the following conditions are met: 򐂰 Verify that the IBM HTTP Server administration server is running. 򐂰 Verify that the Web server host name and port that are defined in the WebSphere administrative console match the IBM HTTP Server administration host name and port. 򐂰 Verify that the firewall is not preventing you from accessing the IBM HTTP Server administration server from the WebSphere administrative console. 򐂰 If you are trying to administer the IBM HTTP Server over a secure SSL connection, verify that you have exported the IBM HTTP Server administration server key database personal certificate and imported it into the WebSphere key database as a signer certificate. The key database will be identified in the com.ibm.ssl.trustStore parameter in the sas.client.props file. 򐂰 Verify the IBM HTTP Server admin_error.log file and the WebSphere Application Server logs (trace.log) do not contain any errors.

IBM HTTP Server on a managed node

When the IBM HTTP Server on a managed node is started using WebSphere administration tools. The node agent issues the same command that you would use from a command line, that is: /bin/apachectl

The deployment manager does not log any messages when you try to start a Web server. The node agent logs a simple message to tell you that it is trying to start the Web server as shown in Example 4-16.

If the Web server fails to start, then you get a failure message in the administrative console and a message in the node agent log, as shown in Example 4-16.

Example 4-16 Node agent messages starting a Web server

[7/12/05 14:56:15:658 EDT] 000000a6 AdminHelper A ADMN1001I: An attempt is made to launchwebserver1 on node m23vnx60Node01.[7/12/05 14:56:15:661 EDT] 000000a6 NodeAgent W ADML0065W: A sync operation prior tostarting an application server failed.

140 WebSphere Application Server V6 Problem Determination for Distributed Platforms

[7/12/05 14:56:22:843 EDT] 000000a6 NodeAgent W ADML0040E: The process timed out waitingfor server "webserver1" initialization: 1200 seconds

You need to review the Web server logs to determine what went wrong.

Also, you can trace both the deployment manager and node agent process, as described in "Collecting data" on page 122, to gather further information about the source of a failure.

4.3.7 Problem: Unable to manage applications

You are trying to manage applications, for example starting, stopping, or installing an application, and are having problems.

You get these problems while installing an application, after installing an

application, or after a server restart.

What to look for

When an error occurs when you are trying to manage an application, the following are things you can do: 򐂰 Verify that there are no resource or classpath errors 򐂰 Verify that the application was installed correctly 򐂰 Verify that the configuration repository is not corrupted

Verify that there are no resource or classpath errors

The application server log files are the first place to look as any problems with the application itself, such as classpath issues or resource issues, are reported there. The error message from the administrative console tells you which node and server to check, as shown in Figure 4-7 on page 142. Should you find such an error, work with your application developers to resolve the application coding or resource problem.

Verify that the application was installed correctly

If the application fails to start on a server after it has been installed, the installation might not have completed successfully. The installation might have

Chapter 4. System management problem determination 141

appeared to be successful, but when you try to start the application, it fails with a message in the administrative console as shown in Figure 4-7 on page 142.

Figure 4-7 Message from administrative console when application fails to start

This message also appears in the deployment manager SystemOut.log, as

shown in Example 4-17.

The application server SystemOut.log shows a similar message, as shown in

Example 4-18.

Example 4-18 Message in application server log when application fails to start[7/12/05 9:44:07:379 EDT] 00000039 ApplicationMg W WSVR0215W: Starting application, HelloApp,failed. The application is not installed.

Verify that the configuration repository is not corrupted

Corruptions in the configuration repository can also cause enterprise applications to not start. For example, a node's serverindex.xml file contains a listing of all servers and ports in a cell with which the node might need to communicate. It also lists applications that are deployed on the servers in those nodes.

142 WebSphere Application Server V6 Problem Determination for Distributed Platforms

If this file becomes corrupt and the list of applications is lost, then the applications will not be started on the server in this node. When the server process starts, it thinks that the application is not installed and does not try and start it. If you try and start the application from the administrative console, you see the message that is shown in Figure 4-7 on page 142. The application server logs tell you that the application is not installed, as shown in Example 4-18 on page 142.

Then in the administrative console, navigate to System Administration →

Nodes. Select the node and choose Full Resynchronize, as shown in Figure 4-8.

Figure 4-8 Full synchronization

4.3.8 Problem: Failure adding a node to a deployment manager

A WebSphere Application Server node is federated to a deployment manager using the addNode command. This can sometimes fail.

Chapter 4. System management problem determination 143

Data to collect The following logs can be helpful in determining why you cannot add a node to a deployment manager: 򐂰 Management script log file: addNode.log 򐂰 Deployment manager logs

Depending on the cause of the failure, you might also need to look at the logs on the deployment manager.

What to look for

Example 4-19 shows that the connection from the addNode script to the deployment manager timed out before the work completed.

Example 4-19 Errors from the addNode script

ADMU0116I: Tool information is being logged in file /opt/IBM/WAS6/AppServer/profiles/Craig02/logs/addNode.log ADMU0128I: Starting tool with the Craig02 profile ADMU0001I: Begin federation of node m23vnx60Node02 with Deployment Manager at kll6571:8879. ADMU0009I: Successfully connected to Deployment Manager Server: kll6571:8879 ADMU0507I: No servers found in configuration under:

144 WebSphere Application Server V6 Problem Determination for Distributed Platforms

ADMU0211I: Error details may be seen in the file: /opt/IBM/WAS6/AppServer/profiles/Craig02/logs/addNode.log

You can get more detailed information from the addNode log as shown in Example 4-20 on page 145.

Example 4-20 Errors from the addNode.log

[7/12/05 15:20:02:155 EDT] 0000000a AdminTool A ADMU0113E: Program exiting with error: com.ibm.websphere.management.exception.ConnectorException: ADMC0009E: The system failed to make the SOAP RPC call: invoke at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPCon nectorClient.java:642) ... resulting from: [SOAPException: faultCode=SOAP-ENV:Client; msg=Read timed out; targetException=java.net.SocketTimeoutException: Read timed out] at org.apache.soap.transport.http.SOAPHTTPConnection.send(Unknown Source)

4.3.9 Problem: Repository synchronization

You are making changes to your configuration in a Network Deployment environment and while you can save the changes, the synchronization request to a remote node is failing.

Chapter 4. System management problem determination 145

What to look for Review SystemOut.log from the deployment manager and from the node agent on the server that is not synchronizing. The administrative console message will tell you which node is failing to synchronize as shown in Figure 4-9.

Figure 4-9 Output from a failed node synchronization

The message shown in Example 4-21 appears in the SystemOut.log of the

deployment manager.

Example 4-21 Unable to synchronize a node

[7/3/05 16:01:17:845 EDT] 00000026 CellSync E ADMS0104I: The system is unable to invokea synchronization request on nodeWebSphere:platform=common,cell=IBM-99TVXRDCell01,version=6.0.1.2,name=nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=IBM-99TVXRDNode01,process=nodeagent.javax.management.InstanceNotFoundException: MBeanServer cannot find MBean with ObjectNameWebSphere:platform=common,cell=IBM-99TVXRDCell01,version=6.0.1.2,name=nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=IBM-99TVXRDNode01,process=nodeagent

146 WebSphere Application Server V6 Problem Determination for Distributed Platforms

The replicated copy of the file being modified might be corrupted or inaccessible on the node where the change is being made. For example, making a change to the Java heap size parameter updates the server.xml file in the repository. The change is made to the master copy of the file, and the administrative console returns a message that the change was saved and that the synchronization was successful, as shown in Figure 4-10.

Figure 4-10 Output from a successful save and synchronization

The deployment manager's SystemOut.log also shows the synchronization as

successful: [7/3/05 16:11:13:343 EDT] 00000085 DeploymentMan A ADMS0208I: The configuration synchronization complete for cell.

However, the node agent's SystemOut.log shows you that the node agent is not able to open the file due to permissions, as shown in Example 4-22.

Example 4-22 Node agent fails to open server.xml

[7/3/05 16:11:13:902 EDT] 00000040 FileDocument E ADMR0104E: The system is unable to readdocument cells/IBM-99TVXRDCell01/nodes/IBM-99TVXRDNode01/servers/server1/server.xml:java.io.FileNotFoundException:C:\IBM\WAS6\AppServer\profiles\AppSrv01\config\cells\IBM-99TVXRDCell01\nodes\IBM-99TVXRDNode01\servers\server1\server.xml (Access is denied)[7/3/05 16:11:13:912 EDT] 00000040 NodeSyncTask A ADMS0016I: The configurationsynchronization failed.

Chapter 4. System management problem determination 147

In this case, the solution is to fix the file permissions and retry the synchronization.

In other cases, such as with a corrupt or empty file, the solution is to delete the file and allow automatic synchronization to replace the file with the good copy from the master repository.

If automatic synchronization is not enabled, you need to manually synchronize

the node either from the administrative console or from the node itself. Figure 4-11 shows the file synchronization settings for the node agent. In this instance, automatic synchronization is enabled.

Figure 4-11 Setting up automatic or manual synchronization

148 WebSphere Application Server V6 Problem Determination for Distributed Platforms

If you are unable to determine why synchronization is failing from the messages in the logs, try a command line synchronization initiated from the node that you cannot synchronize. You need to stop the node agent. From a command line, issue the syncNode command, as shown in Figure 4-23. You need to pass it the host name and SOAP port of the deployment manager. You can also specify the -trace option.

Example 4-23 Running and tracing syncNode

[wasadmin@m23vnx60 bin]$ ./syncNode.sh kll6571 8879 -trace ADMU0115I: Trace mode is on. ADMU0116I: Tool information is being logged in file /opt/IBM/WAS6/AppServer/profiles/Node01/logs/syncNode.log ADMU0401I: Begin syncNode operation for node m23vnx60Node01 with Deployment Manager kll6571: 8879 ADMU0016I: Synchronizing configuration between node and cell. ADMU0111E: Program exiting with error: com.ibm.websphere.management.exception.AdminException: ADMU0005E: Error synchronizing repositories ADMU0211I: Error details may be seen in the file: /opt/IBM/WAS6/AppServer/profiles/Craig01/logs/syncNode.log

Note: You can stop a node agent without having to stop the application servers. The operational impact of stopping the node agent is that it will not monitor the application servers to make sure they are up.

Reviewing the log file shows the problem as shown in Example 4-24.

Example 4-24 syncNode fails to open server.xml

[7/11/05 12:14:00:448 EDT] 0000000c FileDocument E ADMR0105E: The system is unable to writedocument cells/kll6571Cell01/nodes/m23vnx60Node01/servers/server02/server.xml:java.io.FileNotFoundException:/opt/IBM/WAS6/AppServer/profiles/Node01/config/cells/kll6571Cell01/nodes/m23vnx60Craig01/servers/server02/server.xml (Permission denied)

In this example, it was not necessary to run the trace because the error message would have appeared anyway. A trace is useful for finding the problem when there are no obvious errors.

Chapter 4. System management problem determination 149

Example 4-25 Benign FileNotFound exceptionjava.io.FileNotFoundException:/opt/IBM/WAS6/AppServer/profiles/Node01/config/cells/kll6571Cell01/nodes/m23vnx60Node01/servers/nodeagent/variables.xml (No such file or directory)

4.3.10 Problem: Save conflicts in the administrative console

You are seeing messages telling you that there is a save conflict in the administrative repository when you login to the administrative console.

What to look for

After you made changes to the WebSphere Application Server configuration in the administrative console and try and save those changes, you see a message telling you there is a save conflict as shown in Figure 4-12.

Figure 4-12 Save conflict

This error tells you that there is a conflict in updates to one or more files in the configuration repository. The save conflict is not reported in the SystemOut.log unless you choose to overwrite the conflicting change or have enabled tracing.

150 WebSphere Application Server V6 Problem Determination for Distributed Platforms

A save conflict can happen for a number of reasons: 򐂰 More than one administrator has made changes to the configuration at the same time. 򐂰 Changes have been made to the configuration repository files directly. 򐂰 You are deploying applications using WebSphere Rapid Deployment.

Example 4-26 Overwriting configuration changes

[7/11/05 11:20:03:348 EDT] 00000037 FileRepositor A ADMR0016I: User WTRNTDM/config modifieddocument cells/kll6571Node01Cell/nodes/kll6571StandAlone/servers/server1/server.xml.[7/11/05 11:20:10:569 EDT] 0000003f ServletWrappe A SRVE0242I:[/secure/tiles/syncconflict.jsp]: Initialization successful.[7/11/05 11:20:23:808 EDT] 00000038 FileRepositor W ADMR0114W: The system is overwritingdocument cells/kll6571Node01Cell/nodes/kll6571StandAlone/servers/server1/server.xml by request.[7/11/05 11:20:23:848 EDT] 00000038 FileRepositor A ADMR0016I: User WTRNTDM/wasadmin modifieddocument cells/kll6571Node01Cell/nodes/kll6571StandAlone/servers/server1/server.xml.

It is possible to have more than one administrator working on the repository at

the same time and not have problems with save conflicts when those administrators are working with different files and objects in the repository. Unless you constantly check the SystemOut.log, you cannot be sure that the changes you are making are not being overwritten. Thus, we recommend that you only have one administrator making changes at a time.

4.3.11 Problem: enterprise applications missing

You have logged into the administrative console and are trying to manage enterprise applications but none are displayed. The applications might be working fine operationally, you just cannot manage them. This problem is most likely to happen after a server restart.

Chapter 4. System management problem determination 151

Data to collect The following logs can be helpful in determining why applications are missing: 򐂰 Deployment manager log 򐂰 Node agent log 򐂰 Application server log

What to look for

From the administrative console, you should be able to see a list of installed applications under Applications → Enterprise Applications. If your applications are no longer listed here, start by checking the deployment manager log. This problem can be caused by a corruption to the configuration repository or problems with permissions, as shown in Example 4-27.

Example 4-27 Permissions problem in deployment manager log

[7/13/05 14:09:48:993 EDT] 0000003a FileDocument E ADMR0104E: The system is unable to readdocument cells/m23vnx60Cell01/applications/HelloApp.ear/deployments/IBMUTC/deployment.xml:java.io.IOException: No such file or directory

Reviewing the node agent's log would show you a similar message, possibly referring to a different file, as shown in Example 4-28.

Example 4-28 Permissions problem in node agent log

[7/13/05 14:12:03:979 EDT] 00000035 FileDocument E ADMR0109E: An error occurred restoringdocumentcells/m23vnx60Cell01/applications/HelloApp.ear/deployments/HelloApp/META-INF/was.policy:java.io.FileNotFoundException:/opt/IBM/WAS6/AppServer/profiles/AppSrv01/config/cells/m23vnx60Cell01/applications/HelloApp.ear/deployments/HelloApp/META-INF/was.policy (Permission denied)

The application server log also shows that the problem is caused by file permissions as shown in Example 4-29.

Example 4-29 Permissions problem in application server log

[7/13/05 13:55:26:517 EDT] 0000001b FileDocument E ADMR0104E: The system is unable to readdocument cells/m23vnx60Cell01/nodes/m23vnx60Node01/perftuners.xml: java.io.IOException:Permission denied

In all three cases, the messages refer to the different files and different locations in the file system. Checking the file permissions for those files listed shows that the user who is running the processes does not have write access to those files.

In this example, the servers run on a UNIX platform and are configured to run as a non-root user under normal conditions. At some point, the servers were

152 WebSphere Application Server V6 Problem Determination for Distributed Platforms

restarted by the root user and the HelloApp application was deployed. Then the servers were restarted as the normal non-root user and this problem occurred. The non-root user was not able to write to the files that were created when the server was running as root. The problem is resolved by resetting the file permissions.

4.3.12 Problem: Invalid or expired certificates

If your problems seem to be specific to applications that require SSL access, it is a good probability that the problem is due to invalid or expired certificates. This can cause a variety of system management problems. Problems with expired certificates could occur at any time if you are not managing your certificates. You could also run into this problem if you have recently enabled security.

You might also need to take an Admin trace and a security trace, depending on the problem. See "Collecting data" on page 122 for details.

What to look for

WebSphere Application Server writes a warning message (Example 4-30) into the SystemOut.log before a certificate is due to expire. This gives you the opportunity to renew the certificates before the actual expiration dates and before applications stop working. However, at present, WebSphere Application Server V6 does not notify you that a certificate has expired.

Example 4-30 Certificate expiry warning messages

7/6/05 11:37:15:860 EDT] 0000000a SASRas W JSAS0456W: WARNING in sasOutboundSSLConfig:The certificate with alias expires 07/07 from keyStoreC:\IBM\WAS6\AppServer/profiles/StandAlone/etc/ExpiresKeyFile.jks will be expired in 1 days.

Chapter 4. System management problem determination 153

Tracing the application server with the following tracestring shows you if the problem is with certificates, as shown in Example 4-31: traceString=ORBRas=all

Example 4-31 Trace messages showing an expired certificate

[7/7/05 10:23:17:553 EDT] 00000020 ORBRas 1 com.ibm.rmi.transport.ListenerThread run:259LT=3:P=897141:O=0:port=9402 The following exception was loggedjavax.net.ssl.SSLException: No available certificate corresponds to the SSL cipher suites whichare enabled.

If you see messages that indicate a certificate has expired or has a problem, you should verify the certificate using the IBM Key Management tool (ikeyman).

In a Network Deployment installation, you can find this file at:

/profiles//config/cells//security.xml

In security.xml, look for theXML tag as shown in Example 4-32.

Example 4-32 Key database files defined in security.xml

You will need to know the password on the key database file. If you have not replaced the supplied dummy key file databases, the default password is WebAS.

154 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Figure 4-13 shows the DummyServerKeyFile key database open in the ikeymanutility. Click on the certificate name, websphere dummy server in this example,and then click View/Edit. If the certificate has expired, the utility displays amessage box telling you this. Then, it displays the certificate details.

Figure 4-13 Open key database in the ikeyman utility

The Validity field also shows if the certificate has expired. In Figure 4-14 onpage 156, the certificate is only valid up to July 7, 2005. Click OK to close thewindow.

Chapter 4. System management problem determination 155

Figure 4-14 Expired certificate details

Action: Request a new key from a certificate authority

If the certificate has expired, you can resolve this problem by creating a new valid certificate in the existing key database file.

When a client program such as a browser is setting up an SSL connection with a

server, the client can trust the contents of a certificate that is verified by a trusted third party. A certificate authority acts as a trusted third party. The key management utility allows you to request a certificate from a certificate authority. The certificate authority will charge you for this service.

Action: Create a self-signed certificate (not for production)

Alternatively, you can create a self-signed certificate in the dummy key file database. This is a certificate that can be used to set up an SSL connection if the client chooses to accept the certificate even though it has not been verified by a certificate authority. In a production environment, you should not use self-signed certificates for securing SSL connections.

To create a self signed certificate, click New Self Signed as shown in

Figure 4-13 on page 155.

156 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Figure 4-15 Creating a new self signed certificate

Enter the information for the self signed certificate. The maximum validity periodyou can set is 7300 days or approximately 20 years.

After you have created your new certificates, restart WebSphere ApplicationServer and retest.

Where to go from here: If you have exhausted all the possibilities in this section and are still having problems with SSL and certificates, refer to "The next step" on page 162.

Chapter 4. System management problem determination 157

4.3.13 Problem: WebSphere Rapid Deployment You are trying to use WebSphere Rapid Deployment to develop and test an application, and you are not able to connect to an application server or WebSphere Rapid Deployment will not create or update the applications.

What to look for

If you have problems with WebSphere Rapid Deployment, you can take the following actions: 򐂰 Verify the WORKSPACE variable 򐂰 Verify the WebSphere Rapid Deployment configuration 򐂰 Verify the application is being built 򐂰 Verify the deployment manager and application server are running 򐂰 Look for Java coding errors

Verify the WORKSPACE variable

You tell WebSphere Rapid Deployment what directory to monitor by setting the WORKSPACE environment variable. What directory you use is up to you. Ensure that you are putting the files that you want updated in the location that is specified by the WORKSPACE environment variable. For example: WORKSPACE=C:\IBM\WRD

If you specify the directory incorrectly in the WORKSPACE variable, WebSphere

Rapid Deployment starts but it does not build or deploy your applications because it is monitoring the wrong directory.

Verify the WebSphere Rapid Deployment configuration

Ensure that you have run wrd-config to set up your environment for the WebSphere Rapid Deployment project that you are working on. You can check your project settings by looking in the project configuration file at: //.wrdconfig.xml

Example 4-33 shows the contents of this file.

158 WebSphere Application Server V6 Problem Determination for Distributed Platforms

In a Network Deployment environment, the serverName should be the logical

name of the application server that will run the application. The serverJMXHostand serverJMXPort should refer to the host name and SOAP port of thedeployment manager.

Verify the application is being built

You can check the WebSphere Rapid Deployment log (Example 4-34) to ensurethe environment has been initialized and that changes to the workspace arebeing recognized. The messages also shows if the changes are being handledproperly.

!ENTRY org.eclipse.core.runtime Jul 10, 2005 03:37:32.816

!MESSAGE Product org.eclipse.platform.ide could not be found.

!ENTRY com.ibm.ws.rapiddeploy.core 0 0 Jul 10, 2005 03:40:49.16

!MESSAGE Creating a new project 'HelloApp'.

!ENTRY org.eclipse.core.resources 1 1 Jul 10, 2005 03:40:51.936

Chapter 4. System management problem determination 159

!MESSAGE Could not load library: libcore_2_1_0b.so. This library provides platform-specific optimizations for certain file system operations. This library is not present on all platforms, so this may not be an error. The resources plug-in will safely fall back to using java.io.File functionality.

!ENTRY com.ibm.ws.rapiddeploy.core 1 1 Jul 10, 2005 03:40:51.994

!MESSAGE Recording active directory ID to the workspace root.

!ENTRY com.ibm.ws.rapiddeploy.core 0 0 Jul 10, 2005 03:40:52.85

!MESSAGE Configuring style 'WebSphere Free Form Project' for project 'HelloApp'.

!ENTRY com.ibm.ws.rapiddeploy.core 0 0 Jul 10, 2005 03:40:52.87

!MESSAGE Recording the current style and project properties...

!ENTRY com.ibm.ws.rapiddeploy.core 0 0 Jul 10, 2005 03:40:52.782

!MESSAGE Configuring build output location to : '/HelloApp/bin'

!ENTRY com.ibm.ws.rapiddeploy.core 0 0 Jul 10, 2005 03:40:52.861

!MESSAGE Configuring source folder to '/HelloApp'

!ENTRY com.ibm.ws.rapiddeploy.core 0 0 Jul 10, 2005 03:40:52.863

!MESSAGE Configuring source folder to '/HelloApp/gen/src'

!ENTRY com.ibm.ws.rapiddeploy.core 0 0 Jul 10, 2005 03:40:53.110

!MESSAGE Configuring new build output location to '/HelloApp/bin'

Verify the deployment manager and application server are running

If WebSphere Application Server is not available when you start WebSphere Rapid Deployment, you see the following message in the WebSphere Rapid Deployment console: ERROR! Failed to make connection to WebSphere Application Server

Example 4-35 WebSphere Rapid Deployment unable to publish application

Error occurred during download. : publishrecord.remote [01:23:40 PM] Exporting Ear File. [01:23:41 PM] ERROR! [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.ConnectException: Connection refused: connect;

160 WebSphere Application Server V6 Problem Determination for Distributed Platforms

targetException=java.lang.IllegalArgumentException: Error opening socket:java.net.ConnectException: Connection refused: connect]

In a Network Deployment environment, you publish the application to the

deployment manager. However, the application is run on a separate applicationserver. If the deployment manager is not running, you see the message that isshown in Example 4-35 in the console. If the target application server is notrunning, the application publishes but does not start. You see a message in theconsole as shown in Example 4-36.

Example 4-36 WebSphere Rapid Deployment unable to start application

[12:08:01 PM] ERROR! Application Failed to Start. HelloApp[12:08:01 PM] ERROR! MBeanServer cannot find MBean with ObjectNameWebSphere:platform=dynamicproxy,cell=m23vnx60Cell01,version=6.0.1.2,name=ApplicationManager,mbeanIdentifier=ApplicationManager,type=ApplicationManager,node=m23vnx60Node01,process=server1[12:08:01 PM] ERROR! Please see server logs for more details.

After you have restarted the process, you need to make a change to one of theapplication files so that WebSphere Rapid Deployment detects the update andpublishes the application again.

Note: Using WebSphere Rapid Deployment causes save conflicts in the

administrative console if you or another administrator are logged in. This is another reason why you should use WebSphere Rapid Deployment only for development.

Look for Java coding errors

If your application code has errors in it, WebSphere Rapid Deployment showsthe coding error in the console, for example:[01:28:05 PM] 'Syntax error, insert ";" to complete Statement' in resource'HelloServlet.java' on line number 37

Java coding errors are outside the scope of this book.

Chapter 4. System management problem determination 161

Where to go from here: If you have checked all of the problems described thus far and yet still cannot build and deploy an application with WebSphere Rapid Deployment, check the logs for the deployment manager if appropriate and the application server for any messages that are related to the application deployment.

If you have exhausted all the possibilities that are described in this section and are still not able to use WebSphere Rapid Deployment, refer to the next section.

4.4 The next step

The symptoms and problem areas included in this paper are some that you are more likely to experience. However, there are other things that can go wrong, or the cause of the problem might be related to something other than system management components.

Review the problem classifications to see if there are any other components that might be causing the problem.

162 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 MustGather: Profile Creation/Removal Issues for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196228򐂰 MustGather: Federation or Removal of a Node Issues for Version V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196227򐂰 MustGather: Port Management for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196226򐂰 MustGather: Node agent and Deployment Manager discovery problems for all releases and editions of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196220򐂰 MustGather: Usage and creation of templates fail on V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21195439

Chapter 4. System management problem determination 163

164 WebSphere Application Server V6 Problem Determination for Distributed Platforms 5

Chapter 5. Web container problem

determination If users receive unexpected results in a Web browser (such as errors or incorrect information), you might have a problem with the Web components in the application. The runtime environment for Web components is called the Web container. This paper discusses techniques for diagnosing problems that can occur in the Web container. Potential symptoms of a Web container problem can include: 򐂰 Users are not able to access a Web component 򐂰 Unexpected behavior when running a Web component 򐂰 Errors when starting a Web component 򐂰 Problems with JSP compilation 򐂰 Errors or exceptions thrown when running a Web component 򐂰 Messages starting with SRVE (Web container), JSPG (JSPs), or JSFG (JSF)

Problems areas that are related to the Web container also include session management problems and character encoding problems.

Important: We recommend that you start your problem determination process

by reading Approach to Problem Determination in WebSphere Application Server V6 at http://www.redbooks.ibm.com/redpapers/pdfs/redp4073.pdf.

© Copyright IBM Corp. 2005. All rights reserved. 165

5.1 Introduction The Web container is a runtime environment for Web applications. It processes servlets, JSP files, and other types of server-side includes. The Web container also provides session management, static content processing and an inbound transport chain for HTTP requests. Each application server runtime has one logical Web container, which can be modified but not created or removed.

Figure 5-1 illustrates the Web container and its place within the application server.

Application Server

Session Manager

Servlet JSP

EJB Container

JCA Services

Name Server (JNDI)

Security Server

Messaging

Web Services

Figure 5-1 Web container overview

Each Web container provides the following:

򐂰 Web container transport chains Requests are directed to the Web container using the Web container inbound transport chain. The chain consists of a TCP inbound channel that provides the connection to the network, an HTTP inbound channel that serves HTTP 1.0 and 1.1 requests, and a Web container channel over which requests for servlets and JSPs are sent to the Web container for processing.

166 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 Servlet processing When handling servlets, the Web container creates a request object and a response object and then invokes the servlet service method. The Web container invokes the servlet's destroy method when appropriate and unloads the servlet, after which the JVM performs garbage collection.򐂰 HTML and other static content processing Requests for HTML and other static content that are directed to the Web container are served by the Web container inbound chain. However, in most cases, using an external Web server and Web server plug-in as a front-end to a Web container is more appropriate for a production environment.򐂰 Session management Support is provided for the javax.servlet.http.HttpSession interface as described in the Servlet API specification.

Figure 5-2 on page 168 illustrates the directory structure of a Web application.

Chapter 5. Web container problem determination 167

Rational Application Developer Assembly root

WEB-INF JSPs, HTML, etc.

web.xml lib classes

ibm-web-ext.xmi ibm-web-bnd.xmi faces-config.xml Server-side class files Library JAR files

Figure 5-2 Web application components

5.1.1 Initial symptoms

A problem that occurs in the Web container usually causes unexpected results to be displayed in the Web browser. Three common indications are addressed here: 򐂰 HTTP 404 errors 򐂰 HTTP 500 errors 򐂰 Incorrect information

Figure 5-3 on page 169 shows a flow chart of the high-level symptoms and the potential problem areas that might apply to each.

168 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Browser displays

HTTP 404 Error HTTP 500 Error Incorrect Information

The page cannot be found JSP Processing Error HTTP session

Failed to find resource IllegalStateExceptions Web resource reloading

WebGroup/Virtual Host error Encoding

Static resource

Figure 5-3 Initial symptoms for Web container problems

5.2 Work the problem

You begin the problem determination process by collecting the appropriate data that is required to diagnose the problem. We give you a list of all the documentation that might be required and how to collect it. If you have limited ability to recreate the problem, you might want to collect all the documentation at once, before starting the problem determination process.

And lastly, we provide guidance on the next step to take for resolution, whether it be a support site, contacting IBM, information about configuration, or some other suggestion as to how to proceed.

5.2.1 Data to collect

The following data will help you with the problem determination process in a Web container execution environment: 򐂰 WebSphere Application Server JVM logs: SystemOut and SystemErr files 򐂰 WebSphere Application Server process logs: native_stderr.log and native_stdout.log log files 򐂰 Web server log files

Chapter 5. Web container problem determination 169

For information about collecting the JVM and Process logs, see WebSphere Application Server V6: Diagnostic Data at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4085.pdf

The Web server log files names and locations are specific to the product (IBM HTTP Server, Apache, SunOne, IIS, and Domino®) that is used for this function.

If the problem is difficult to recreate or disruptive to business operations, see

"The next step" on page 209 for a complete list of documentation to collect before continuing. In particular, you should review the MustGather documents for a complete list of documentation that is required by IBM support.

5.2.2 High-level symptom analysis

Consider the following error types. If you do not find the description of your problem here, go to "The next step" on page 209.

What to look for

Many indicators of a Web container problem first appear to a user as an unexpected Web browser page or page contents. The unexpected information can be an error page that results from an HTTP error or a page that has incorrect or incomplete information displayed.

HTTP 404 errors

HTTP 404 errors can have different underlying causes. Table 5-1 shows the HTTP 404 errors that we address in this paper.

Table 5-1 Where to go for information about HTTP 404 errors

If the errors include Then go to

Page can't be found or displayed "Symptom: HTTP 404 error - The page cannot be displayed" on page 172

JSPG0036E: Failed to find resource "Symptom: HTTP 404 error - Failed to find message resource" on page 175

WebGroup/virtual host not define "Symptom: HTTP 404 error -

WebGroup/virtual host not defined" on page 177

170 WebSphere Application Server V6 Problem Determination for Distributed Platforms

HTTP 500 errorsHTTP 500 errors can also have different underlying causes, which include thefollowing:򐂰 If the errors indicate JSP processor errors or incorrect syntax in JSPs, go to "Symptom: HTTP 500 error - JSP processing error" on page 179.򐂰 If the symptoms include an HTTP 500 error page on the browser and in the SystemOut.log file you see an IllegalStateException, go to "Symptom: HTTP 500 error - IllegalStateException" on page 181.

Incorrect information displayed on Web pages

If the users are receiving the correct page on the Web browser but the pagecontains invalid information, check these symptoms:򐂰 Users report that pages appear with missing elements. If a Web page appears but is missing static resources such as text, images, or file segments, go to "Static resources not displayed" on page 190.򐂰 Users report seeing an old version of application pages. If a JSP file has been modified and deployed to the server but the changes do not appear to the browser interface, you might need to ensure that the application has been enabled for JSP reloading. Go to "Web resources not reloading" on page 192.򐂰 Users report that pages contain invalid information such as multiple question marks (???) or garbage characters. It is possible that the application uses DBCS characters (Japanese, Chinese, Korean languages) or specific characters for other languages that are not included in the default character encoding. In these cases, a correct character encoding configuration is necessary to display and process this information without problems. For more details related to encoding configurations, go to "Encoding and internationalization issues" on page 195.򐂰 Users report problems such as lost data during a session. Users repeatedly have to enter information that should be saved during the session, lose shopping cart information, or have other short-term data loss. This data loss might be caused by a problem with HTTP sessions. In a clustered environment, the problem could be with the Web server plug-in. However, there are other possible causes, such as configuration or application errors. To further analyze this problem, go to "HTTP session management" on page 202.

Chapter 5. Web container problem determination 171

5.2.3 Symptom: HTTP 404 error - The page cannot be displayed Figure 5-4 shows an overview of the steps that you can take to find the cause of an HTTP 404 error caused by a page cannot be displayed condition.

no Web Module starts sucessfully? Review JVM logs problems

yes no Web Server OK? HTTP Server support

yes

Application accessible through the yes

Web container but not Web server? HTTP plug-in problems

no

Browser URL verification steps

Application server OK? yes

Review application development

no

Look at available online support Obtain help from IBM Support

Figure 5-4 Page cannot be displayed verification steps

Verify that the Web module has started successfully

Check the JVM logs for the application server, and look for messages in the SystemOut.log file (Example 5-1) to verify that the Web container and Web module started successfully.

Example 5-1 Web Module start process results in SystemOut.log file

ApplicationMg A WSVR0200I: Starting application: [application_name] WebContainer A SRVE0161I: IBM WebSphere Application Server - Web Container. Copyright IBM Corp. 1998-2004 WebContainer A SRVE0162I: Servlet Specification Level: 2.4 WebContainer A SRVE0163I: Supported JSP Specification Level: 2.0 WebGroup A SRVE0169I: Loading Web Module: [web_module_name] ApplicationMg A WSVR0221I: Application started: [application_name]

172 WebSphere Application Server V6 Problem Determination for Distributed Platforms

One possible reason that would keep the Web container from starting is aproblem with the session manager. To determine if this is the case:򐂰 Look for any errors or exceptions containing a package name of com.ibm.ws.webcontainer.httpsession. You will normally find these errors between the starting application message and the application started message.򐂰 Look in the logs for session manager related messages. These messages will be in the format SESNxxxxE for errors and SESNxxxxW for warnings. You might also see transaction messages that indicate a problem with session data (see Example 5-2).

Example 5-2 Session manager message error

Error "SRVE0054E: An error occurred while loading session context and Webapplication.

If you find a session manager error but the explanation is not sufficient to solvethe problem, go to "The next step" on page 209.

Verify that the Web server is working

Verify that the Web server is healthy by accessing the URL http://server_namefrom a browser and seeing whether the Welcome page appears. This actionindicates whether the Web server is up and running, regardless of the state ofWebSphere Application Server.

Verify that the Web server plug-in is working

If you have recently updated or installed the application, ensure that the Webserver plug-in was regenerated and propagated to the Web server. Also, ensurethat the Web server is using the new plug-in configuration file.

If you have regenerated the plug-in and are sure it is in use but still have aproblem, you can bypass the Web server and access the application directly

Chapter 5. Web container problem determination 173

from the application server. This is not the recommended method of serving a production Web site. However, it provides a good diagnostic tool when it is not clear whether a problem resides in the Web server, WebSphere Application Server, or the Web server plug-in.

Verify that the URL is correct

If you cannot access the page directly from the application server, verify that the URL being used to access the page is correct. For more information, see "Application URL specification" on page 186.

Verify that the application server is working

If the URL appears to be correct but you cannot access the resource directly through the application server, verify the health of the hosting application server and Web module by doing the following: 1. View the hosting application server and Web module in the administrative console to verify that they are up and running. In a single server environment, you can check the application server process to see if it is running, or you can use the serverStatus command as follows: c:\cd WebSphere\AppServer\profiles\AppSrv01\bin serverStatus server1

174 WebSphere Application Server V6 Problem Determination for Distributed Platforms

2. Copy a simple HTML or JSP file to the Web module document root and try to access it. The Web module document root is located in: /profiles//installedApps// .ear\.war If successful, the problem is with the resource. View the SystemOut.log file for the application server to discover why the resource cannot be found or served.

Look at available online support

If none of these steps fixes your problem, check to see if the problem has been identified and documented by looking at the available online support (hints and tips, tech notes, and fixes) that are related to Web container problems: http://www.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc1=SSCMPDF

If the searches fail to find the problem, then go to "The next step" on page 209 for information about gathering the MustGather documentation for HTTP status code 404 "Not Found" problems.

5.2.4 Symptom: HTTP 404 error - Failed to find resource

If the browser displays a JSPG0036E: Failed to find resource message, the JSP processor cannot find the specified JSP page in the Web module, as shown in Figure 5-5 on page 176.

Chapter 5. Web container problem determination 175

Figure 5-5 JSPG0036E: Failed to find resource

Verify that the URL is correct

Follow the steps described in "Application URL specification" on page 186 to see if the URL being specified is the correct URL.

If the URL appears to be correct, it is possible that this page does not exist in the Web module deployment unit (WAR file). Verify that the requested page is located in the Web module directory structure:

/profiles//installedApps// .ear\.war

Look at available online support

If these steps have failed to identify the problem, see "The next step" on page 209.

176 WebSphere Application Server V6 Problem Determination for Distributed Platforms

5.2.5 Symptom: HTTP 404 error - WebGroup/virtual host not defined The error message SRVE0017W: WebGroup/Virtual Host has not been defined can appear in the SystemOut.log file for many reasons. Check the following to correct the problem.

Verify that the Web module started

Verify that the Web module has started successfully (see "Verify that the Web module has started successfully" on page 172).

Verify that the host alias is unique

A duplicate host alias that is defined in multiple virtual hosts can cause this type of problem. For example, in Example 5-3, the host alias test:80 is duplicated in both virtual hosts because a URI that contains test:80 would match aliases in both virtual hosts (*:80 and test:80).

Example 5-3 Virtual host definitions

To resolve this problem:

1. In the WebSphere administrative console, click Environment → Virtual Hosts. 2. Select the target virtual host, then click Host Aliases, and select the *:80 host alias. 3. Change the Host Name field to a specific alias instead of * alias. 4. Restart the application server.

Note: The existence of localhost as an alias can cause duplicate entries.

If localhost is used as an alias entry, check the etc/hosts file to ensure that all host names (aliases) associated with the loop back address (127.0.0.1) are part of the same virtual host grouping (for example, default_host).

Chapter 5. Web container problem determination 177

Verify that the URL is correct See "Application URL specification" on page 186 to determine that the URL is correct.

If you receive this error message, refer to Technote # 1193379 at the following Web address for the cause and solution details: http://www.ibm.com/support/docview.wss?uid=swg21193379

Look at available online support

If none of these steps fixes your problem, check to see if the problem has been identified and documented by looking at the available online support (hints and tips, technotes, and fixes) that are related to Web container problems: http://www.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc1=SSCMPDF

If your searches fail to find the problem, then go to "The next step" on page 209 for information about gathering the MustGather documentation for servlet engine and Web container problems.

178 WebSphere Application Server V6 Problem Determination for Distributed Platforms

5.2.6 Symptom: HTTP 500 error - JSP processing error Figure 5-6 shows the diagnostic steps to take when the symptoms of the problem include an HTTP 500 - JSP processing error.

no HTML or servlet displays correctly HTTP 404 - Page not found steps

yes

no JSP Processor starts normally

Figure 5-6 JSP processing error

Determine if the failing file is the only failure

You should determine whether other resource types such as HTML files or servlets are being requested and displayed correctly. If the failure does not appear to be limited to one JSP file, go to "Symptom: HTTP 404 error - The page cannot be displayed" on page 172.

Ensure that the JSP processor started normally

If other resources are being displayed correctly, determine whether the JSP processor has started normally. Browse the SystemOut.log file of the server that hosts the JSP files that you are trying to access. The messages shown in Example 5-4 indicate that the JSP processor has started normally.

Example 5-4 JSP processor messages in SystemOut.log file

WebContainer A SRVE0239I: Extension Factory [class com.ibm.ws.jsp.webcontainerext.JSPExtensionFactory] was registered successfully. WebContainer A SRVE0240I: Extension Factory [class com.ibm.ws.jsp.webcontainerext.JSPExtensionFactory] has been associated with patterns [*.jsp *.jspx *.jsw *.jsv ].

If the JSP processor fails to load, go to "The next step" on page 209 for information about gathering the MustGather documentation for JSP exceptions.

Chapter 5. Web container problem determination 179

Look for JSP code errors If the JSP processor starts normally, the problem might be with the JSP file itself. The JSP might have invalid JSP syntax that cannot be processed by the JSP processor. To look for JSP code errors: 򐂰 Examine the SystemOut.log file of the target application server for invalid JSP directive syntax messages. Errors similar to that shown in Example 5-5 indicate this kind of problem.

Example 5-5 JSP directive syntax error message

Message: /test.jsp(2,1)JSPG0076E: Missing required attribute page for jsp element jsp:include

򐂰 Examine the SystemOut.log file for problems with invalid Java syntax. Errors that contain text similar to "failed to compile (shown in Example 5-6) indicate this kind of problem.

Example 5-6 Invalid Java syntax error message

com.ibm.ws.jsp.JspCoreException: JSPG0049E: /test.jsp failed to compile : JSPG0091E: An error occurred at line: 16 in the file: /test.jsp JSPG0093E: Generated servlet error from file: /test.jsp

򐂰 Look up the extended error definitions for JSP messages at:

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.messages.doc/doc/JSPG.html 򐂰 Correct the error in the JSP file and retry the file.

Look at available online support

If you have not identified the problem, check to see if the problem has been documented by looking at the available online support (hints and tips, technotes, and fixes) for JSP problems at: http://www-1.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc1= SSCC2GL&rankprofile=8&dc=DB520+D800+D900+DA900+DA800&dtm

If the searches fail to resolve your problem, go to "The next step" on page 209 for information about gathering the MustGather documentation for JSP exceptions.

180 WebSphere Application Server V6 Problem Determination for Distributed Platforms

5.2.7 Symptom: HTTP 500 error - IllegalStateException If you have an illegal state exception error, the exception should give you an indication of the illegal state causing the problem. This section addresses the following: 򐂰 Invalid session object 򐂰 Response generation problems

Invalid session object

The session manager component uses the HttpSession interface to create a session between an HTTP client and the server. When a new session object is created, a unique session ID is assigned to it. The session ID, which is then passed as part of every request, matches the user with the session object. Session tracking gives servlets the ability to maintain state and user information across multiple requests.

Sessions might get invalidated automatically due to a session timeout (see

"Session timeout interval" on page 207) or can be ended explicitly by application code. The HttpSession interface provides the following method to terminate a session explicitly: public void invalidate();

When a session terminates, the session object and the information that is stored in it are lost permanently. The session manager unbinds any objects bound to the session before it destroys the session.

Figure 5-7 illustrates the HttpSession interface life cycle.

Retrieve the session object

Invoke a session object method Store object in the session

Retrieve object from session

no Invalid session object ? Remove object from the session yes Invalidate session object Throw IllegalStateException

Figure 5-7 HttpSession interface life cycle

Chapter 5. Web container problem determination 181

When the application tries to use an invalidated session object, the IllegalStateException occurs, as shown in Example 5-7.

Example 5-7 IllegalStateException in invalid session object

[7/7/05 16:41:30:627 ART] 00000028 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet TestServlet. Exception thrown : java.lang.IllegalStateException: Session Object Internals: id : pi55X7syi-ExTjyyhFn5Cu7 hashCode : 1449590567 create time : Thu Jul 07 16:24:54 ART 2005 last access : Thu Jul 07 16:41:30 ART 2005 max inactive interval : 1800 user name : anonymous valid session : false new session : true overflowed : false non-serializable app specific session data : {} serializable app specific session data : {} at com.ibm.ws.webcontainer.httpsession.SessionData.getValueGuts(SessionData.java(C ompiled Code)) at com.ibm.ws.webcontainer.httpsession.SessionData.getValue(SessionData.java(Inlin ed Compiled Code)) at com.ibm.ws.webcontainer.httpsession.SessionData.getAttribute(SessionData.java(I nlined Compiled Code)) at com.ibm.ws.webcontainer.httpsession.HttpSessionFacade.getAttribute(HttpSessionF acade.java(Compiled Code))

Check the servlet or JSP code that threw the exception to make sure the session is not being invalidated too early. If that is not the case, check the session timeout interval to ensure it is not too short.

Look at available online support

If these steps do not identify the problem, the following resources can be of help: 򐂰 Review the Java Servlet Specification Version 2.4, Section SRV.15.1.7 related to the HttpSession interface and methods definitions, to obtain more details for IllegalStateException creation causes: http://jcp.org/aboutJava/communityprocess/final/jsr154/index.html

182 WebSphere Application Server V6 Problem Determination for Distributed Platforms

򐂰 Look up the extended error definitions for session and user profiles messages: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.messages.doc/doc/SESN.html

If these actions do not identify the problem, go to "The next step" on page 209 forinformation about gathering the MustGather documentation for session andsession management problems.

Response generation problems

When an HTTP request from a client is delegated to a servlet, the service()method of the HttpServlet class is invoked. The HttpServlet class adds additionalmethods, such as doGet(), doPost(), doPut() and doHead(), for HTTP-basedrequest processing. Figure 5-8 illustrates the life cycle of the service() method.

Retrieve initialization parameters

Retrieve request headers

Invoke RequestDispacher Retrieve request parameters

Generate response object Send response by servlet

Send response by web resource Execute sendRedirect method

No Response already committed ? Display page in browser Response already committed ?

Yes

Throw IllegalStateException Throw IllegalStateException

Figure 5-8 Servlet service() method life cycle

The java.lang.IllegalStateException: Response already committed

exception is thrown by the HttpServletResponse interface during the responsegeneration process (Example 5-8 on page 184). If the response has beencommitted, you cannot execute any method that is related to

Chapter 5. Web container problem determination 183

HttpServletResponse object modification. For example, if you have written something in the response buffer, you cannot forward a page using the RequestDispacher interface methods.

Example 5-8 IllegalStateException in response generation

[7/8/05 20:36:25:694 ART] 0000004f ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet TestServlet. Exception thrown : java.lang.IllegalStateException at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendRedirect(WebAppDispa tcherContext.java:486) at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendRedirect(SRTServletResponse. java:810) at web.TestServlet.doGet(TestServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)

[7/8/05 20:36:25:764 ART] 0000004f LocalTranCoor E WLTC0017E: Resources

rolled back due to setRollbackOnly() being called. [7/8/05 20:36:25:774 ART] 0000004f WebApp E SRVE0026E: [Servlet Error]-[TestServlet]: java.lang.IllegalStateException at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendRedirect(WebAppDispa tcherContext.java:486) at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendRedirect(SRTServletResponse. java:810) at web.TestServlet.doGet(TestServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)

[7/8/05 20:36:25:784 ART] 0000004f SRTServletRes W WARNING: Cannot set

status. Response already committed.

Other issues to look for in an application that can cause a

java.lang.IllegalStateException are the following calls when the response has already been committed: 򐂰 Calling setBufferSize() 򐂰 Calling ServletResponse.reset() or ServletResponse.resetBuffer() 򐂰 Calling either HttpServletResponse.sendError() or HttpServletResponse.sendRedirect(). 򐂰 Calling RequestDispatcher.forward(), which includes performing a jsp:forward

184 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Note: Remember that if you call forward() or sendRedirect() in your code, any lines of code following these will still execute.

Two common variants of this exception are:

򐂰 java.lang.IllegalStateException: Header already sent One or more headers have been committed to the client, so you cannot set that header again.򐂰 java.lang.IllegalStateException: Cannot forward as Output Stream or Writer has already been obtained The calling servlet has called response.getWriter() or response.getOutputStream(). Because the response has been written, it is unsuitable for forwarding.

Look at available online support

If you still have not identified the cause of the problem, see the Java ServletSpecification Version 2.4 at the following URL to obtain more details forIllegalStateException generation causes in response generation process:http://jcp.org/aboutJava/communityprocess/final/jsr154/index.html

Review the following sections:

򐂰 SRV.14.2.5 RequestDispatcher interface򐂰 SRV.14.2.16 ServletRequest interface򐂰 SRV.14.2.22 ServletResponse interface

If these steps do not resolve your problem, go to "The next step" on page 209 forinformation about gathering the MustGather documentation for servlet engineand Web container problems.

Chapter 5. Web container problem determination 185

5.3 Analyzing problem areas Your analysis of the data you gathered will most likely lead you to one of the following areas. If not, please see "The next step" on page 209.

5.3.1 Application URL specification

In a new installation or new application, it is possible that the URL of the application is not being specified correctly. To determine the URL of the installed application, you need to use the administrative console to view the configuration of a number of items.

The format of the URL is as follows:

Figure 5-9 Listing the virtual host for a Web module

186 WebSphere Application Server V6 Problem Determination for Distributed Platforms

To find the host and port numbers that are valid for the virtual host:1. Select Environment → Virtual Hosts.2. Choose the virtual host, and then under Additional Properties, click Host Aliases.3. The list contains the host name and port combinations that can be used to access this virtual host (Figure 5-10). The host column should contain values that are registered in a DNS server as a host name for the WebSphere server. An asterisk (*) in the host column indicates that any name can be used. In this case, use the server host name. If port 80 is listed, usually the request is being forwarded from a Web server. (The user specified the URL of the Web server, which is normally 80.) The WC_defaulthost (for example, 9080) and WC_defaulthost_secure (for example, 9443) ports for the application server should also be listed. You can see the corresponding ports by looking at the list of ports in the Communications section of the application server. Use either the host/port combination that accesses the virtual host through the Web server or the host/port combination for using WC_* ports to access the application directly through the application server.

Figure 5-10 Finding the alias and ports for the virtual host

Chapter 5. Web container problem determination 187

If you need SSL, you can check the Web container transport chains for the corresponding ports to see if SSL has been enabled by doing the following: a. Select Servers → Application servers. b. Select the application server name. c. Under Container Settings, open the Web Container Settings list. d. Click Web container transport chains and check the status and SSL enablement for the port that is specified in the virtual host alias.

Figure 5-11 Finding the context root

188 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Important: Remember that the Web container context root value "/" is used by the sample application named DefaultApplication. For this reason, if you need use this value and this sample application is installed, first uninstall the sample application, install your application, and regenerate the Web server plug-in.

Example 5-9 Servlets and welcome files in Web deployment descriptor

Chapter 5. Web container problem determination 189

5.3.2 Static resources not displayed

If the browser displays text output that is related to a JSP or servlet Web page but images or HTML files are not displayed, you could have a problem in the Web module packaging or in how the files are referenced within the application. Another possibility is that the file serving enablement feature needs to be turned on for your application.

Verify the static resource file locations

The first step is to verify that your files are in the right place and that the document root directory of the Web application module follows the J2EE standard, that is that the document root is in the .war directory of the deployed application ear file. Typically this directory is in this location: /profiles//.ear/ .war/

If the image files are in a subdirectory of the document root, verify that the reference to the image reflects that, as shown in Example 5-11.

Example 5-11 Image reference in HTML tags

File Location: /images/test.gif Correct HTML tag:Incorrect HTML tag:

190 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Note: Do not place files to be served to the client in the WEB-INF directory.

Check the file serving enablement feature

File serving allows a Web application to serve static file types (HTML, images,and style sheets) using the enable file servlet, also known as the file servingservlet or file serving enabler. This servlet serves up any resource file that ispackaged in the WAR file, and the file serving enabled attribute is set to true bydefault. For more information about this feature, see:http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/cweb_flserv.html

This behavior is implemented by setting the fileServingEnabled property to

true when configuring the Web module. If it is set to false, the Web serverplug-in does not send requests for static content to the application server butleaves it up to the Web server to serve them. This is sometimes done forperformance reasons to ensure that the content is served by the Web server.Serving the page from the Web server provides a shorter path to the page andusually provides more customization options than the file servlet can offer.

For more information, see:

򐂰 Customizing SimpleFileServlet to disable file serving at: http://www.ibm.com/support/docview.wss?uid=swg21116838򐂰 Supported assembly tools in WebSphere Application Server V6: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/catk_assemblytools.html

The fileServingEnabled property is located in the ibm-web-ext.xmi configuration

file at:/profiles//installedApps//.ear\.war/WEB-INF/ibm-web-ext.xmi

To update the property using the Application Server Toolkit or a Rational

Developer tool:1. Go to the J2EE Hierarchy view (Project Explorer in Rational), and select the target Web application module.2. Double-click the Web deployment descriptor (web.xml), and select the Extensions tab to see the IBM Web module extensions (Figure 5-12 on page 192).3. Under the General section, select or deselect File serving enabled to enable or disable the static file serving.

Chapter 5. Web container problem determination 191

4. Save the Web deployment descriptor file. 5. Redeploy the Web module. 6. Regenerate the Web server plug-in, and propagate it to the Web server. 7. Stop and restart the Web server or allow enough time for the new plug-in configuration to be reloaded. 8. Retry the Web request.

Figure 5-12 IBM Web module extensions

5.3.3 Web resources not reloading

If, after modifying and saving a servlet or JSP file, the change does not show up in the browser, you need to check the reload settings in the Web module configuration and the JSP runtime reload settings.

192 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Web module reloadingFor Web resources such as servlets and JSPs, the Web container reloads a Webmodule only when the IBM extension reloadingEnabled in the ibm-web-ext.xmifile is set to true. You can set reloadingEnabled to true when editing the Webmodule's extended deployment descriptors in an assembly tool.

To review these settings with the Application Server Toolkit or Rational

Application Developer:1. Go to the J2EE Hierarchy view (Project Explorer for Rational) and select the target Web application module.2. Double-click the Web deployment descriptor (web.xml), and select the Extensions tab to see the IBM Web module extensions (Figure 5-12 on page 192).3. In the General section, check the Reloading Enabled flag and the Reload Interval value.4. Select Reloading Enabled, or if it is already selected, then set the Reload Interval lower.5. Save the Web deployment descriptor file.6. Redeploy the Web module.7. Regenerate the Web server plug-in and propagate it to the Web server.8. Stop and restart the Web server or wait for the new plug-in file to take effect.9. Retry the Web request.

The entries in ibm-web-ext.xmi look similar to that shown in Example 5-12.

Example 5-12 Web module reloading settings in ibm-web-ext.xmi

Configuring JSP runtime reloading

You have the ability to modify the JSP processor behavior for different JSPstages (such as development, testing, or production environments). This done byconfiguring specific attributes in the IBM Web module extensions that affect theJSP runtime reload behavior.

Chapter 5. Web container problem determination 193

To review these settings with the Application Server Toolkit or Rational Application Developer: 1. Go to the J2EE Hierarchy view (Project Explorer for Rational), and select the target Web application module. 2. Double-click in Web deployment descriptor (web.xml), and select the Extensions tab. 3. Under the JSP Attributes section (Figure 5-13), click Add. 4. Enter reloadEnabled in the Name field, true in the Value field, and click Finish. 5. Save the Web deployment descriptor file. 6. Redeploy the Web module. 7. Regenerate the Web server plug-in and verify it is working (see "Verify that the Web server plug-in is working" on page 173). 8. Stop and restart the Web server. 9. Retry the Web request.

Figure 5-13 JSP attributes in IBM Web module extensions

194 WebSphere Application Server V6 Problem Determination for Distributed Platforms

The JSP engine settings are stored in ibm-web.ext.xmi, as shown in Example 5-13.

Example 5-13 JSP engine settings in ibm-web-ext.xmi

5.3.4 Encoding and internationalization issues

Encoding and internationalization problems usually surface as garbage characters in Web pages (Figure 5-14 on page 196). Another common symptom is that input from the user is interpreted incorrectly.

Chapter 5. Web container problem determination 195

Figure 5-14 Web page with invalid character encoding settings

Character encoding in J2EE environment

Web components usually use java.io.PrintWriter object to produce responses; PrintWriter automatically encodes using ISO 8859-1 character encoding. Servlets can also output binary data using java.io.OutputStream objects, which perform no encoding. An application that cannot use the default encoding must explicitly set a different encoding.

For Web components, three encodings must be considered:

196 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Request encodingRequest encoding is the character encoding in which parameters in an incomingrequest are interpreted. Currently, many browsers do not send a requestencoding qualifier with the content-type HTTP header. In such cases, a Webcontainer uses the default encoding: ISO-8859-1 to parse request data. If theclient has not set character encoding and the request data is encoded with adifferent encoding from the default, the data will not be interpreted correctly.

To correct this situation, you can use the setCharacterEncoding(String enc)

method to override the character encoding supplied by the container, as shownin Example 5-14.

Example 5-14 setCharacterEncoding() method implementation

public class TestServlet extends HttpServlet {

public void doPost(HttpServletRequest req, HttpServletResponse resp)

throws ServletException, IOException {

You must call the method before parsing any request parameters or reading anyinput from the request. Calling the method or tag after data has been read will notaffect the encoding.

If none of these is provided, ISO-8859-1 is used as the default page encoding.

Chapter 5. Web container problem determination 197

The pageEncoding and contentType attributes determine the page character

encoding of only the file that physically contains the page directive.

If none of these is provided, ISO-8859-1 is used as the default response

The setContentType() method in a servlet can be called to change the character

encoding. Calls made after the getWriter() method has been called or after the response is committed have no effect on the character encoding.

MVC/Struts character encoding settings

When using a Model-View-Controller architecture, including the use of Struts, consider the following for encoding.

198 WebSphere Application Server V6 Problem Determination for Distributed Platforms

Request character encodingIn a Struts environment, call the setCharacterEncoding() method in theActionForm, as shown in Example 5-16.

Example 5-16 Request encoding in Struts implementation

.public class PostMessageForm extends ActionForm { public void reset(ActionMapping mapping, HttpServletRequest request) { try { request.setCharacterEncoding("UTF-8"); }catch (UnsupportedEncodingException e) { e.printStackTrace(); } }}.

Response character encoding

Specify the response encoding using the contentType attribute with a charsetvalue in the JSP page directive.

WebSphere Application Server configuration

WebSphere determines the character encoding used for request/response byparsing the client input values in getParameter() and writing the output per thevalue in the accept-language header of the incoming request.

The language value and corresponding character encoding names are

associated in the encoding.properties file, which is located in the/properties directory (Example 5-17).

Example 5-17 encoding.properties file

Chapter 5. Web container problem determination 199

There might be cases where you want to override the definition in encoding.properties. For example, when you want to use UTF-8 for the entire application server, use the JVM command line argument client.encoding.override for the selected application server.

Specify -Dclient.encoding.override=UTF-8 for the generic JVM arguments

field in the Java Virtual Machine section.

For more information, see:

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/trun_svr_utf.html

autoRequestEncoding and autoResponseEncoding

Starting with WebSphere Application Server Version 5, the Web container no longer sets request and response encodings and response content types automatically. The default value for both extensions is false, then the request and response character encoding is set to the Servlet 2.4 Specification default, which is ISO-8859-1.

Use an assembly tool (Figure 5-12 on page 192) to change the default values for the autoRequestEncoding and autoResponseEncoding extensions.

Review the autoRequestEncoding and autoResponseEncoding encoding

examples for a description of Web container behavior when these values are set to true: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/rweb_autoreq.html

200 WebSphere Application Server V6 Problem Determination for Distributed Platforms

SummaryFigure 5-15 shows a summary of how the encoding values are determined.

Application uses explicit Java no

methods for character encoding yes

autoRequestEncoding and false

autoResponseEncoding values Servlet 2.4 methods JSP 2.0 attributes true

setCharacterEncoding() contextType autoRequestEncoding procedure

setContextType() pageEncoding autoResponseEncoding procedure

Java Servlet 2.4 Spec default

Figure 5-15 Encoding determination process

Look at available online support

If none of these steps fixes your problem, the following resources might behelpful:򐂰 Globalize your On Demand Business for tips on character encoding: http://www-306.ibm.com/software/globalization/j2ee/encoding.jsp򐂰 The Java Servlet Specification Version 2.4 for details about the character encoding available methods: http://jcp.org/aboutJava/communityprocess/final/jsr154/index.html򐂰 The JavaServer Pages Specification Version2.0, section JSP.4 that are related to internationalization issues: http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html򐂰 Internationalization: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/rin_resources.html

Chapter 5. Web container problem determination 201

For current information available from IBM Support on known issues and resolutions that are related to encoding, see: http://www.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc


PREV: About basic forms - Power Apps | Microsoft Docs

NEXT: Client-Server Architecture - an overview | ScienceDirect Topics

Popular Articles

Hot Articles

Navigation Lists

Back to Top