Changes for page Troubleshooting Deployment

Summary

• Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

...	...	@@ -23,106 +23,120 @@
23	23	* Use the WOLongReponsePage to handle long running requests.
24	24	* Increase the Connect Timeout and Receive Timeout values in JavaMonitor so that the woadaptor will wait long enough for your application to provide the response.
25	25
26		-{{Note}}{{/Note}}
	26	+{{note}}
27	27
28	28	If this message happens in other contexts (i.e. not when a request takes a long time to process) it might just mean that the user hit stop in their browser or clicked another link.
29	29
30		-{{note}}
	30	+{{/note}}
31	31
	32	+== Where's my stderr? ==
32	32
33		-h2. Where's my stderr\!?
34		-
35	35	Wotaskd launches new WOA instances using a script called SpawnOfWotaskd.sh that is located in /System/Library/WebObjects/JavaApplications/wotaskd.woa/Contents/Resources/SpawnOfWotaskd.sh on OS X. For some reason, this script was written to throw away stdout and redirrect stderr to stdout. This means that if you ever want to get a thread stack dump, you're out of luck. Fortunately this is an easy fix. If you edit SpawnOfWotaskd.sh, the stock version looks like:
36		-{code}
	35	+
	36	+{{code}}
	37	+
37	37	#!/bin/sh
38	38	$@ 1>/dev/null 2>&1 &
39		-{code}
	40	+
	41	+{{/code}}
	42	+
40	40	Notice that stream 1 goes to /dev/null, and stream 2 writes to stream 1. Boo. Change it to:
41		-{code}
	44	+
	45	+{{code}}
	46	+
42	42	#!/bin/sh
43	43	$@ 1>>/var/log/webobjects.err 2>&1 &
44		-{code}
	49	+
	50	+{{/code}}
	51	+
45	45	The one problem now is that typically WebObjects apps run as appserver user, which means they won't be able to write to this file. To fix this, you can 'touch /var/log/webobjects.err' as root, and then chown the blank file to whatever user your WebObjects apps run as.
46	46
47		-h2. Can't connect to window server - not enough permissions
	54	+== Can't connect to window server - not enough permissions ==
48	48
49	49	Ah yes, the joys of AWT. If you touch any AWT class (event the ones that don't make sense, like Dimension, or Rectangle), an AWT Toolkit will be created (in a static block) and AWT will attempt to connect to the window server. Of course when you ran in development mode, you had access to the window server and everything was peachy. As soon as you tried to deploy as the appserver user, all hell broke loose. You have a couple of options for fixing this one:
	57	+
50	50	* Just stop using them. This is the most sure fire way to not get hit by this problem. Don't use AWT classes when you don't need to. Obviously this is often easier said than done.
51		-* Add \-Djava.awt.headless=true to your VM launch parameters. This tells AWT to use headless mode. If you are using older versions of Apple's JVM, this may still cause issues (weird debug statements requesting for you to hit a key on the console).
52		-* [PJA Toolkit|http://www.eteks.com/pja/en/]. This is actually kind of a cool library. It's a pure java implementation of an AWT toolkit. On a normal system, your AWT implementation is JNI (to talk to your native window system). PJA provides pura java implementations of all of the capabilities a normal AWT toolkit would attempt to perform. Obviously it doesn't actually use the window server, rather it's working akin to a VNC server with a virtual frame buffer.
	59	+* Add Djava.awt.headless=true to your VM launch parameters. This tells AWT to use headless mode. If you are using older versions of Apple's JVM, this may still cause issues (weird debug statements requesting for you to hit a key on the console).
	60	+* [[PJA Toolkit>>http://www.eteks.com/pja/en/]]. This is actually kind of a cool library. It's a pure java implementation of an AWT toolkit. On a normal system, your AWT implementation is JNI (to talk to your native window system). PJA provides pura java implementations of all of the capabilities a normal AWT toolkit would attempt to perform. Obviously it doesn't actually use the window server, rather it's working akin to a VNC server with a virtual frame buffer.
53	53	* Run as root. This is not generally recommended, but you can modify your /System/Library/StartupItems/WebObjects/WebObjects script to run WO as root instead of appserver user. You'll find the instructions on doing so inside of that script.
54	54
55		-h2. Java Monitor Issues
	63	+== Java Monitor Issues ==
56	56
57	57	For those who have problem with the monitor on Mac OS X please check the following parameters:
	66	+
58	58	* Check that the machine is properly identified in the DNS including the reverse lookup.
59	59	* Verify that when you added the machine in the monitor you included the fully qualifier name i.e. "machine.domain.com" and not only "machine" using the shorter version may appear to work but it does not (this is documented by Apple).
60	60	* Check the adaptor configuration (/System/Library/WebObjects/Adaptors/Apache/apache.conf) and in particular the instance discovery method: this needs to be in accordance with the command line launch of wotaskd (see the script in /System/Library/StartupItems/). You may need to modify the script by hand.
61	61	* When everything is setup properly reboot. It should work.
62	62
63		-h2. WOtaskd Didn't Start
	72	+== WOtaskd Didn't Start ==
64	64
65	65	In older versions of OS X server, WOtaskd and JavaMonitor were launched using the StartupItems system. In more recent versions of OS X server and WebObjects, Apple uses its new launchd framework instead. If you install WebObjects with developer tools, you typically only end up with the Startupitem scripts, so you should follow those instructions.
66	66
67		-h3. WOtaskd/WOMonitor LaunchDaemon
	76	+=== WOtaskd/WOMonitor LaunchDaemon ===
68	68
69		-For an overview of launchd, you can read [Apple's documentation|http://developer.apple.com/macosx/launchd.html].
	78	+For an overview of launchd, you can read [[Apple's documentation>>http://developer.apple.com/macosx/launchd.html]].
70	70
71	71	The WebObjects launch daemon items are disabled by default, so the first thing you will need to do is enable them:
72		-# cd /System/Library/LaunchDaemons
73		-# edit "com.apple.womonitor.plist" and "com.apple.wotaskd.plist"
74		-# change the Disabled key from "true" to "false"
75		-# save and exit
76	76
	82	+1. cd /System/Library/LaunchDaemons
	83	+1. edit "com.apple.womonitor.plist" and "com.apple.wotaskd.plist"
	84	+1. change the Disabled key from "true" to "false"
	85	+1. save and exit
	86	+
77	77	Next you will need to load the launch daemon items manually:
78		-# Run "launchctl" as root
79		-# "list" and look for com.webobjects.womonitor and com.webobjects.wotaskd
80		-# If they do exist, you will need to load them manually
81		-## "load /System/Library/LaunchDaemons/com.apple.wotaskd.plist"
82		-## "load /System/Library/LaunchDaemons/com.apple.womonitor.plist"
83		-# "list" again and verify they both appear
84		-# "start com.webobjects.wotaskd"
85		-# "start com.webobjects.womonitor"
86		-# Quit launchctl (ctrl-c it)
87	87
88		-wotaskd and womonitor should now both be running. You can test monitor by going to [http://localhost:56789].
	89	+1. Run "launchctl" as root
	90	+1. "list" and look for com.webobjects.womonitor and com.webobjects.wotaskd
	91	+1. If they do exist, you will need to load them manually
	92	+11. "load /System/Library/LaunchDaemons/com.apple.wotaskd.plist"
	93	+11. "load /System/Library/LaunchDaemons/com.apple.womonitor.plist"
	94	+1. "list" again and verify they both appear
	95	+1. "start com.webobjects.wotaskd"
	96	+1. "start com.webobjects.womonitor"
	97	+1. Quit launchctl (ctrl-c it)
89	89
	99	+wotaskd and womonitor should now both be running. You can test monitor by going to [[http://localhost:56789]].
	100	+
90	90	If this does not work, check for file system permissions on /Library/WebObjects/Configuration, which should be appserver:appserverusr or at least they should have write permission to this directory.
91	91
92	92	Running this following command in a terminal window will tell you exactly why wotaskd is not launching...
93		-{code:none|title=Command to Manually Maunch wotaskd}
	104	+
	105	+{{code title="Command to Manually Maunch wotaskd"}}
	106	+
94	94	sudo -u appserver /System/Library/WebObjects/JavaApplications/wotaskd.woa/Contents/Resources/javawoservice.sh \
95	95	-appPath /System/Library/WebObjects/JavaApplications/wotaskd.woa/wotaskd
96		-{code}
97	97
98		-h3. WOtaskd/WOMonitor StartupItems
	110	+{{/code}}
99	99
	112	+=== WOtaskd/WOMonitor StartupItems ===
	113	+
100	100	On developer installs and older Mac OS X Server installs, WebObjects is launched using the old StartupItems system. By default, wotaskd and womonitor are disabled. Fortunately, enabling them is easy:
101		-# cd /System/Library/StartupItems/WebObjects
102		-# Edit "WebObjects"
103		-# Search for "-appPath" and you will find four commented out lines in this area of the script.
104		-#* To run WebObjects as root, uncomment the first two commands (you should see a WOTASKD and a WOMONITOR line)
105		-#* To run WebObjects as another user (appserver user by default), uncomment the second set of the lines
106		-# Save and exit
107		-# Start the WebObjects service using one of the two methods:
108		-#* sudo systemstarter start "WebObjects Services"
109		-#* ./WebObjects start
110	110
	116	+1. cd /System/Library/StartupItems/WebObjects
	117	+1. Edit "WebObjects"
	118	+1. Search for "-appPath" and you will find four commented out lines in this area of the script.
	119	+1*. To run WebObjects as root, uncomment the first two commands (you should see a WOTASKD and a WOMONITOR line)
	120	+1*. To run WebObjects as another user (appserver user by default), uncomment the second set of the lines
	121	+1. Save and exit
	122	+1. Start the WebObjects service using one of the two methods:
	123	+1*. sudo systemstarter start "WebObjects Services"
	124	+1*. ./WebObjects start
	125	+
111	111	After these changes, wotaskd and womonitor will autostart after a reboot.
112	112
113		-Note: Do not follow these instructions on a MacOS X 10.4.\* Server machine with WO 5.3.*. There, the process is started as LaunchDeamon (described above). If you enable the Startup Item, the system tries to start wotaskd twice which leads to nothing working at all.
	128	+Note: Do not follow these instructions on a MacOS X 10.4. Server machine with WO 5.3.**. There, the process is started as LaunchDeamon (described above). If you enable the Startup Item, the system tries to start wotaskd twice which leads to nothing working at all.**
114	114
115		-h2. Dealing with Deadlocks/Application Hanging
	130	+== Dealing with Deadlocks/Application Hanging ==
116	116
117	117	There are several techniques for dealing with deadlocks and application hangs.
118	118
119		-If you are using a [profiler|Programming__WebObjects-Web_Applications-Development-Profiling_WO_Apps] or debugger, you can generally pause execution and see where exactly your application is dying.
	134	+If you are using a [[profiler>>Programming__WebObjects-Web_Applications-Development-Profiling_WO_Apps]] or debugger, you can generally pause execution and see where exactly your application is dying.
120	120
121		-If you are using JDK 1.4, you can send your application a QUIT signal and it will dump all active threads. On OS X, find the pid of your application, then execute a 'kill \-QUIT yourAppPID'. The thread stack traces will dump out to the log files. If you do not see any output in your logs, see the _Where's my stderr\!?_ above for possible reasons.
	136	+If you are using JDK 1.4, you can send your application a QUIT signal and it will dump all active threads. On OS X, find the pid of your application, then execute a 'kill QUIT yourAppPID'. The thread stack traces will dump out to the log files. If you do not see any output in your logs, see the //Where's my stderr?// above for possible reasons.
122	122
123	123	If you are using JDK 1.5, in addition to being to send the application a QUIT signal, you can also attach to it (as root or the launching user) and get the thread stack dumps using the 'jstack' application by calling 'jstack yourAppPID'.
124	124
125	125	If you see hangs in WOWorkerThreads waiting to restore a session, this generally means a previous request threw an exception from a location that didn't properly check its session back in. Session access in WebObjects is single-threaded. Common cases that can cause this are throwing exceptions from your awake methods as well as throwing exceptions from DirectAction methods.
126	126
127		-In JDK 1.4, you will be able to see the list of object locks that each stack trace is holding. You can find deadlocks by looking for multiple threads that contain two or more overlapping locks. In JDK 1.5, the VM detects deadlock situations and will identify them in the thread stack dumps
128		-{{/note}}
	142	+In JDK 1.4, you will be able to see the list of object locks that each stack trace is holding. You can find deadlocks by looking for multiple threads that contain two or more overlapping locks. In JDK 1.5, the VM detects deadlock situations and will identify them in the thread stack dumps.