A current app I'm working freezes up for about a minute when calling editingContext.saveChanges. The following message is added to the system output:
<WorkerThread3> <WOWorkerThread id=3 socket=Socket[addr=/xxx.xxx.xxx.xxx,port=51634,localport=51563]> Exception while sending response: java.net.SocketException: Broken pipe
The odd thing is that no exception is actually being thrown and the changes are saved to the database. Any clues out there about what this message means?
This message means that the woadaptor did not receive a response from your application within the alloted amount of time. At this point, the woapdator hangs up on your application assuming that it is dead or overly busy. If you have multiple instances running, the woadaptor will then forward the request to another instance. For component actions this will produce a session expired or can't restore page error message.
When your application finally finishes processing the request it attempts to return the result to the woadaptor only to discover that it has hung up on it (aka closed the socket, broken the pipe). At this point the exception above is output to the log.
There are a few ways to avoid this:
If this message happens in other contexts (i.e. not when a request takes a long time to process) it might just mean that the user hit stop in their browser or clicked another link.
Wotaskd launches new WOA instances using a script called SpawnOfWotaskd.sh that is located in /System/Library/WebObjects/JavaApplications/wotaskd.woa/Contents/Resources/SpawnOfWotaskd.sh on OS X. For some reason, this script was written to throw away stdout and redirrect stderr to stdout. This means that if you ever want to get a thread stack dump, you're out of luck. Fortunately this is an easy fix. If you edit SpawnOfWotaskd.sh, the stock version looks like:
#!/bin/sh $@ 1>/dev/null 2>&1 &
Notice that stream 1 goes to /dev/null, and stream 2 writes to stream 1. Boo. Change it to:
#!/bin/sh $@ 1>>/var/log/webobjects.err 2>&1 &
The one problem now is that typically WebObjects apps run as appserver user, which means they won't be able to write to this file. To fix this, you can 'touch /var/log/webobjects.err' as root, and then chown the blank file to whatever user your WebObjects apps run as.
Ah yes, the joys of AWT. If you touch any AWT class (event the ones that don't make sense, like Dimension, or Rectangle), an AWT Toolkit will be created (in a static block) and AWT will attempt to connect to the window server. Of course when you ran in development mode, you had access to the window server and everything was peachy. As soon as you tried to deploy as the appserver user, all hell broke loose. You have a couple of options for fixing this one:
For those who have problem with the monitor on Mac OS X please check the following parameters:
In older versions of OS X server, WOtaskd and JavaMonitor were launched using the StartupItems system. In more recent versions of OS X server and WebObjects, Apple uses its new launchd framework instead. If you install WebObjects with developer tools, you typically only end up with the Startupitem scripts, so you should follow those instructions.
For an overview of launchd, you can read Apple's documentation.
The WebObjects launch daemon items are disabled by default, so the first thing you will need to do is enable them:
Next you will need to load the launch daemon items manually:
wotaskd and womonitor should now both be running. You can test monitor by going to http://localhost:56789.
If this does not work, check for file system permissions on /Library/WebObjects/Configuration, which should be appserver:appserverusr or at least they should have write permission to this directory.
Running this following command in a terminal window will tell you exactly why wotaskd is not launching...
sudo -u appserver /System/Library/WebObjects/JavaApplications/wotaskd.woa/Contents/Resources/javawoservice.sh \ -appPath /System/Library/WebObjects/JavaApplications/wotaskd.woa/wotaskd
On developer installs and older Mac OS X Server installs, WebObjects is launched using the old StartupItems system. By default, wotaskd and womonitor are disabled. Fortunately, enabling them is easy:
After these changes, wotaskd and womonitor will autostart after a reboot.
Note: Do not follow these instructions on a MacOS X 10.4.* Server machine with WO 5.3.*. There, the process is started as LaunchDeamon (described above). If you enable the Startup Item, the system tries to start wotaskd twice which leads to nothing working at all.
There are several techniques for dealing with deadlocks and application hangs.
If you are using a profiler or debugger, you can generally pause execution and see where exactly your application is dying.
If you are using JDK 1.4, you can send your application a QUIT signal and it will dump all active threads. On OS X, find the pid of your application, then execute a 'kill -QUIT yourAppPID'. The thread stack traces will dump out to the log files. If you do not see any output in your logs, see the Where's my stderr!? above for possible reasons.
If you are using JDK 1.5, in addition to being to send the application a QUIT signal, you can also attach to it (as root or the launching user) and get the thread stack dumps using the 'jstack' application by calling 'jstack yourAppPID'.
If you see hangs in WOWorkerThreads waiting to restore a session, this generally means a previous request threw an exception from a location that didn't properly check its session back in. Session access in WebObjects is single-threaded. Common cases that can cause this are throwing exceptions from your awake methods as well as throwing exceptions from DirectAction methods.
In JDK 1.4, you will be able to see the list of object locks that each stack trace is holding. You can find deadlocks by looking for multiple threads that contain two or more overlapping locks. In JDK 1.5, the VM detects deadlock situations and will identify them in the thread stack dumps.