Linux pcmonitor daemon stopped working

[beavisrp]

Hey guys,

I'm having this problem since a few hours ago. (5 to be precise). The dashboard and the mobile app notified me that one of the servers is offline. Went to check on it and it was fully operational, even though PcMonitor was saying otherwise. I tried restarting the daemon but no change. 3 of the server cores are at 100%.

Here is a screenshot from my webmin running processes.

if I run /etc/init.d/pcmonitor stop , the daemon is still running and the dashboard receives a "server shutting down" notification. I tried rebooting the server but the problem persisted.

I am thinking that some data got corrupted and pcmonitor has no idea what do to ... so it infinite loops.

[Calin]

Hello,

Could you please verify the pcmonitor.log ?

[beavisrp]

Here it is.

24/06/2012 09:48:37.000726 - INFO - Application started successfully

25/06/2012 05:00:19.000579 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@5870883)

25/06/2012 05:00:49.000734 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@4d796a5f)

25/06/2012 05:01:04.000787 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@459b80d6)

25/06/2012 11:27:50.000122 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@1b1ed549)

25/06/2012 11:28:04.000915 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@481bbecc)

26/06/2012 05:00:02.000969 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@eef6b04)

26/06/2012 05:00:17.000971 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@18a2f49)

27/06/2012 05:00:10.000066 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@651b597d)

27/06/2012 05:00:25.000299 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@2434da3c)

28/06/2012 05:00:08.000224 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@3b77329a)

28/06/2012 05:00:23.000224 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@1e15a466)

28/06/2012 05:00:53.000224 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@6d91d2fc)

29/06/2012 05:00:02.000784 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@1ec0e07f)

29/06/2012 05:00:32.000607 - ERROR - Error registering computer : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@680c52cb)

29/06/2012 05:00:47.000784 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@3a093f41)

29/06/2012 07:29:06.000888 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@4f647002)

30/06/2012 05:00:07.000385 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@29326f97)

30/06/2012 05:00:22.000364 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@751d4d1d)

30/06/2012 08:00:07.000400 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@634a1fe3)

30/06/2012 08:00:28.000966 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@3c867f0)

30/06/2012 08:19:10.000417 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@3e95d3fd)

30/06/2012 08:28:22.000468 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@2e08ac71)

30/06/2012 08:28:52.000427 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@78399d25)

30/06/2012 08:59:37.000518 - ERROR - Error registering computer : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@5187a5c)

30/06/2012 08:59:52.000488 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@5922afa2)

30/06/2012 09:04:51.000553 - ERROR - Error registering computer : java.net.SocketTimeoutException: connect timed out

30/06/2012 09:05:11.000612 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 09:05:31.000633 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 09:06:14.000166 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@60f3886)

30/06/2012 09:06:34.000393 - ERROR - Couldn't register online : java.net.SocketTimeoutException: Read timed out

30/06/2012 09:06:48.000862 - ERROR - Error registering computer : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@74db9fbd)

30/06/2012 09:07:24.000415 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:31:23.000688 - ERROR - Couldn't register online : java.net.UnknownHostException: ws13.pulseway.com

30/06/2012 02:31:23.000693 - ERROR - Error registering computer : java.net.UnknownHostException: ws13.pulseway.com

30/06/2012 02:32:58.000775 - ERROR - Error registering computer : java.net.UnknownHostException: ws13.pulseway.com

30/06/2012 02:32:58.000776 - ERROR - Error registering computer : java.net.UnknownHostException: ws13.pulseway.com

30/06/2012 02:34:33.000855 - ERROR - Couldn't register online : java.net.UnknownHostException: ws13.pulseway.com

30/06/2012 02:34:33.000856 - ERROR - Error registering computer : java.net.UnknownHostException: ws13.pulseway.com

30/06/2012 02:36:08.000935 - ERROR - Couldn't register online : java.net.UnknownHostException: ws13.pulseway.com

30/06/2012 02:36:08.000936 - ERROR - Error registering computer : java.net.UnknownHostException: ws13.pulseway.com

30/06/2012 02:37:04.000538 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:37:24.000563 - ERROR - Error registering computer : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:37:45.000002 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:38:05.000019 - ERROR - Error registering computer : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:38:25.000281 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:38:45.000302 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:39:10.000565 - ERROR - Error registering computer : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:39:30.000587 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:40:00.000845 - ERROR - Error registering computer : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:40:20.000865 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:41:05.000919 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:41:25.000940 - ERROR - Error registering computer : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:41:45.000993 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:42:06.000015 - ERROR - Error registering computer : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:43:21.000102 - ERROR - Couldn't register online : java.net.SocketTimeoutException: connect timed out

30/06/2012 02:43:41.000123 - ERROR - Error registering computer : java.net.SocketTimeoutException: connect timed out

30/06/2012 03:42:13.000137 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@22ac3118)

30/06/2012 03:42:43.000305 - ERROR - Error registering computer : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@4b9e4ff8)

30/06/2012 07:34:28.000205 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@64707b02)

30/06/2012 07:34:43.000384 - ERROR - Error registering computer : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@7cec2267)

30/06/2012 07:59:43.000403 - ERROR - Error registering computer : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@70205513)

30/06/2012 07:59:58.000444 - ERROR - Couldn't register online : org.xmlpull.v1.XmlPullParserException: unexpected type (position:TEXT The service is u...@1:27 in java.io.InputStreamReader@675071e9)

30/06/2012 09:39:07.000458 - ERROR - Invalid mac received.

01/07/2012 12:24:55.000309 - WARNING - ShutdownHook started

01/07/2012 12:25:50.000431 - WARNING - ShutdownHook started

01/07/2012 12:26:52.000439 - WARNING - ShutdownHook started

01/07/2012 12:37:08.000591 - WARNING - ShutdownHook started

01/07/2012 05:58:09.000129 - WARNING - ShutdownHook started

[dan.lee]

I had the exact same problem and believe it or not, it turned out to be the July 1 leap second issue described here:

http://artipc10.vub.ac.be/wordpress/2012/07/01/leap-second-causing-ksoftirqd-and-java-to-use-lots-of-cpu-time/

I didn't figure it out until I was using Eclipse to build a plug-in and it was dog slow too. A bit of googling pointed me in the right direction. Hope that helps you!

Dan

[beavisrp]

Worked like a charm. Thank you Dan.