Agent dies

BrianAz · January 14, 2014

~~On my two home Ubuntu 12.04 boxes (VMs under esxi 5.1), the agent runs great.~~ (Also seeing the issue on these servers as well, see post below)

On the server I lease, I cant get it to stay alive.

I am seeing these errors:

user@server:/var/log$ sudo cat syslog | grep pcmonitor
Jan 13 22:38:51 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x0000000000d07ea0 ***
Jan 13 22:40:03 server pcmonitor: Error while registering computer:Empty response
user@server:/var/log$ date
Mon Jan 13 22:42:45 EST 2014
user@server:/var/log$ sudo cat syslog.1 | grep pcmonitor
Jan 12 13:21:47 server pcmonitor: Received request Resume notification 'COMPUTER_OFFLINE_FAA5378C-02CB-4D5F-B610-7C737C6CA6CC' 
Jan 12 13:37:35 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x00007fd1a4029220 ***
Jan 12 14:07:04 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x0000000000f61730 ***
Jan 12 14:12:35 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x00007fc0980290d0 ***
Jan 12 16:03:12 server pcmonitor: Error while registering computer:Empty response

Any suggestions? I installed the updated agent all at the same time, so I am doubtful that it's related to the install but I will re-install tomorrow to be sure.

Thanks,

Brian

Edited January 16, 2014 by BrianAz

Calin · January 15, 2014

Hello.

Please give us more info about your machine .

Result of uname -a and your config file (please remove sensitive information) would be useful.

BrianAz · January 16, 2014

Hi Calin, thanks for taking a look.

uname -a (sanitized)

brianaz@myserver:~$ uname -a
Linux myserver.ip-xxx-xxx-xx.net 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

My config is attached.

No issue since running it with valgrind yesterday:

root 2692 0.4 1.0 195820 82488 ? Ssl Jan14 6:18 /usr/bin/valgrind.bin /usr/sbin/pcmonitord -v

config.xml

BrianAz · January 16, 2014

The agent on one of my ESXI 5.1 Ubuntu VMs died a couple times today. My leased server is still going strong since I started it with valgrind though. Going to try that on my server below now. Thanks for looking into the issue.

Config is identical to server in post above (see attachment).

brian@PlexMediaServer:~$ uname -a
Linux PlexMediaServer 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

cat /var/log/syslog.1 | grep pcmonitor

Jan 14 13:20:33 PlexMediaServer pcmonitor: Error while registering online:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012"Error observed by underlying SSL/TLS BIO: Connection reset by peer"#012Detail: SSL_connect error in tcp_connect()
Jan 14 13:21:56 PlexMediaServer pcmonitor: Error while registering computer:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012""#012Detail: connect failed in tcp_connect()
Jan 14 13:22:59 PlexMediaServer pcmonitor: Error while registering computer:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012""#012Detail: connect failed in tcp_connect()
Jan 14 13:23:06 PlexMediaServer pcmonitor: Error while registering computer:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012""#012Detail: connect failed in tcp_connect()
Jan 14 13:24:23 PlexMediaServer pcmonitor: Error while registering computer:Empty response
Jan 14 13:24:23 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 13:24:45 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 22:59:58 PlexMediaServer pcmonitor: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 23:00:18 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 23:00:36 PlexMediaServer pcmonitor: Error while registering computer:Empty response
Jan 14 23:00:52 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 15 03:14:57 PlexMediaServer pcmonitor: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">#015#012<HTML><HEAD><TITLE>Service Unavailable</TITLE>#015#012<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>#015#012<BODY><h2>Service Unavailable</h2>#015#012<hr><p>HTTP Error 503. The service is unavailable.</p>#015#012</BODY></HTML>#015#012
Jan 15 03:15:17 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 15 03:16:27  pcmonitor: last message repeated 3 times
Jan 15 03:16:28 PlexMediaServer pcmonitor: Error while registering computer:Empty response

Calin · January 16, 2014

The last output you included seems to point to some connection issues. When you said it died , did you mean crashed or was it just offline do to connection issues ?

BrianAz · January 16, 2014

I was watching my iPhone and noticed the server go grey on the PCMonitor app. Upon investigating I noticed I could still connect to it via ssh/VNC w/o issue so I ran "/etc/init.d/pcmonitor status" and it told me the agent was running. I let everything sit and roughly 5-10 min later the server had not come back online (per pcmonitor), I ran status again and the agent was no longer running. I have no other evidence of connection issues around the same time, so I think the agent was crashing.

I then installed valgrind and ran the agent through it as mentioned in the similar thread in the Bugs forum and it's been up and running great ever since.

Calin · January 29, 2014

Hi ,

Can you post the output of:

ldd /usr/sbin/pcmonitord ?

Also , is there any core file created in /var/pcmonitor ? If so , please email it to us, otherwise first set the core limit to unlimited by ulimit -c unlimited , then try to run the program again and see if the core file appears

BrianAz · January 30, 2014

Here is the output from ldd.

user@SB:~$ sudo ldd /usr/sbin/pcmonitord
	linux-vdso.so.1 =>  (0x00007ffff41ff000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc045994000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc04578c000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc045587000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc045370000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc045070000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc044d73000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc0449b3000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc04479d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc045bbb000)

I am attempting to start the agent, but it's not working:

user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ sudo /etc/init.d/pcmonitor start
user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ ls -la /var/pcmonitor/
total 16
drwxr-xr-x  2 root root 4096 Jan  9 22:39 .
drwxr-xr-x 15 root root 4096 Jan 27 12:22 ..
-rw-r--r--  1 root root   68 Jan  9 22:39 pcmonitor.id
-rw-rw-rw-  1 root root   10 Jan  9 22:39 pcmonitor.ver2

user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ sudo /etc/init.d/pcmonitor start
user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ sudo tail /var/log/syslog

Jan 30 16:46:01 SB CRON[3469]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:47:01 SB CRON[3520]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:48:01 SB CRON[3572]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:49:01 SB CRON[3961]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:50:01 SB CRON[4021]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:50:38 SB pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x00007f6fd0029480 ***

Jan 30 16:51:01 SB CRON[4111]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:51:30 SB pcmonitor: Error while registering computer:Empty response

Jan 30 16:51:33 SB pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x0000000000e878d0 ***

Jan 30 16:52:01 SB CRON[4225]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

user@SB:~$ sudo /etc/init.d/pcmonitor status

 * pcmonitor is not running

So I started it manually with valgrind... and it works:

user@SB:~$ sudo valgrind /usr/sbin/pcmonitord -v > /dev/null &
user@SB:~$ ps aux | grep pcmonitor
root      4682  4.1  0.7 176448 61964 ?        Ssl  16:58   0:03 /usr/bin/valgrind.bin /usr/sbin/pcmonitord -v

For your second question re: ulimit and core files. I did not find any core files and interestingly, once I set it to unlimited, pcmonitor started via the script.. if it crashes, I'll look for the core file again.

user@SB:~$ ps aux | grep pcmonitor

1000      5250  0.0  0.0   8592   932 pts/1    S+   17:03   0:00 grep --color=auto pcmonitor

user@SB:~$ sudo ulimit -c unlimited

sudo: ulimit: command not found

user@SB:~$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62941

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62941

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

user@SB:~$ ulimit -c unlimited

user@SB:~$ ulimit -a

core file size          (blocks, -c) unlimited

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62941

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62941

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

user@SB:~$ sudo /etc/init.d/pcmonitor status

 * pcmonitor is not running

user@SB:~$ sudo /etc/init.d/pcmonitor start

user@SB:~$ ls -la /var/pcmonitor/

total 16

drwxr-xr-x  2 root root 4096 Jan  9 22:39 .

drwxr-xr-x 15 root root 4096 Jan 27 12:22 ..

-rw-r--r--  1 root root   68 Jan  9 22:39 pcmonitor.id

-rw-rw-rw-  1 root root   10 Jan  9 22:39 pcmonitor.ver2

user@SB:~$ sudo /etc/init.d/pcmonitor status

 * pcmonitor is running

user@SB:~$

Thanks, appreciate the help!

neoark · April 9, 2015

I am also having same problem..

Apr 8 04:03:53 test pulseway: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Apr 9 04:40:54 test pulseway: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">#015#012<HTML><HEAD><TITLE>Service Unavailable</TITLE>#015#012<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>#015#012<BODY><h2>Service Unavailable</h2>#015#012<hr><p>HTTP Error 503. The service is unavailable.</p>#015#012</BODY></HTML>#015#012

Chris · April 13, 2015

Hi,

If you restart the Pulseway service do you still see the same message? Does the agent crash after showing this error message?

Regards,

Chris

Pulseway Support

Sign In

Agent dies

Recommended Posts

BrianAz

Calin

BrianAz

BrianAz

Calin

BrianAz

Calin

BrianAz

neoark

Chris

Create an account or sign in to comment

Create an account

Sign in

Browse

Activity