Agent dies

Posted January 14, 201411 yr

~~On my two home Ubuntu 12.04 boxes (VMs under esxi 5.1), the agent runs great.~~ (Also seeing the issue on these servers as well, see post below)

On the server I lease, I cant get it to stay alive.

I am seeing these errors:

user@server:/var/log$ sudo cat syslog | grep pcmonitor
Jan 13 22:38:51 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x0000000000d07ea0 ***
Jan 13 22:40:03 server pcmonitor: Error while registering computer:Empty response
user@server:/var/log$ date
Mon Jan 13 22:42:45 EST 2014
user@server:/var/log$ sudo cat syslog.1 | grep pcmonitor
Jan 12 13:21:47 server pcmonitor: Received request Resume notification 'COMPUTER_OFFLINE_FAA5378C-02CB-4D5F-B610-7C737C6CA6CC' 
Jan 12 13:37:35 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x00007fd1a4029220 ***
Jan 12 14:07:04 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x0000000000f61730 ***
Jan 12 14:12:35 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x00007fc0980290d0 ***
Jan 12 16:03:12 server pcmonitor: Error while registering computer:Empty response

Any suggestions? I installed the updated agent all at the same time, so I am doubtful that it's related to the install but I will re-install tomorrow to be sure.

Thanks,

Brian

Edited January 16, 201411 yr by BrianAz

January 15, 201411 yr

Hello.

Please give us more info about your machine .

Result of uname -a and your config file (please remove sensitive information) would be useful.

January 16, 201411 yr

Author

Hi Calin, thanks for taking a look.

uname -a (sanitized)

brianaz@myserver:~$ uname -a
Linux myserver.ip-xxx-xxx-xx.net 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

My config is attached.

No issue since running it with valgrind yesterday:

root 2692 0.4 1.0 195820 82488 ? Ssl Jan14 6:18 /usr/bin/valgrind.bin /usr/sbin/pcmonitord -v

config.xml

January 16, 201411 yr

Author

The agent on one of my ESXI 5.1 Ubuntu VMs died a couple times today. My leased server is still going strong since I started it with valgrind though. Going to try that on my server below now. Thanks for looking into the issue.

Config is identical to server in post above (see attachment).

brian@PlexMediaServer:~$ uname -a
Linux PlexMediaServer 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

cat /var/log/syslog.1 | grep pcmonitor

Jan 14 13:20:33 PlexMediaServer pcmonitor: Error while registering online:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012"Error observed by underlying SSL/TLS BIO: Connection reset by peer"#012Detail: SSL_connect error in tcp_connect()
Jan 14 13:21:56 PlexMediaServer pcmonitor: Error while registering computer:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012""#012Detail: connect failed in tcp_connect()
Jan 14 13:22:59 PlexMediaServer pcmonitor: Error while registering computer:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012""#012Detail: connect failed in tcp_connect()
Jan 14 13:23:06 PlexMediaServer pcmonitor: Error while registering computer:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012""#012Detail: connect failed in tcp_connect()
Jan 14 13:24:23 PlexMediaServer pcmonitor: Error while registering computer:Empty response
Jan 14 13:24:23 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 13:24:45 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 22:59:58 PlexMediaServer pcmonitor: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 23:00:18 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 23:00:36 PlexMediaServer pcmonitor: Error while registering computer:Empty response
Jan 14 23:00:52 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 15 03:14:57 PlexMediaServer pcmonitor: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">#015#012<HTML><HEAD><TITLE>Service Unavailable</TITLE>#015#012<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>#015#012<BODY><h2>Service Unavailable</h2>#015#012<hr><p>HTTP Error 503. The service is unavailable.</p>#015#012</BODY></HTML>#015#012
Jan 15 03:15:17 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 15 03:16:27  pcmonitor: last message repeated 3 times
Jan 15 03:16:28 PlexMediaServer pcmonitor: Error while registering computer:Empty response

January 16, 201411 yr

The last output you included seems to point to some connection issues. When you said it died , did you mean crashed or was it just offline do to connection issues ?

January 16, 201411 yr

Author

I was watching my iPhone and noticed the server go grey on the PCMonitor app. Upon investigating I noticed I could still connect to it via ssh/VNC w/o issue so I ran "/etc/init.d/pcmonitor status" and it told me the agent was running. I let everything sit and roughly 5-10 min later the server had not come back online (per pcmonitor), I ran status again and the agent was no longer running. I have no other evidence of connection issues around the same time, so I think the agent was crashing.

I then installed valgrind and ran the agent through it as mentioned in the similar thread in the Bugs forum and it's been up and running great ever since.

2 weeks later...

January 29, 201411 yr

Hi ,

Can you post the output of:

ldd /usr/sbin/pcmonitord ?

Also , is there any core file created in /var/pcmonitor ? If so , please email it to us, otherwise first set the core limit to unlimited by ulimit -c unlimited , then try to run the program again and see if the core file appears

January 30, 201411 yr

Author

Here is the output from ldd.

user@SB:~$ sudo ldd /usr/sbin/pcmonitord
	linux-vdso.so.1 =>  (0x00007ffff41ff000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc045994000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc04578c000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc045587000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc045370000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc045070000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc044d73000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc0449b3000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc04479d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc045bbb000)

I am attempting to start the agent, but it's not working:

user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ sudo /etc/init.d/pcmonitor start
user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ ls -la /var/pcmonitor/
total 16
drwxr-xr-x  2 root root 4096 Jan  9 22:39 .
drwxr-xr-x 15 root root 4096 Jan 27 12:22 ..
-rw-r--r--  1 root root   68 Jan  9 22:39 pcmonitor.id
-rw-rw-rw-  1 root root   10 Jan  9 22:39 pcmonitor.ver2

user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ sudo /etc/init.d/pcmonitor start
user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ sudo tail /var/log/syslog

Jan 30 16:46:01 SB CRON[3469]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:47:01 SB CRON[3520]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:48:01 SB CRON[3572]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:49:01 SB CRON[3961]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:50:01 SB CRON[4021]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:50:38 SB pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x00007f6fd0029480 ***

Jan 30 16:51:01 SB CRON[4111]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:51:30 SB pcmonitor: Error while registering computer:Empty response

Jan 30 16:51:33 SB pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x0000000000e878d0 ***

Jan 30 16:52:01 SB CRON[4225]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

user@SB:~$ sudo /etc/init.d/pcmonitor status

 * pcmonitor is not running

So I started it manually with valgrind... and it works:

user@SB:~$ sudo valgrind /usr/sbin/pcmonitord -v > /dev/null &
user@SB:~$ ps aux | grep pcmonitor
root      4682  4.1  0.7 176448 61964 ?        Ssl  16:58   0:03 /usr/bin/valgrind.bin /usr/sbin/pcmonitord -v

For your second question re: ulimit and core files. I did not find any core files and interestingly, once I set it to unlimited, pcmonitor started via the script.. if it crashes, I'll look for the core file again.

user@SB:~$ ps aux | grep pcmonitor

1000      5250  0.0  0.0   8592   932 pts/1    S+   17:03   0:00 grep --color=auto pcmonitor

user@SB:~$ sudo ulimit -c unlimited

sudo: ulimit: command not found

user@SB:~$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62941

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62941

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

user@SB:~$ ulimit -c unlimited

user@SB:~$ ulimit -a

core file size          (blocks, -c) unlimited

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62941

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62941

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

user@SB:~$ sudo /etc/init.d/pcmonitor status

 * pcmonitor is not running

user@SB:~$ sudo /etc/init.d/pcmonitor start

user@SB:~$ ls -la /var/pcmonitor/

total 16

drwxr-xr-x  2 root root 4096 Jan  9 22:39 .

drwxr-xr-x 15 root root 4096 Jan 27 12:22 ..

-rw-r--r--  1 root root   68 Jan  9 22:39 pcmonitor.id

-rw-rw-rw-  1 root root   10 Jan  9 22:39 pcmonitor.ver2

user@SB:~$ sudo /etc/init.d/pcmonitor status

 * pcmonitor is running

user@SB:~$

Thanks, appreciate the help!

1 year later...

April 9, 201510 yr

I am also having same problem..

Apr 8 04:03:53 test pulseway: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Apr 9 04:40:54 test pulseway: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">#015#012<HTML><HEAD><TITLE>Service Unavailable</TITLE>#015#012<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>#015#012<BODY><h2>Service Unavailable</h2>#015#012<hr><p>HTTP Error 503. The service is unavailable.</p>#015#012</BODY></HTML>#015#012

April 13, 201510 yr

Staff

Hi,

If you restart the Pulseway service do you still see the same message? Does the agent crash after showing this error message?

Regards,

Chris

Pulseway Support

Create an account or sign in to comment

https://forum.pulseway.com/topic/1042-agent-dies/

Followers

Go to topic listing

Sign In

Agent dies

Featured Replies

Create an account or sign in to comment

Account

Navigation

Search

Configure browser push notifications

Chrome (Android)

Chrome (Desktop)

Safari (iOS 16.4+)

Safari (macOS)

Edge (Android)

Edge (Desktop)

Firefox (Android)

Firefox (Desktop)