Jump to content

Agent dies


BrianAz
 Share

Recommended Posts

On my two home Ubuntu 12.04 boxes (VMs under esxi 5.1), the agent runs great. (Also seeing the issue on these servers as well, see post below)

 

 

On the server I lease, I cant get it to stay alive.

 

I am seeing these errors:

user@server:/var/log$ sudo cat syslog | grep pcmonitor
Jan 13 22:38:51 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x0000000000d07ea0 ***
Jan 13 22:40:03 server pcmonitor: Error while registering computer:Empty response
user@server:/var/log$ date
Mon Jan 13 22:42:45 EST 2014
user@server:/var/log$ sudo cat syslog.1 | grep pcmonitor
Jan 12 13:21:47 server pcmonitor: Received request Resume notification 'COMPUTER_OFFLINE_FAA5378C-02CB-4D5F-B610-7C737C6CA6CC' 
Jan 12 13:37:35 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x00007fd1a4029220 ***
Jan 12 14:07:04 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x0000000000f61730 ***
Jan 12 14:12:35 server pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x00007fc0980290d0 ***
Jan 12 16:03:12 server pcmonitor: Error while registering computer:Empty response

Any suggestions? I installed the updated agent all at the same time, so I am doubtful that it's related to the install but I will re-install tomorrow to be sure.

 

Thanks,

Brian

Edited by BrianAz
Link to comment
Share on other sites

Hi Calin, thanks for taking a look.

 

uname -a (sanitized)

brianaz@myserver:~$ uname -a
Linux myserver.ip-xxx-xxx-xx.net 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

My config is attached.

 

No issue since running it with valgrind yesterday:

root      2692  0.4  1.0 195820 82488 ?        Ssl  Jan14   6:18 /usr/bin/valgrind.bin /usr/sbin/pcmonitord -v

config.xml

Link to comment
Share on other sites

The agent on one of my ESXI 5.1 Ubuntu VMs died a couple times today. My leased server is still going strong since I started it with valgrind though. Going to try that on my server below now. Thanks for looking into the issue. 

 

Config is identical to server in post above (see attachment).

brian@PlexMediaServer:~$ uname -a
Linux PlexMediaServer 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
 

 cat /var/log/syslog.1 | grep pcmonitor

Jan 14 13:20:33 PlexMediaServer pcmonitor: Error while registering online:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012"Error observed by underlying SSL/TLS BIO: Connection reset by peer"#012Detail: SSL_connect error in tcp_connect()
Jan 14 13:21:56 PlexMediaServer pcmonitor: Error while registering computer:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012""#012Detail: connect failed in tcp_connect()
Jan 14 13:22:59 PlexMediaServer pcmonitor: Error while registering computer:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012""#012Detail: connect failed in tcp_connect()
Jan 14 13:23:06 PlexMediaServer pcmonitor: Error while registering computer:SOAP 1.1 fault: SOAP-ENV:Client[no subcode]#012""#012Detail: connect failed in tcp_connect()
Jan 14 13:24:23 PlexMediaServer pcmonitor: Error while registering computer:Empty response
Jan 14 13:24:23 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 13:24:45 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 22:59:58 PlexMediaServer pcmonitor: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 23:00:18 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 14 23:00:36 PlexMediaServer pcmonitor: Error while registering computer:Empty response
Jan 14 23:00:52 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 15 03:14:57 PlexMediaServer pcmonitor: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">#015#012<HTML><HEAD><TITLE>Service Unavailable</TITLE>#015#012<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>#015#012<BODY><h2>Service Unavailable</h2>#015#012<hr><p>HTTP Error 503. The service is unavailable.</p>#015#012</BODY></HTML>#015#012
Jan 15 03:15:17 PlexMediaServer pcmonitor: Error while registering computer:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Jan 15 03:16:27  pcmonitor: last message repeated 3 times
Jan 15 03:16:28 PlexMediaServer pcmonitor: Error while registering computer:Empty response
 
Link to comment
Share on other sites

I was watching my iPhone and noticed the server go grey on the PCMonitor app. Upon investigating I noticed I could still connect to it via ssh/VNC w/o issue so I ran "/etc/init.d/pcmonitor status" and it told me the agent was running. I let everything sit and roughly 5-10 min later the server had not come back online (per pcmonitor), I ran status again and the agent was no longer running. I have no other evidence of connection issues around the same time, so I think the agent was crashing.

 

I then installed valgrind and ran the agent through it as mentioned in the similar thread in the Bugs forum and it's been up and running great ever since.

Link to comment
Share on other sites

  • 2 weeks later...
  • Staff

Hi ,

 

Can you post the output of:

ldd /usr/sbin/pcmonitord ?

 

Also , is there any core file created in /var/pcmonitor ? If so , please email it to us, otherwise first set the core limit to unlimited by ulimit -c unlimited , then try to run the program again and see if the core file appears

Link to comment
Share on other sites

Here is the output from ldd.

user@SB:~$ sudo ldd /usr/sbin/pcmonitord
	linux-vdso.so.1 =>  (0x00007ffff41ff000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc045994000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc04578c000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc045587000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc045370000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc045070000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc044d73000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc0449b3000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc04479d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc045bbb000)


I am attempting to start the agent, but it's not working:

user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ sudo /etc/init.d/pcmonitor start
user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ ls -la /var/pcmonitor/
total 16
drwxr-xr-x  2 root root 4096 Jan  9 22:39 .
drwxr-xr-x 15 root root 4096 Jan 27 12:22 ..
-rw-r--r--  1 root root   68 Jan  9 22:39 pcmonitor.id
-rw-rw-rw-  1 root root   10 Jan  9 22:39 pcmonitor.ver2

user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ sudo /etc/init.d/pcmonitor start
user@SB:~$ sudo /etc/init.d/pcmonitor status
 * pcmonitor is not running
user@SB:~$ sudo tail /var/log/syslog

Jan 30 16:46:01 SB CRON[3469]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:47:01 SB CRON[3520]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:48:01 SB CRON[3572]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:49:01 SB CRON[3961]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:50:01 SB CRON[4021]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:50:38 SB pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x00007f6fd0029480 ***

Jan 30 16:51:01 SB CRON[4111]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

Jan 30 16:51:30 SB pcmonitor: Error while registering computer:Empty response

Jan 30 16:51:33 SB pcmonitor: *** glibc detected *** /usr/sbin/pcmonitord: double free or corruption (fasttop): 0x0000000000e878d0 ***

Jan 30 16:52:01 SB CRON[4225]: (root) CMD (/usr/local/rtm/bin/rtm 11 > /dev/null 2> /dev/null)

user@SB:~$ sudo /etc/init.d/pcmonitor status

 * pcmonitor is not running

So I started it manually with valgrind... and it works:

user@SB:~$ sudo valgrind /usr/sbin/pcmonitord -v > /dev/null &
user@SB:~$ ps aux | grep pcmonitor
root      4682  4.1  0.7 176448 61964 ?        Ssl  16:58   0:03 /usr/bin/valgrind.bin /usr/sbin/pcmonitord -v

For your second question re: ulimit and core files. I did not find any core files and interestingly, once I set it to unlimited, pcmonitor started via the script.. if it crashes, I'll look for the core file again.

user@SB:~$ ps aux | grep pcmonitor

1000      5250  0.0  0.0   8592   932 pts/1    S+   17:03   0:00 grep --color=auto pcmonitor

user@SB:~$ sudo ulimit -c unlimited

sudo: ulimit: command not found

user@SB:~$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62941

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62941

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

user@SB:~$ ulimit -c unlimited

user@SB:~$ ulimit -a

core file size          (blocks, -c) unlimited

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62941

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62941

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

user@SB:~$ sudo /etc/init.d/pcmonitor status

 * pcmonitor is not running

user@SB:~$ sudo /etc/init.d/pcmonitor start

user@SB:~$ ls -la /var/pcmonitor/

total 16

drwxr-xr-x  2 root root 4096 Jan  9 22:39 .

drwxr-xr-x 15 root root 4096 Jan 27 12:22 ..

-rw-r--r--  1 root root   68 Jan  9 22:39 pcmonitor.id

-rw-rw-rw-  1 root root   10 Jan  9 22:39 pcmonitor.ver2

user@SB:~$ sudo /etc/init.d/pcmonitor status

 * pcmonitor is running

user@SB:~$ 

 

 

 

 

 

Thanks, appreciate the help!

Link to comment
Share on other sites

  • 1 year later...

I am also having same problem..

 

 

Apr  8 04:03:53 test pulseway: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: The service is unavailable.
Apr  9 04:40:54 test pulseway: Error while registering online:Error 503 fault: SOAP-ENV:Server[no subcode]#012"HTTP/1.1 503 Service Unavailable"#012Detail: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">#015#012<HTML><HEAD><TITLE>Service Unavailable</TITLE>#015#012<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>#015#012<BODY><h2>Service Unavailable</h2>#015#012<hr><p>HTTP Error 503. The service is unavailable.</p>#015#012</BODY></HTML>#015#012

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...