Fedora Server 30 Pulseway Client crashing repeatedly

[Can0n]

Hello All I have a Pro license for Pulseway currently watching over a PI3 for homebridge, two unRAID server and an unbuntu SFF PC for my PiHole DNS ad blocking, and just recently a Fedora Server 30  VM running on one of the unRAID servers that was working for a couple days when all of a sudden it started crashing a lot and frequently I am kind of stumped, if i manually restart the VM its stable for a while but starts crashing again, I am running kernel 5.3.6-200.fc30.x86_64 as that is the latest my reverse proxy script runs with.

the logs are showing core stack dumps  like this with different Process ID's of course:

Process 22212 (pulsewayd) of user 0 dumped core. Stack trace of thread 22333: #0 0x00007f3e0ac5d227 __strlen_avx2 (libc.so.6) #1 0x00007f3e0ab69a1e __vfprintf_internal (libc.so.6) #2 0x00007f3e0abf7754 __vsyslog_internal (libc.so.6) #3 0x00007f3e0abf7bba syslog (libc.so.6) #4 0x00000000005ab5f4 send_udp_message (pulsewayd) #5 0x00000000005ab529 send_to_remote_syslog (pulsewayd) #6 0x0000000000617eff _Z13syslog_threadPv (pulsewayd) #7 0x00007f3e0b7a74c0 start_thread (libpthread.so.0) #8 0x00007f3e0abfd163 __clone (libc.so.6) Stack trace of thread 22225: #0 0x00007f3e0abc97fd __nanosleep (libc.so.6) #1 0x00007f3e0abc96fe sleep (libc.so.6) #2 0x0000000000611696 _Z37user_login_logout_notification_threadPv (pulsewayd) #3 0x00007f3e0b7a74c0 start_thread (libpthread.so.0) #4 0x00007f3e0abfd163 __clone (libc.so.6) Stack trace of thread 22224: #0 0x00007f3e0abc97fd __nanosleep (libc.so.6) #1 0x00007f3e0abc96fe sleep (libc.so.6) #2 0x00000000005e569c _Z23check_for_alerts_threadPv (pulsewayd) #3 0x00007f3e0b7a74c0 start_thread (libpthread.so.0) #4 0x00007f3e0abfd163 __clone (libc.so.6) Stack trace of thread 22212: #0 0x00007f3e0abc97fd __nanosleep (libc.so.6) #1 0x00007f3e0abc96fe sleep (libc.so.6) #2 0x0000000000626730 main (pulsewayd) #3 0x00007f3e0ab24f43 __libc_start_main (libc.so.6) #4 0x000000000040f729 _start (pulsewayd) A systemctl yields this   [user@proxybox ~]$ sudo systemctl status pulseway ● pulseway.service - Remotely Monitor and Control IT Systems    Loaded: loaded (/etc/systemd/system/pulseway.service; enabled; vendor preset: disabled)    Active: active (running) since Sat 2020-02-15 21:44:42 MST; 18min ago  Main PID: 917 (pulsewayd)     Tasks: 5 (limit: 4915)    Memory: 12.1M    CGroup: /system.slice/pulseway.service            └─917 /usr/sbin/pulsewayd --no-daemonize Feb 15 21:44:42 proxybox systemd[1]: Started Remotely Monitor and Control IT Systems. Feb 15 21:44:42 proxybox pulseway[917]: Starting Pulseway Daemon version 6.5 (2019082205) Feb 15 21:44:42 proxybox pulseway[917]: Configuration read successfully! Feb 15 21:44:42 proxybox pulseway[917]: Could not get server info SOAP 1.1 fault SOAP-ENV:Server[no subcode]                                                     "Host not found"                                                     Detail: get host by name failed in tcp_connect() Feb 15 21:44:42 proxybox pulseway[917]: Could not get existing notifications: SOAP 1.1 fault SOAP-ENV:Server[no subcode]                                                     "Host not found"                                                     Detail: get host by name failed in tcp_connect() Feb 15 21:44:42 proxybox pulseway[917]: Last error getting group policy SOAP 1.1 fault SOAP-ENV:Server[no subcode]                                                     "Host not found"                                                     Detail: get host by name failed in tcp_connect() Feb 15 21:44:46 proxybox pulseway[917]: [566B blob data] Feb 15 21:44:46 proxybox pulseway[917]: Syslog: reporting completed successfully. Feb 15 21:44:46 proxybox pulseway[917]: [1.0K blob data] Feb 15 21:44:46 proxybox pulseway[917]: Syslog: reporting completed successfully.     Any help would be appreciated

 

[Can0n]

it just crashed again, using cockpit here is what i was able to grab

Path/etc/systemd/system/pulseway.service

---

---

Requires

• sysinit.target
• system.slice

Wanted By

• multi-user.target

Conflicts

• shutdown.target

Before

• shutdown.target
• multi-user.target

After

• systemd-journald.socket
• basic.target
• system.slice
• network.target
• sysinit.target
• syslog.target
  
  
  logs 
   

  Process 917 (pulsewayd) of user 0 dumped core. Stack trace of thread 1241: #0 0x00007f25f28c5227 __strlen_avx2 (libc.so.6) #1 0x00007f25f27d1a1e __vfprintf_internal (libc.so.6) #2 0x00007f25f285f754 __vsyslog_internal (libc.so.6) #3 0x00007f25f285fbba syslog (libc.so.6) #4 0x00000000005ab5f4 send_udp_message (pulsewayd) #5 0x00000000005ab529 send_to_remote_syslog (pulsewayd) #6 0x0000000000617eff _Z13syslog_threadPv (pulsewayd) #7 0x00007f25f340f4c0 start_thread (libpthread.so.0) #8 0x00007f25f2865163 __clone (libc.so.6) Stack trace of thread 944: #0 0x00007f25f28317fd __nanosleep (libc.so.6) #1 0x00007f25f28316fe sleep (libc.so.6) #2 0x00000000006011b1 _Z19collect_data_threadPv (pulsewayd) #3 0x00007f25f340f4c0 start_thread (libpthread.so.0) #4 0x00007f25f2865163 __clone (libc.so.6) Stack trace of thread 1045: #0 0x00007f25f28317fd __nanosleep (libc.so.6) #1 0x00007f25f28316fe sleep (libc.so.6) #2 0x0000000000611696 _Z37user_login_logout_notification_threadPv (pulsewayd) #3 0x00007f25f340f4c0 start_thread (libpthread.so.0) #4 0x00007f25f2865163 __clone (libc.so.6) Stack trace of thread 917: #0 0x00007f25f28317fd __nanosleep (libc.so.6) #1 0x00007f25f28316fe sleep (libc.so.6) #2 0x0000000000626730 main (pulsewayd) #3 0x00007f25f278cf43 __libc_start_main (libc.so.6) #4 0x000000000040f729 _start (pulsewayd)
  CODE_FILE	../src/coredump/coredump.c
  CODE_FUNC	submit_coredump
  CODE_LINE	834
  COREDUMP_CGROUP	/system.slice/pulseway.service
  COREDUMP_CMDLINE	/usr/sbin/pulsewayd --no-daemonize
  COREDUMP_COMM	pulsewayd
  COREDUMP_CWD	/
  COREDUMP_ENVIRON	LANG=en_CA.UTF-8 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin INVOCATION_ID=e62de2e097f6413fb63e849796fe1bc3 JOURNAL_STREAM=9:27058
  COREDUMP_EXE	/usr/sbin/pulsewayd
  COREDUMP_FILENAME	/var/lib/systemd/coredump/core.pulsewayd.0.e60b512c214445638487f96bfa56ba5e.917.1581831887000000.lz4
  COREDUMP_GID	0
  COREDUMP_HOSTNAME	proxybox.kungfury.ca
  COREDUMP_OPEN_FDS	0:/dev/null pos: 0 flags: 0100000 mnt_id: 24 1:socket:[27058] pos: 0 flags: 02002 mnt_id: 9 2:socket:[27058] pos: 0 flags: 02002 mnt_id: 9 4:/var/pulseway/pulseway.id pos: 0 flags: 0100000 mnt_id: 97 5:/var/log/wtmp pos: 511488 flags: 0100000 mnt_id: 97
  COREDUMP_PID	917
  COREDUMP_PROC_CGROUP	11:memory:/ 10:hugetlb:/ 9:devices:/ 8:blkio:/ 7:freezer:/ 6:pids:/ 5:net_cls,net_prio:/ 4:cpuset:/ 3:cpu,cpuacct:/ 2:perf_event:/ 1:name=systemd:/ 0::/system.slice/pulseway.service
  COREDUMP_PROC_LIMITS	Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 31703 31703 processes Max open files 1024 524288 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 31703 31703 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
  COREDUMP_PROC_MAPS	[no data]
  COREDUMP_PROC_MOUNTINFO	[no data]
  COREDUMP_PROC_STATUS	Name: pulsewayd Umask: 0022 State: D (disk sleep) Tgid: 917 Ngid: 0 Pid: 917 PPid: 1 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 128 Groups: NStgid: 917 NSpid: 917 NSpgid: 917 NSsid: 917 VmPeak: 452916 kB VmSize: 387380 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 12888 kB VmRSS: 12768 kB RssAnon: 3648 kB RssFile: 9120 kB RssShmem: 0 kB VmData: 46536 kB VmStk: 132 kB VmExe: 3404 kB VmLib: 6728 kB VmPTE: 120 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 1 THP_enabled: 1 Threads: 5 SigQ: 2/31703 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000001000 SigIgn: 0000000000001001 SigCgt: 00000001800042ae CapInh: 0000000000000000 CapPrm: 0000003fffffffff CapEff: 0000003fffffffff CapBnd: 0000003fffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp: 0 Speculation_Store_Bypass: thread vulnerable Cpus_allowed: 7 Cpus_allowed_list: 0-2 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 1954 nonvoluntary_ctxt_switches: 18
  COREDUMP_RLIMIT	18446744073709551615
  COREDUMP_ROOT	/
  COREDUMP_SIGNAL	11

[Can0n]

****updated

i completely uninstalled it removed it from my profile on my.pulseway.com and reinstalled it using "su"


I now have approx 10min of stability where it was crashing every few minutes before

[Chris]

Hi @Can0n,

This issue is caused by the fact that for some reason you are getting the following error on your system:

Feb 15 21:44:42 proxybox pulseway[917]: Could not get existing notifications: SOAP 1.1 fault SOAP-ENV:Server[no subcode]
                                                    "Host not found"
                                                    Detail: get host by name failed in tcp_connect()

 

Please setup the mtr to run trace towards your Pulseway server and then when Pulseway is reporting that connection can not be made, then please check what is going on with the packet trace.