ERROR (abnormal termination) - mnemosyne & hpfeeds-broker about mhn HOT 17 CLOSED

I close the topic.

It works well on older Ubuntu S version.

No support on 14.04

from mhn.

Sorry for the late response. This should work on Ubuntu 14.04. We have tested on there. I will try it again today, maybe some recent changes broke it.

from mhn.

I was able to setup MHN on Ubuntu 14.04 today. So, I am not sure what that error was about.

from mhn.

Hello !
Thanks for your help !

Yes, seems a bit strange...

I've used the last Ubuntu Server 64bit available on their website and had setted up nothing more on it (in way to avoid uncompatible process)

I've also retry but on the Ubuntu Server 12.04 exactly the same way I did on 14.04 but without any problems and it's working well.

Maybe some files doesn't download correctly or something like this because it only concerns 2 process.

I'll retry a bit later on US 14.04 and give some news

from mhn.

Ok, I've found the problem.

It's when you need to reboot the machine.
The server can't close correctly it's mnemosyne & hpfeeds-broker threads
and than, crash it again and again after the reboot.

The thread can't be killed because it doesn't start
AND it's impossible to start/restart it because it says "ERROR - Abnormal Termination".

Nothing in syslog, Supervisorctl started well without any problems.

When i'm trying to manually restart from supervisorctl pane,
All process are in "STARTING" mode (which is ok)
but "mnemosyne" & "hpfeeds-broker" are in "BACKOFF" mode.

PROCESS ERROR LOG :
** HPFEEDS-BROKER

pymongo.errors.ConnectionFailure : Errno 111
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr

**MNEMOSYNE

pymongo.errors.ConnectionFailure : Errno 111
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr

MongoDB is running well (as the exception pymongo could say the opposite)

from mhn.

Can you try this, after it is booted up:

sudo supervisorctl stop all
sudo service supervisord restart
sleep 10 # just to let the processes start and settle
sudo supervisorctl status

I haven't seen this error before. Could it be that these processes are
starting before mongo starts at bootup?

--Jason
​

from mhn.

Hi !

I've just tested but still same result:
"Abnormal Termination" on mnemosyne & hpfeeds-broker when all other process had restart well.

from mhn.

Could you provide the full logs from:

/var/log/mnemosyne.*
/var/log/hpfeeds-broker.*

Also, could you provide the output of this?

sudo iptables -L
sudo netstat -luntp

from mhn.

Here are the logs :

• MNEMOSYNE.LOG (empty)

• MNEMOSYNE.ERR

  sys.excepthook is missing
  lost sys.stderr
  2014-10-06 10:56:45,089 (root) Starting mnemosyne. (Git: 384386e9566bb5b3a591b248c51185a32b53b622)
  2014-10-06 10:56:45,090 (persistance.mnemodb) Connecting to mongodb, using "mnemosyne" as database.
  Traceback (most recent call last):
  File "runner.py", line 116, in
  db = mnemodb.MnemoDB(c['mongo_db'])
  File "/opt/mnemosyne/persistance/mnemodb.py", line 36, in init
  conn = MongoClient(auto_start_request=False)
  File "/opt/mnemosyne/env/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 369, in init
  raise ConnectionFailure(str(e))
  pymongo.errors.ConnectionFailure: [Errno 111] Connection refused
  Unhandled exception in thread started by
  sys.excepthook is missing
  lost sys.stderr
  2014-10-06 10:56:47,538 (root) Starting mnemosyne. (Git: 384386e9566bb5b3a591b248c51185a32b53b622)
  2014-10-06 10:56:47,543 (persistance.mnemodb) Connecting to mongodb, using "mnemosyne" as database.
  Traceback (most recent call last):
  File "runner.py", line 116, in
  db = mnemodb.MnemoDB(c['mongo_db'])
  File "/opt/mnemosyne/persistance/mnemodb.py", line 36, in init
  conn = MongoClient(auto_start_request=False)
  File "/opt/mnemosyne/env/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 369, in init
  raise ConnectionFailure(str(e))
  pymongo.errors.ConnectionFailure: [Errno 111] Connection refused
  Unhandled exception in thread started by
  sys.excepthook is missing
  lost sys.stderr
  2014-10-06 10:56:50,979 (root) Starting mnemosyne. (Git: 384386e9566bb5b3a591b248c51185a32b53b622)
  2014-10-06 10:56:50,980 (persistance.mnemodb) Connecting to mongodb, using "mnemosyne" as database.
  Traceback (most recent call last):
  File "runner.py", line 116, in
  db = mnemodb.MnemoDB(c['mongo_db'])
  File "/opt/mnemosyne/persistance/mnemodb.py", line 36, in init
  conn = MongoClient(auto_start_request=False)
  File "/opt/mnemosyne/env/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 369, in init
  raise ConnectionFailure(str(e))
  pymongo.errors.ConnectionFailure: [Errno 111] Connection refused
  Unhandled exception in thread started by
  sys.excepthook is missing
  lost sys.stderr

• HPFEEDS-BROKER.LOG (empty)

• HPFEDDS-BROKER.ERR

  CRITICAL:root:Database connection closed (SO_ERROR: ECONNREFUSED). Exiting.
  CRITICAL:root:Database connection closed (SO_ERROR: ECONNREFUSED). Exiting.
  CRITICAL:root:Database connection closed (SO_ERROR: ECONNREFUSED). Exiting.
  CRITICAL:root:Database connection closed (SO_ERROR: ECONNREFUSED). Exiting.

• IPTABLES -L
  I'm not using it so IFO on ACCEPT

  Chain INPUT (policy ACCEPT)
  target prot opt source destination
  Chain FORWARD (policy ACCEPT)
  target prot opt source destination
  Chain OUTPUT (policy ACCEPT)
  target prot opt source destination

• NETSTAT -LUNTP

  Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
  tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 966/redis-server
  tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 902/nginx
  tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 682/sshd
  tcp6 0 0 :::22 :::* LISTEN 682/sshd
  tcp6 0 0 :::3000 :::* LISTEN 1366/server

Would it be more a problem about the MongoDB than MHN-Server ?
I've just noticed that, even if the systems says "MongoDB has started" and gave the process number, I can't find it..

from mhn.

Yeah, it appears that mongodb is not running. Can you look in its logs and see whats going on?

from mhn.

Ok, found the problem !

It was Mongodb which was configured to journalize in a bad place (so there was insufficient free space on this folder).

I've changed it into the .conf file and now MHN is working again.

Thanks a lot !

from mhn.

JT....seeing the same instance here with mnemosyne and hpfeeds-broker.
went through the things that i can think of and need a second set of eyes.
12.04 LTS...what arnaud said on 10/4 (this thread) is right on the money with what i'm see....thoughts?

from mhn.

Can you check your mongodb log. This error indicates that either Mongo is not running or Mongo is in a weird state. It is possible that supervisord is starting mnemosyne and hpfeeds-broker before mongo starts, but I have been unable to reproduce this error by rebooting.

from mhn.

JT, thanks for the pointers. the SO article that you pointed out worked great. http://stackoverflow.com/questions/13700261/mongodb-wont-start-after-server-crash

Ran the following to correct the hpfeeds and mnemosyne
sudo -u mongodb mongod --repair --dbpath /var/lib/mongodb/
sudo reboot

logged back in and checked supervisorctl
sudo supervisorctl status
all running as expected

from mhn.

Nice. Glad to help!

from mhn.

Ok, found the problem !

It was Mongodb which was configured to journalize in a bad place (so there was insufficient free space on this folder).

I've changed it into the .conf file and now MHN is working again

somebody please help me how to fix like that?
sorry i'm newbies

from mhn.

Hi, I installed MHN as per mentioned direction on Gihub/threastram mhn.........
but the out is only honeymap.!
no hpfeeds broker, no mnomosyne,
please can any one help me

from mhn.