1.4.0 - Upload issues about nicotine-plus HOT 20 CLOSED

sorry, but it looks like there's a good chance my problem was caused by enabling the 'block unresolvable ips' setting. Will repost if that isn't the case. Apologies for the possibly pointless report!

from nicotine-plus.

Thought for a second it was a false alarm, but sometime within the last hour uploads have stopped working again.
More notes:

• I can browse my files and add them to transfers, but their status is "Queued".
• When I search for my files with the Search function, nothing shows up most of the time. However, periodically I do get results for a brief window of time (maybe a minute?). If I add these files to my Transfers, their status is "Queued" as well.

from nicotine-plus.

Hi @sunox1,

That's quite a strange behaviour indeed.
Are you using UPnP or are you doing a static port redirection ?

from nicotine-plus.

Hi @gfarmerfr,

Thanks for the response.

I am forwarding the listening port, and one port directly above it to a static IP (SoulseekQT requires you to open the listening port and an "obfuscated port", and I do the same with nicotine-plus just to be safe). My router runs OpenWRT, if that matters. I have a local firewall as well, but the appropriate ports are all open. And during the time when I'm having the issue, the built in open-port-checker tool reads that the ports are open.

I recall seeing a few "PierceFireWall" messages in the debug window, but I wasn't sure how to interpret them, or if they were relevant.

from nicotine-plus.

Argh, I just got home and it seems to be working for now. When I was having difficulty earlier today I was at school, so maybe there was something about my school's network that was causing problems? Although I was able to download from other users just fine. I'm confused. Anyway, I'll close it again for now. Thanks for your time.

from nicotine-plus.

No problem, keeps us updated about the situation.

from nicotine-plus.

Uploading problems
When the I can receive direct connections box is unchecked, I get about half the uploads as I do in QT, which I assume is expected behaviour. Occasionally, I become unreachable via search, and I can't form connections with anyone at all - this is a mystery.

When the I can receive direct connections box is checked, however, things change quite a bit. It oscillates back and forth between seeming to be working fine (haven't been able to compare it with QT yet, so can't confirm), and being completely unreachable (I can't find my files via search, and lots of Can't connect... messages in the log). About an equal amount of time is spent in each state.

Port status
This seems to be the big clue: When I am in the unreachable state, the nicotine port checker shows the ports as closed. Otherwise, it shows them as open. A third party port checker website corroborates what the nicotine checker says about the ports nicotine is listening on.

I think the ports are being forwarded correctly
I don't know exactly how to interpret all of this, but it doesn't seem like my router would be responsible. The QT port checker doesn't show any of this volatility, which matches its lack of volatility upload-wise.

Thanks again.

from nicotine-plus.

Hi @sunox1,

The connection issue at your school doesn't really surprise me because, as you said, you can't do anything about port redirection there. So if you have people in the same situation on the other side (no forwarding) you will see the giving up messages. For those who have redirection on the other side it should work.
I'm not entirely familiar how the peers connection scheme works but that's what I'm understanding from this doc.
Has QT the same connectivity issues at school by any chance ?

The I can receive direct connections checkbox basically mean what it says: it tells Nicotine and other clients that they can contact your PC directly either via a port forwarding (UPnP or static) or if you have a public IP on your PC (bridge mode).
OpenWRT should be tunable enough to activate the bridge mode I think. I know this is not a desirable state but have you tried it to see how Nicotine would react ? An alternative to test, if OpenWRT permits it, is to put your PC in a DMZ (should be a similar behaviour that the bridge mode).

from nicotine-plus.

Hi @gfarmerfr,

Thanks for the response, and I appreciate the help. I will give that a try tonight.

Sorry, I wasn't clear about the school thing. I was on my Windows laptop using QT and trying to download from my home pc running nicotine. I tried downloading from a bunch of different people, and my home pc was the only account that gave me problems. Today I did the same thing though and I can download from my home pc running nicotine just fine, so I'm no longer sure what to make of what happened the other day.

The can't connect... errors and all the other errors appear on the logs of my home machine when idling (and also when I check the I can receive direct connections box.

The behaviour of nicotine after I check the I can receive... box definitely means that port forwarding is the issue then. I suppose it is now a question of whether it is my network config or the client that is causing the issue. It doesn't seem like my network config is the issue, because both QT and nicotine report that the ports are being forwarded, QT appears to be working perfectly fine with in the same environment, and in the past I have successfully port forwarded with other applications.

I'm not certain that I am doing port forwarding corectly, but this is how I am doing it at home (I arbitrarily chose a 5 digit listening port - let's say it's 20000): myRouter : 20000 => myPC (static IP) : 20000. I suppose there could be something wrong with that, but it would be strange considering that a number of indicators all say that it is being done correctly. It also leaves it a mystery why QT seems to work ok with the same client settings in the same network - I upload approximately twice as many files on QT as on nicotine-plus on the same machine sharing the same files (I just listen on different ports).

I will try playing with the router settings tonight when I get home later, and hopefully I will find something.

from nicotine-plus.

Hi @gfarmerfr,

I was able to put my pc into a dmz, and nicotine behaves the same way as before. That is, if I check I can receive direct connections, I am unable to make any connections whatsoever; if I leave this box unchecked, I upload files at half the rate of QT at the best of times, and I occasionally I go through periods where I am unsearchable and not making any connections at all.

Please let me know if there is anything else I can do.

from nicotine-plus.

Hello,

Sorry for the message spam, but I've managed to shrink down my former novel on the issues (a few posts up) to just a few paragraphs and I thought it deserved its own post. I've learned that a lot of what I was discussing as potential problems are actually expected behaviour, and that the problem is a little different than I thought it was. I've pasted it below for visibility's sake.

Uploading problems
When the I can receive direct connections box is unchecked, I get about half the uploads as I do in QT, which I assume is expected behaviour. Occasionally, I become unreachable via search, and I can't form connections with anyone at all - this is a mystery.

When the I can receive direct connections box is checked, however, things change quite a bit. It oscillates back and forth between seeming to be working fine (I still get half the uploads, but I'm only reachable half the time), and being completely unreachable (I can't find my files via search, and lots of Can't connect... messages in the log). About an equal amount of time is spent in each state.

Port status
This seems to be the big clue: When I am in the unreachable state, the nicotine port checker shows the ports as closed. Otherwise, it shows them as open. A third party port checker website corroborates what the nicotine checker says about the ports nicotine is listening on.

I think the ports are being forwarded correctly
I don't know exactly how to interpret all of this, but it doesn't seem like my router would be responsible. The QT port checker doesn't show any of this volatility, which matches its lack of volatility upload-wise.

from nicotine-plus.

Hi @sunox1,

Great debugging :)
You're right : the fact that N+ behaviour doesn't change in a DMZ clearly indicate that it's not a port forwarding issue.

Just to be sure there's no config corruption can you provide the two following values from your ~/.nicotine/config file:

firewalled = XXX
portrange = (YYY, ZZZ)

As you previously indicate you have a local firewall: I suppose your firewall is configured to have the ports in the portrange open, yes ?

The fact that the port status oscillates is quite puzzling.
A few wild guess that might be worth exploring when you observe the "port closed" status:

1. The router for some reason has stop to forward traffic to N+ port (very unlikely): no simple way to test that, maybe combining nc and tcpdump ?
2. N+ local socket is dead: can be checked via this command line (replace 2234 by the port you've configured N+ to listen on, most likely it will be the first port of your portrange):

netstat -laputen | grep 2234   
tcp        0      0 0.0.0.0:2234            0.0.0.0:*               LISTEN      1000       11648341   11838/python
...

3. The number of socket connections has exceed the limit (maybe a leak in the connection pool ?)
   You can check your limit via this command (replace 11838 with n+ pid):

cat /proc/11838/limits | grep -i "Max open files"
Max open files            4096                4096                 files    

and the current number of open socket by N+ is in the bottom right status bar.
N+ won't go further that 90% of the "Max open files".

4. Sometime I've seen SYN flood reported by the kernel via dmesg

[660395.150814] TCP: request_sock_TCP: Possible SYN flooding on port 2234. Sending cookies.  Check SNMP counters.

I'm not sure of the consequences of this and how your firewall is configured to react to that particular nasty behaviour.

If everything is correct for these parts we can definitively be sure that N+ has some bug lurking around that I haven't identified yet.

from nicotine-plus.

Thank you for the advice. I haven't got a definitive answer for 1. yet, but I was able to get answers for everything else.

In ~/.nicotine/config:
firewalled = False
portrange = 55525

I do normally have a firewall (ufw) enabled (with all of those ports open), but I disabled it last night just so I don't have to worry about it as a possible cause of any issues.

1. I looked at the output of tcpdump | grep 55525 during the transition from open to close and I didn't see anything out of the ordinary - looking at tcpdump alone, you couldn't tell anything is wrong. I can keep trying to look.
   I am reading about nc now (I've never used this before).
   If the router stop forwarding ports and then starts again, would this be logged in my router's logfile? I checked and there isn't anything unusual there.

2. Everything looks good in netstat: my first line looks identical to your sample output.

3. My Max open files is 2048, which is higher than N+'s self-imposed limit of 1843.

4. It looks like I get the same message a few times per day, except instead of Sending cookies mine says Dropping request. These messages happen far less frequently than port oscillations, though, so maybe that means they aren't related? Also, not sure if this is relevant or not, but last night I actually turned off SYN flood protection in OpenWRT to see if it would help anything (nothing changed).

Further steps...

I just tried going back to 1.2.16 (version in Jessie repos) to double check that it behaves the same, and it does.

Could be important: The number of port-closing instances is proportional to the number of files I am sharing (the number of connections I am making)

I tried sharing only a small folder of about 10 files to reduce the number of connections I was making, and the ports still close once in a while, but far, far less frequently. I then added about 1000 files to my shared folder and the port closes started happening more frequently, but still far less than when I shared all my files.

So maybe it isn't the number of connections that is causing the issue, but rather a certain kind of 'bad' connection? If it was the number of connections, then it seems like the problem should go away completely if I'm only sharing a few files. If it's a certain type of connection that causes the problem, and you reduce the number of connections, then you would reduce the number of port closes (less connections -> less 'bad' connections) - but not eliminate them entirely.
...

I would be willing to upgrade to Debian Testing if you thought there was a reasonable chance it might help.

from nicotine-plus.

Re @sunox1,

1. I'm not sure what the router will log.
2. OK
3. OK
4. I've research further and these kind of messages should only be triggered when lots of connections are established in a short amount of time, like when you do a search. So it should not be related to your problem from what I understand.

Researching 4) I've put together a experimental patch in one of my repo branch https://github.com/gfarmerfr/nicotine-plus/tree/slskproto-put-nowait
I doubt it will change anything to your problem but it wouldn't hurt to try I guess...

from nicotine-plus.

I think I switched branches successfully. I cloned the main repo and then did git checkout -b /origin/slskproto-put-nowait, which seemed to work, and then I launched nicotine from the directory root. The version number was still 1.4.0, though.

Anyway, no change unfortunately.

Another quick note: I mentioned above that the ports close less often if I share fewer files. I found later that if I uncheck Enable sending search results to other users, the port never closes. If I enable search and set it to a maximum of one, it seems to close just as often as it does at 50. I'm not sure if this is interesting or useful at all, but just thought I would mention.

from nicotine-plus.

Whoa! So it turns out I wasn't installing the branch correctly. I finally figured out how to do so

AND

I've been seeding all my files for about 15 minutes now without any port closures. Whether it is totally fixed, I don't know, but that is at the very least a massive improvement. In all my testing in the last few days it never lasted more than a minute or so before the port closed, and it usually took far less than that.

I'm just heading out for the day, but I will let it run alongside QT and we will see how it goes. I hesitate to say it is fixed, but things look very promising right now ! I will provide an update by tonight (EST) at the latest.

Update
From school I tried to download the same file from both clients. My N+ account queued me immediately, but my QT account downloaded fine. It looks like I am queueing a lot of people, as one of my queued torrents is 12th in line at the moment. I won't be able to investigate the cause of this until later tonight, unfortunately.

from nicotine-plus.

So the issue is indeed solved (woo!), and it looks like the reason is that I upgraded to Debian Stretch. I could have swore that I tested n+ after upgrading and still encountered the port closing issue, but maybe not :/

I think that takes care of all the issues for now, so I'll to close this. Thanks so much for your time and effort - I really appreciate it. I learned a fair amount during this process, and I hope it wasn't a total waste of your time.

Cheers!

from nicotine-plus.

No problem.
So the problem has disappear by upgrading debian alone or did you still need to run the experimental patch on top of the upgrade ?
I'm a little lost and maybe tired too :)

from nicotine-plus.

Sorry, that wasn't clear.

It looks like just upgrading from Debian Jessie to Stretch (stable to testing) fixed it. Late yesterday I downgraded to the N+ version in the Debian testing repos (quite old) and everything ran fine - no port closures. I originally thought the patch had fixed because I thought (mistakenly) I tested the master version in Stretch, before trying the experimental patch, and observed the same old bug behaviour. I guess I started to imagine things after staring at logs for 2 days! :)

Luckily, Stretch will become the Stable release in a few months, so this should be a non-issue going forward.

from nicotine-plus.

Thx for the head's up :)

from nicotine-plus.