High processor usage when active download about flowlib HOT 26 OPEN

I think i know where you get the problem.
It is on Protocols\transfernmdcprotocol.cs:line 270.

Here the file is created and filled with zeros.
This is done to allow segmented downloading (where you could download last part
before the first part).
To verify that this is the cause, can you replace AllocateFile in
Utils\FileOperations.cs with the following method and report back?

        public static void AllocateFile(string target, long size)
        {
            if (!PathExists(target))
            {
                //File.WriteAllBytes(target, new byte[ size]);
                // Doing it as below is working on compact .Net and we can handle big
files (if you didnt have 2gb of free ram and wanted to write a 3 gb big file 
this
made a problem before)
                int SEGSIZE = 1024 * 1024;
                using (FileStream fs = File.Create(target, SEGSIZE))
                {
                    int i = 0;
                    int segmentSize = SEGSIZE;
                    do
                    {
                        // Set length to segment size
                        segmentSize = SEGSIZE;
                        // Is segment size bigger then content size?
                        if (segmentSize > size)
                        {
                            segmentSize = (int)size;
                        }
                        // Are we at last segment
                        else if ((1 + i) * segmentSize > size)
                        {
                            segmentSize = (int)(size % segmentSize);
                        }
                        fs.Write(new byte[segmentSize], 0, segmentSize);
                        System.Threading.Thread.Sleep(1);
                        i++;
                    } while ((i * SEGSIZE) < size);
                }
            }
        }

I have made changes, but this dont solves problem. I set brakepoint to 
System.Threading.Thread.Sleep(1); line, it fires only once at download begin. 
But I 
have 85-100% processor usage during all download time.

ok :)
Next try then.

In Connections\TcpConnection.cs
Change:
        protected byte[] buffer = new byte[1024];
To:
        protected byte[] buffer = new byte[8024];


In Protocols\transfernmdcprotocol.cs
Change:
                    if (!e.Handled)
To:
                    System.Threading.Thread.Sleep(1);
                    if (!e.Handled)

in second file if (!e.Handled) meets several times i have changed following:
in void OnTimer
in both void ParseRaw
total 3 delay insert
now downloading takes 60-70% as TaskManager said

Some additional information here. 60-70% it shows on Amd Athlon X2 6000+, 2 GB 
ram.
On Pentium 4 it still shows 80-100% processor load. Usual download speed is 
between 
500KB/s and 3 MB/s and very low latency. My client works in local network.

Another part of information. While testing multisource download i found that 
number 
of transfers does not make infuence to processor load. It shows 60-70% with 1 
source 
and with 40 sources.

What happens if you change the below?
In Connections\TcpConnection.cs
Change:
        protected byte[] buffer = new byte[8024];
To:
        protected byte[] buffer = new byte[1048576];

70-100% but now flowlib shows download speed 100-300 MBytes/s -it is unreal my 
network works on 100 Mbit full duplex. if size is 1024 speed shows correctly

maybe you can give me some start point to solve this issue? I'm not so good 
understand all mechanism of download working. It is interesting that uploads 
connections does not take significant processor time.

I have realized single filestream for transfer (it was dynamic for each data 
part), 
but it is not a reason of problem.

[deleted comment]

After solving  issue 29  it shows 20-50% processor usage.
I'm using protected byte[] buffer = new byte[102400];

WOW! That is a huge difference :)
Is that line the only difference between the svn version and your local copy?

no, a have changed transfernmdcprotocol to support multisource download, all 
changes 
in issue 30

now it is all except the buffer? :)

I'm not sure because I have changed many files in flowlib to make possible its 
use 
in client. I will try to download new copy and test it in my client.

Full change list

DownloadManager.cs

1. Added struct KeyValuePairS<TKey, TValue> to support xml serialization 
(save/load)
2. Implemented DownloadManager save/load methods
3. Implemented DownloadManager method FlowSortedList<Source> 
GetSources(DownloadItem 
d)
4. Added property: public SortedList<DownloadItem, FlowSortedList<Source>> 
DownloadItems

DownloadItem.cs

1. Added properties:
public long TotalBytesReceived {get; set; }
public DownloadPriority Priority { get; set; }

Connections\Hub.cs

1. Added properties:
public long TotalShare { get; set; }

2. Modified property RegMode added in set section: if (regMode == -1) 
TotalShare = 0;
3. Modified protected void OnProtocolUpdate: 
Actions.UserOnline: TotalShare += long.Parse(ui.Share);
Actions.UserOffline: TotalShare -= long.Parse(userInfo.Share);
4. Added function: public User GetUserByStoreID(string storeID)

transfernmdcprotocol.cs

1. In ActOnInMessage: where message is MyNick 
trans.Source.UserId = trans.User.StoreID; changed to
trans.Source.UserId = "o-go.ru411" + trans.User.StoreID;
this needs to allow passive download (it dont works else, i dont know how to do 
this 
better)

2. Added after fs.Unlock:
if (trans.DownloadItem.Priority != DownloadItem.DownloadPriority.Pause)
   trans.DownloadItem.TotalBytesReceived += length;

FileOperations.cs

1. AllocateFile function modified to skip filling with zeros, its only sets 
needed 
size. I'm controlling file integrity using bitarray segmentsDownloaded.
(if I start download without this change I got exception because second stream 
starts writing before first has finished filling file)

TcpConnection.cs  - buffer size set to 102400

After these changes i have 30-60% CPU usage.

I have realized persist filestream object for protocol, now it takes 8-15% CPU 
usage, 
i think this percent got by my gui animation draw functions and other 
processes. It 
takes 15% when flowlib throws SocketException. I think this is enough 
perfomance. 
Thanks for help =)

after that I can't debug it=) client correctly downloads the file but don't 
unlocking 
it and I can't use it. This happens because protocol handler continues 
recieving data 
after segment has finished. 
Here is another problem i found: I'm trying to download 1,4 Gb file from 40 
sources, 
my client timer every 10 seconds checking sources which not in 
transferManager.Transfers and calls StartTransfer for each of them.
It works and flowlib establish connection with 18-20 sources (its all in 
transferManager) other sources has no free slots or something else.
While download windows taskmanager shows constantly increase of system used 
memory 
and when it takes all available memory windows gets huge lags. After download 
has 
been finished windows releases  memory. BUT my client used memory dont grows! 
wtf? 
Can you give me any advise or idea to solve this?

I have returned to old not persist filetream version of 
transfernmdcprotocol.cs. It 
anyway eats memory when downloading. Most of cpu usage going to manage this 
memory.

I have found the problem - it because transfer continues reciving data after 
DownloadItem.Cancel was called, and it calls DownloadItem.Finished() after 
recieving.  
Single filestream works fine, on Windows 7100 it dont eats memory.

It appears the main reason of this issue is wrong behaviuor of disconnect. 
HEEEELP :(  
this is last thing separating me from relese

Sadly my time is limited right now but if you can give me all source files 
needed for
your client i can have a look when i get time over.

I dont get a high processor usage when downloading anymore.
Can you download latest svn and confirm this?

Hi Mattias, nice to see you=)
I'm trying to use pure svn library in my client. Here the problems:
First of all I can not use IShare interface because it is not completly 
realized.
For example hubnmdcprotocol.cs line 360. This is code that handles search 
messages. 
In my case there is no need to iterate through all items in share because 
search can 
be done much faster in SQL (I'm using SQL share)
Next BaseFileList.cs function CreateFilelist() - this function requires share 
object 
to implement ICollection interface. And then it trying to sort content. This 
work 
also can be done by SQL.
My suggestion is to create two virtual functions in Share class(or in IShare). 
First 
is Search and second is GetVirtualList. I have already done this work, you can 
see it 
in my example (when I tried to download all filelists om hub) Both functions 
located 
at document bottom.
Now about Issue. I have prepared special version of my client to work with 
latest SVN 
version. When I downloading from 10 sources (1Mbyte/s) I get 20-30% constant 
CPU 
usage (Athlon X2 6000!!!)
I suggest you to implement constant filestream for writing operation of 
protocol. (In 
my version it takes 5-10% when downloading from 13 sources (1Mbyte/s)) You can 
find 
my implementation in my last example.

Hi :)
As you may have noticed, my time is limited :(

Can you open a new issue about IShare?
It is not related to this issue.

I will be back about this issue.