Comments (26)
I think i know where you get the problem.
It is on Protocols\transfernmdcprotocol.cs:line 270.
Here the file is created and filled with zeros.
This is done to allow segmented downloading (where you could download last part
before the first part).
To verify that this is the cause, can you replace AllocateFile in
Utils\FileOperations.cs with the following method and report back?
public static void AllocateFile(string target, long size)
{
if (!PathExists(target))
{
//File.WriteAllBytes(target, new byte[ size]);
// Doing it as below is working on compact .Net and we can handle big
files (if you didnt have 2gb of free ram and wanted to write a 3 gb big file
this
made a problem before)
int SEGSIZE = 1024 * 1024;
using (FileStream fs = File.Create(target, SEGSIZE))
{
int i = 0;
int segmentSize = SEGSIZE;
do
{
// Set length to segment size
segmentSize = SEGSIZE;
// Is segment size bigger then content size?
if (segmentSize > size)
{
segmentSize = (int)size;
}
// Are we at last segment
else if ((1 + i) * segmentSize > size)
{
segmentSize = (int)(size % segmentSize);
}
fs.Write(new byte[segmentSize], 0, segmentSize);
System.Threading.Thread.Sleep(1);
i++;
} while ((i * SEGSIZE) < size);
}
}
}
Original comment by blomman84
on 14 Mar 2009 at 9:55
- Changed state: Accepted
from flowlib.
I have made changes, but this dont solves problem. I set brakepoint to
System.Threading.Thread.Sleep(1); line, it fires only once at download begin.
But I
have 85-100% processor usage during all download time.
Original comment by [email protected]
on 15 Mar 2009 at 2:20
from flowlib.
ok :)
Next try then.
In Connections\TcpConnection.cs
Change:
protected byte[] buffer = new byte[1024];
To:
protected byte[] buffer = new byte[8024];
In Protocols\transfernmdcprotocol.cs
Change:
if (!e.Handled)
To:
System.Threading.Thread.Sleep(1);
if (!e.Handled)
Original comment by blomman84
on 15 Mar 2009 at 3:24
from flowlib.
in second file if (!e.Handled) meets several times i have changed following:
in void OnTimer
in both void ParseRaw
total 3 delay insert
now downloading takes 60-70% as TaskManager said
Original comment by [email protected]
on 15 Mar 2009 at 3:49
from flowlib.
Some additional information here. 60-70% it shows on Amd Athlon X2 6000+, 2 GB
ram.
On Pentium 4 it still shows 80-100% processor load. Usual download speed is
between
500KB/s and 3 MB/s and very low latency. My client works in local network.
Original comment by [email protected]
on 16 Mar 2009 at 7:44
from flowlib.
Another part of information. While testing multisource download i found that
number
of transfers does not make infuence to processor load. It shows 60-70% with 1
source
and with 40 sources.
Original comment by [email protected]
on 18 Mar 2009 at 5:44
from flowlib.
What happens if you change the below?
In Connections\TcpConnection.cs
Change:
protected byte[] buffer = new byte[8024];
To:
protected byte[] buffer = new byte[1048576];
Original comment by blomman84
on 18 Mar 2009 at 7:03
from flowlib.
70-100% but now flowlib shows download speed 100-300 MBytes/s -it is unreal my
network works on 100 Mbit full duplex. if size is 1024 speed shows correctly
Original comment by [email protected]
on 19 Mar 2009 at 2:41
from flowlib.
maybe you can give me some start point to solve this issue? I'm not so good
understand all mechanism of download working. It is interesting that uploads
connections does not take significant processor time.
Original comment by [email protected]
on 26 Mar 2009 at 5:20
from flowlib.
I have realized single filestream for transfer (it was dynamic for each data
part),
but it is not a reason of problem.
Original comment by [email protected]
on 27 Mar 2009 at 10:57
from flowlib.
[deleted comment]
from flowlib.
After solving issue 29 it shows 20-50% processor usage.
I'm using protected byte[] buffer = new byte[102400];
Original comment by [email protected]
on 9 Apr 2009 at 5:03
from flowlib.
WOW! That is a huge difference :)
Is that line the only difference between the svn version and your local copy?
Original comment by blomman84
on 9 Apr 2009 at 5:13
from flowlib.
no, a have changed transfernmdcprotocol to support multisource download, all
changes
in issue 30
Original comment by [email protected]
on 9 Apr 2009 at 5:22
from flowlib.
now it is all except the buffer? :)
Original comment by blomman84
on 13 Apr 2009 at 9:18
from flowlib.
I'm not sure because I have changed many files in flowlib to make possible its
use
in client. I will try to download new copy and test it in my client.
Original comment by [email protected]
on 17 Apr 2009 at 8:53
from flowlib.
Full change list
DownloadManager.cs
1. Added struct KeyValuePairS<TKey, TValue> to support xml serialization
(save/load)
2. Implemented DownloadManager save/load methods
3. Implemented DownloadManager method FlowSortedList<Source>
GetSources(DownloadItem
d)
4. Added property: public SortedList<DownloadItem, FlowSortedList<Source>>
DownloadItems
DownloadItem.cs
1. Added properties:
public long TotalBytesReceived {get; set; }
public DownloadPriority Priority { get; set; }
Connections\Hub.cs
1. Added properties:
public long TotalShare { get; set; }
2. Modified property RegMode added in set section: if (regMode == -1)
TotalShare = 0;
3. Modified protected void OnProtocolUpdate:
Actions.UserOnline: TotalShare += long.Parse(ui.Share);
Actions.UserOffline: TotalShare -= long.Parse(userInfo.Share);
4. Added function: public User GetUserByStoreID(string storeID)
transfernmdcprotocol.cs
1. In ActOnInMessage: where message is MyNick
trans.Source.UserId = trans.User.StoreID; changed to
trans.Source.UserId = "o-go.ru411" + trans.User.StoreID;
this needs to allow passive download (it dont works else, i dont know how to do
this
better)
2. Added after fs.Unlock:
if (trans.DownloadItem.Priority != DownloadItem.DownloadPriority.Pause)
trans.DownloadItem.TotalBytesReceived += length;
FileOperations.cs
1. AllocateFile function modified to skip filling with zeros, its only sets
needed
size. I'm controlling file integrity using bitarray segmentsDownloaded.
(if I start download without this change I got exception because second stream
starts writing before first has finished filling file)
TcpConnection.cs - buffer size set to 102400
After these changes i have 30-60% CPU usage.
Original comment by [email protected]
on 20 Apr 2009 at 7:11
from flowlib.
I have realized persist filestream object for protocol, now it takes 8-15% CPU
usage,
i think this percent got by my gui animation draw functions and other
processes. It
takes 15% when flowlib throws SocketException. I think this is enough
perfomance.
Thanks for help =)
Original comment by [email protected]
on 24 Apr 2009 at 1:57
from flowlib.
after that I can't debug it=) client correctly downloads the file but don't
unlocking
it and I can't use it. This happens because protocol handler continues
recieving data
after segment has finished.
Here is another problem i found: I'm trying to download 1,4 Gb file from 40
sources,
my client timer every 10 seconds checking sources which not in
transferManager.Transfers and calls StartTransfer for each of them.
It works and flowlib establish connection with 18-20 sources (its all in
transferManager) other sources has no free slots or something else.
While download windows taskmanager shows constantly increase of system used
memory
and when it takes all available memory windows gets huge lags. After download
has
been finished windows releases memory. BUT my client used memory dont grows!
wtf?
Can you give me any advise or idea to solve this?
Original comment by [email protected]
on 24 Apr 2009 at 9:18
from flowlib.
I have returned to old not persist filetream version of
transfernmdcprotocol.cs. It
anyway eats memory when downloading. Most of cpu usage going to manage this
memory.
Original comment by [email protected]
on 27 Apr 2009 at 3:12
from flowlib.
I have found the problem - it because transfer continues reciving data after
DownloadItem.Cancel was called, and it calls DownloadItem.Finished() after
recieving.
Single filestream works fine, on Windows 7100 it dont eats memory.
Original comment by [email protected]
on 3 May 2009 at 1:25
from flowlib.
It appears the main reason of this issue is wrong behaviuor of disconnect.
HEEEELP :(
this is last thing separating me from relese
Original comment by [email protected]
on 14 May 2009 at 2:50
from flowlib.
Sadly my time is limited right now but if you can give me all source files
needed for
your client i can have a look when i get time over.
Original comment by blomman84
on 17 May 2009 at 1:30
from flowlib.
I dont get a high processor usage when downloading anymore.
Can you download latest svn and confirm this?
Original comment by blomman84
on 7 Sep 2009 at 9:29
- Changed state: Fixed
from flowlib.
Hi Mattias, nice to see you=)
I'm trying to use pure svn library in my client. Here the problems:
First of all I can not use IShare interface because it is not completly
realized.
For example hubnmdcprotocol.cs line 360. This is code that handles search
messages.
In my case there is no need to iterate through all items in share because
search can
be done much faster in SQL (I'm using SQL share)
Next BaseFileList.cs function CreateFilelist() - this function requires share
object
to implement ICollection interface. And then it trying to sort content. This
work
also can be done by SQL.
My suggestion is to create two virtual functions in Share class(or in IShare).
First
is Search and second is GetVirtualList. I have already done this work, you can
see it
in my example (when I tried to download all filelists om hub) Both functions
located
at document bottom.
Now about Issue. I have prepared special version of my client to work with
latest SVN
version. When I downloading from 10 sources (1Mbyte/s) I get 20-30% constant
CPU
usage (Athlon X2 6000!!!)
I suggest you to implement constant filestream for writing operation of
protocol. (In
my version it takes 5-10% when downloading from 13 sources (1Mbyte/s)) You can
find
my implementation in my last example.
Original comment by [email protected]
on 8 Sep 2009 at 1:58
from flowlib.
Hi :)
As you may have noticed, my time is limited :(
Can you open a new issue about IShare?
It is not related to this issue.
I will be back about this issue.
Original comment by blomman84
on 10 Sep 2009 at 7:56
- Changed state: Accepted
from flowlib.
Related Issues (20)
- TransferManager.EndTransfer() ITransfer.Disconnect() locks down application HOT 22
- Unable to download big file from many sources HOT 33
- Unable to download big file from many sources HOT 2
- Unable to download big file from many sources HOT 1
- Unable to download big file from many sources HOT 1
- Event UserOnline, Arguments property UserInfo.IsOperator is always false HOT 4
- Input string has wrong format HOT 3
- Wrong speed calculation HOT 5
- Exception when adding Virtual Dir to share in connected hub HOT 2
- Exception when Saving currently hashing share HOT 1
- Loaded DownloadManager.DownloadItem clears segmentsDownloaded bitArray at first call of GetAvailable() HOT 1
- Passive download don't work HOT 1
- XmlSerializer bug occurs when trying to save DownloadManager while windows preparing to shutdown HOT 1
- Memory leak when uploading HOT 8
- Integration of IShare interface HOT 2
- Protocol_ChangeDownloadItem event has wrong trans.Source.UserID HOT 1
- FlowLib is hard to use HOT 4
- Unable to stop UDPConnection HOT 5
- Self Signed SSL Certificates HOT 3
- Filelist download error HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flowlib.