maded2 / plotng Goto Github PK

Plotng should pause or skip devices that are running low on capacity. For example, a destination drive can continue to have plots prepared for it despite not enough free space existing on the drive to store the plots.

Perhaps plotng can calculate how many plots can be made for a destination drive when starting the plotting process, and then automatically stop that queue when the drive is full?

It would be supercool:

1. Using powerShell im setting delay for every plot, to free cpu threads from other plotting
2. If i could set a loop of plotting script = super! Infinite plotting
3. Choose at the bottom tabs with logs is more than a nice feature

ps. im a windows user

Hey, great tool! Thanks for sharing it.

Would be nice if -r (number of threads, currently defaulted to r=2) and -b (memory per plot, currently hardcoded to 6000) could be made configurable in the .json file.

Thanks :)

Hi - first, thank you for this great tool!

Is there a way to move between sections in the UI so we can scroll down beyond what is displayed? For example, I have 39 "Archived Plots" but I can only see the most recent 7.

Hello

I wondering whether we could have a different config for each drive because if you have a few drives with different speed maybe you want to have one fast drive with 3 parallel plots and a slow one with just 1 plot. I understand that plotng don't care about the drive's speed and you can't specify any config for that.

Have a good farming

First, thanks for this tool. Your client/server model was a great idea.

Issue: Time seems in seconds instead of minutes for Windows.
Expected behavior: "DelaysBetweenPlot": 90 # in 90 minutes it should kick off the next plot
Actual Behavior: Next plot starts in 90 seconds.

So it could either be a documentation change to let windows users know, or see what the fix might be for windows interpreting the time in seconds vs minutes.

This chia-blockchain Wiki page shows a table estimate of progress percentages, could this information be used to display percent/progress to the UI?

https://github.com/Chia-Network/chia-blockchain/wiki/Beginners-Guide#create-a-plot

Thanks for this great utility!

--Tony

I'm plotting on hdds, which can each manage one simultaneous plot.
Currently, I'm running one plotng instance per hdd, because if I add each hdd as both temp folder and target folder, a plot might be split over two hdd, causing slowdown.
Unfortunately this doesn't allow plotng's awesome staggering/delaying to be applied?

Would it be possible to allow plotting to a single drive as both tmp and target, but having multiple of those?

I try it on ubuntu linux, and can't get it to work.
When start in gui mode, there is Log message: Get "http://localhost:8484/": dial tcp 127.0.0.1:8484 connect: connection refused.

With no gui there is only a message "Plotting exit with error: exit status 1"

For those of us using https://dashboard.chia.foxypool.io/ it would be great for plotng-managed plots to be visible in the dashboard, especially if we're running multiple machines.

As per felixbrucker/chia-dashboard-satellite#11 on the dashboard dev's repo, plot managers could expose a similar plotting API to the main Chia GUI to allow his software to pick up on it.

Would be cool to be able to store the plot data in a file so the user can analyze that themselves or run their own scripts against that data.
The pane in the -ui option is a nice feature but sometimes you just need the actual data in a file.

Hi, thanks for your great work.
Since I use multiple plotting machines, can I start 1 server on every machine and connect the same UI on my Desktop to multiple servers?
This would be amazing.

This would save a bit of vertical space, and the level of precision doesn't seem needed since it's not a realtime clock. (Also, maybe always format seconds as two digits?) Thank you!

Currently I'm running into an issue where my ethernet traffic is bottlenecked because all the plots are done but is uploading to a hard drive on the network all at once, causing the plots to not be available until all of them are copied over. It would be very useful to be able to limit stage 4 upload to one at a time. Would this be possible?

Hello,
Im plotting in windows and it looks like regualr plotting, wall of text, it looks Nothing like the screenshot posted by the dev.

Am i doing something wrong?
I run it from CMD, should it be run from PowerShell?

Thanks

It would be nice if you supported -f & -p as an alternative to -a

This allows you to create plots without having the private keys on the plotter.

This is probably not the right place to implement such feature but I am going to make the suggestion anyway...

when you move/copy a large file to mechanical drive, the write speed is going to be the bottleneck. however if copy the file to an SSD first, the disk space on the plotting drives can be freed sooner so that you can start the next plot sooner while maximizing the free space on the plotting drives. while this might sounds similar to -2 argument, it isn't exactly the same as it only deals with the finalized plot instead of involving with phase 3 and 4.

I think it will minimize time to start the next plot given the max active plots constraint.

Im having this issue on Windows:

https://prnt.sc/13g2ev6

Ty!

In the process of making columns sortable, and adding a couple of other features I wanted, I've basically re-architected the entire UI.

Personally I think it's a cleaner design, but obviously I'm biased. Would you be open to a set of PRs changing how a lot of the UI is managed? Each panel can be updated in a self contained PR to keep them easier to review.

There's also a few changes around race conditions and security (specifically not listening on 0.0.0.0 by default).

add rsyng feature to -d diskt to remote machine
https://github.com/ericaltendorf/plotman/wiki/Archiving

05-03 08:37:46 Starting archive: rsync --bwlimit=80000 --remove-source-files -P /mnt/dst1/plot-k32-2021-05-03-01-50-b4271f88a74b36b516c242151e00fdda20e3f31ce1f8624465bf05a195009ecd.plot rsync://[email protected]:12000/chia/plots2

Copy the part after 05-03 08:37:46 Starting archive: . In our example that would be:
rsync --bwlimit=80000 --remove-source-files -P /mnt/plots1/plot-k32-2021-05-03-01-50-b4271f88a74b36b516c242151

This is my "Helper Plotter" its my main PC and it only plots when I use it to help the main one.

Here are my settings, it set to start all 4 plots with 5 minute delay, but as you can see from the screenshot, the last one was started with more then hour delay.

The I drive where the 3 plots run has 878GB of space, its more then enough for 3 plots that at max use 270gb of temp space
[its two 1TB drives in RAID 0 with big overprovisioning so i reduced the size as much as posible]

"Threads": 2,
"Buffers": 0,
"NumberOfParallelPlots": 4,
"TempDirectory": ["D:/ChiaTemp", "I:/1", "I:/2", "I:/3"],
"TargetDirectory": ["F:/Chia"],
"StaggeringDelay": 0,
"ShowPlotLog": false,
"DiskSpaceCheck": true,
"DelaysBetweenPlot": 5,
"MaxActivePlotPerTarget": 0,
"DisableBitField": false,
"MaxActivePlotPerTemp": 1,
"MaxActivePlotPerPhase1": 0,
"UseTargetForTmp2": true,
"BucketSize": 0

Any ideas?

Thanks

plotng is working great for me. I'm using it to do 3 plots in parallel with a 30m stagger. Over time, the plotting looks like this:

#####    #####    #####    #####
  #####    #####    #####    #####
    #####    #####    #####    #####

As that diagram shows, there's wasted time/resources at the beginning and end of each cycle where it's only doing 1 or 2 plots in parallel. Ideally, I'd like plotng to do this…

##### ##### ##### ##### ##### #####
  ##### ##### ##### ##### ##### #####
    ##### ##### ##### ##### ##### #####

…but I haven't figured out the combination of settings that would keep 3 plots going at all time. Is anyone doing this, and if so how did you configure it?

Hi. I would prefer to not have plotng getting Internet traffic and would much prefer to bind it to localhost only, can you please add support for that? Thanks :)

I use different NVME's (2 x 2TB and 1 x 1TB).
With PlotNG is it not possible to set a "MaxActivePlotPerTemp" for each Temp directory. The result is, that the 1TB NVME run out of space because PlotNG don't recognize that there is less space than an the other two NVME's.

Is it possible to fix that?

Thanks and Regards
Simon

I'd like to be able to stagger the plots evenly or by phase.

E.g., have only one plot running in phase 1 (which uses multiple threads).
Or, if I have 5 parallel plots, have them start at whenever the previous arrived at 100/5=20%.

Right now my plots are IO bound, so without manually increasing the number of parallel plots, they pretty much synchronize towards the end. At some point, I'm waiting for 5 plots to get copied at the same time over a slow network.

Thanks for this wonderful time saver!

I'd like to plot2 configured to the target folder.
In my setup, that would save one copy process.

edit what would also be nice if chia could plot into some tmp folder, then use a combined tmp2 + output folder. then, plotng could move the file in one of the configured target folders.

I can't make it to save logs. I even added the SavePlotLogs option as i saw in some other issue thread.

This is on Windows 10 (gaming rig, wont be installing Linux).

I run PlotNG UI on a portrait ultrawide monitor, and it'd be nice to be able to see the full stats in this scenario - right now, post multi-host-ui (which is wonderful BTW, thank you), my screen cuts off around the middle of "end time" in archived plots, preventing me from seeing durations of archived plots.

If the Plot ID, which is mostly irrelevant anyway, was visually truncated, say to 10 characters (which is more than enough to uniquely ID a plot using file search or whatever, in normal usage), I could almost definitely see everything at once on even this narrow screen.

I've been experimenting with different DelaysBetweenPlot values, and suddenly, even when set to 0 or a low value, plotng wouldn't start a new plot.

I looked at the code, I suspect that when DelaysBetweenPlot was set to a high value, createNewPlot() would set server.targetDelayStartTime to that time in the future, in one of these lines:

If now a lower value is set, createNewPlot() will still always abort here:

... never taking into account a changed DelaysBetweenPlot. maybe server.targetDelayStartTime should be reset on config or value reload?

Hello,

I get this message when starting the plotter on mac os big sur. tested on two installations. Unfortunately the message doesn't provide any infos besides that:

Plotting Exit with Error: exit status 255

Tried with different config.json variations, but no luck to getting it work. Plotting with original chia plotter commands works although.

Thank you!

Phase 1 plots limit wasn't working, but it is working now. The problem is plot timers (delay+stagger) are not working properly. See other comments below. (Updated)

Hello,
Can this feature be considered please?
Example 7 Parallel plots on 7 drives staggered 3 times, with Maximum Phase1 plots set to 7

When one of the first 7 plots finished a new staggered one should start on the drive that Finished Phase 1.
Right now it will start on the next drive on the list, which is the first one since list restarted.

Regards

Hello,
I would like to suggest a feature.

For the plotng to copy created plots in orderly manner one by one to designated HDD or HDD's.

Right now many people use an SSD as "buffer" to speed up plotting [TargetDirectory], when plot files created they saved to this SSD [saves up to 20 minutes per p lot]
Also when bunch of parallel plots finish their jobs together and start copying to HDD [most people fill one HDD at time] it creates a bottleneck and slows the speed even more, but if you have an SSD, it doesnt really that big of a diffrence if its one file or 3 files copied to it, its still plenty fast.

Up till now I was manually copying the plot file from the "buffer" SSD to the HDD.

Also if its posible to RENAME the plot files while it copies, becuase the Chia Farming has a bug when you copy file it sees it as defective untill you restart the farming gui.
To solve this, i just rename .plot files to .pl before I copy, and when they copied i rename back to .plot.

It would be great to have more possibilities regarding the actual plotting.

One thing would be the possibility to disable bitfield if preferred.

Hi. I'm using chia under plotng on a desktop machine, and would welcome an option in config file to give a "nice" priority to chia processes, so that they won't be as heavy penalizers and impact interactive GUI sessions as much. Can you please add this option for a "nice" value (i think they go from 19 to -20), 19 the lowest, and -20 the highest. I'd like to set all chia plotting processes to something like 10 so that the normal 0 priority processes don't get bogged by Chia.

Thanks

I must say that I really enjoy this tool and it has made my first experience using the chia CLI tool so much smoother.

Would it be possible to store a lot of the data your displaying as a running CSV? I want to use this tool to help determine optimal staggering behavior. Having all the nice log data you have displayed in these windows would make it so much easier to do a post plot analysis.

Things like: plot id, start time, end time, plot dir, destination dir, total time for each phase, number of threads used

and whatever else may come to mind.

To prevent overwriting the previous server logs, it may be helpful to use the approximate startup time of the server. Maybe even have a command line option to enable or disable this kind of logging for people using small or fragile boot drives.

Hi,
Please add a setting to the config file for the Chia queue system.
Reason: Me for example i start 7=8 parallel plots but each has a queue enabled
This way when plotting ends it starts a new one without killing chia.exe
And I can enable CPU affinity for each Chia.exe, i get each plot its 2 dedicated cores and it stays like this untill plotting ends.

Also it will allow to use the software at measured intervals, say you want to plot just 10 plots before you do soemthing with the PC, you can set a queue and it will end when all plots are done.

Thanks you

Currently, it seems that plotng round-robins the specified temp directories for each new plot process. Since the temp directories might be on drives with different amounts of free space, this can lead to incorrect numbers of parallel plots per drive. One possible solution is to have a parallel array in the config:

...
"NumberOfParallelPlots": 10,
"TempDirectory": ["/mnt/2tb", "/mnt/1tb"],
"TempDirectoryLimits": [7, 3],
...

Or, re-structure the config similar to how plotman allows specifying tmp_overrides options. I can help make these changes but want to make sure you like the design/config structure first.

Edit: MaxActivePlotPerTemp seems like a good start, but this still requires all the temp dirs to have the same free space.

Option to write a log per plot into a specified directory (like the Chia Windows GUI does).
The last x lines per plot on the GUI are good - i would love to have the option for whole log files for each plot tho!

Might be a problem on my side, but chia generates absolutely no logs for me, even tho i have set logging to "info"
Thus it would be much appreciated if your tool would just make a "> {plot_start_time}_log.txt" for each plot :)

#if Chia usually already does this, then ofc. it won't be neccesary and i just have a problem on all my servers

Besides that - great tool, thanks for your awesome work!

We have the ability to lower the number of plots by editing the config.

It would be nice to be able to hit a button, say d in the UI to top the specific plot that's being plotted.
It would be important to have a confirmation window appear before actually canceling.

Is it possible to fix the UI display in Windows, some of the letter has been over lapped and the boarder frame doesn't show up correctly

hi! First of all I think this tool is super cool and has made my plotting life a lot better than just using the stupid broken Chia GUI.

right now we have DiskSpaceCheck. Which checks space in the destination directory for approx 105GB free...

1. as a possible bug report, when I run plotng on a windows system and my destination drive is a mapped SMB share, I always get an error that the drive (0b free)is full despite having space.

2. I think it would be good if we could specify our own disk space requirement. Maybe we're plotting bigger work units, or maybe we just need to reserve space for other things.

3. I also think we should have an option to check the temp drive for space. Because:
   a) the temp drive can also run out of space which is bad
   b) if the temp drive and destination are the same, then we'd need more space
   c) to accomodate for other things going on in that drive. like other chia plotters or regular productivity usage.

Perhaps I am missing something, but I cannot find any configuration for a separate "temp2" directory (configured with the -2 switch in the chia command line. Did I just miss it somewhere ? - thanks ! 😄

@squizzling

Seeing reports from users, saying the active plot table now show '-1/4' phase and -1% progress. Can you double check.

Thanks

version 0.6, regardless of the value, it will always start the next plot in exactly 1 min (parallels plotting 2).

(venv) [root@chia ~]# go install plotng/cmd/plotng
go install: version is required when current directory is not in a module
        Try 'go install plotng/cmd/plotng@latest' to install the latest version

I'm not sure what I've done wrong, but this issue appears when following your instructions.

I had some plots end with an Errored status (not sure why yet) and the temp files caused some ongoing plotting issues. It would be cool if plotng knew to delete the tmp files from the temp drive in that case.

I'm plotting on Win10 with below settings, which are function normally with pure CLI.
As you can see I set the NumberOfParallelPlots to 12.
Before it running up to 8 parallel jobs it was great, but no more parallel job is created since then.
I even waited until the the 1st job is completed, and the next 1st job is started, still there are only 8 parallel jobs.
Is this some windows only issue?

{
"Fingerprint": "",
"FarmerPublicKey": "",
"PoolPublicKey": "",
"Threads": 4,
"Buffers": 4000,
"NumberOfParallelPlots": 12,
"TempDirectory": ["E:\p", "F:\p"],
"TargetDirectory": ["D:\p"],
"StaggeringDelay": 2,
"ShowPlotLog": true,
"DiskSpaceCheck": true,
"DelaysBetweenPlot": 20,
"MaxActivePlotPerTarget": 0,
"DisableBitField": false,
"MaxActivePlotPerTemp": 6
}

Currently it seems that DelaysBetweenPlot and StaggeringDelay are both global, that is to say, they apply across temp drives if more than one temp drive is set, not within a single temp drive.

What I'm trying to do, with 2 TempDirectory entries:
TempDirectory 1:
Max 4 parallel
2 max phase 1
delay 90 min between plots on this drive

TempDirectory 2 would be the same.

The result I'm looking for would be that every 90 minutes, PlotNG fires one plot on each drive, until MaxActivePlotPerPhase1 is met across both drives. After that, when 90 minutes has elapsed and a plot on either drive moves out of Phase 1, a new plot should fire on that drive.

What I'm seeing instead, with either StaggeringDelay or DelaysBetweenPlot, is 1 plot on TempDirectory 1, wait 90 min, 1 plot on TempDirectory2, wait 90 min, 1 plot on TempDirectory 1, wait 90 min... and so on.

Am I screwing up the config here, or is there not a way to do this currently?

I just started using plotng, and my config file sets NumberOfParallelPlots to 4 with a StaggeringDelay of 60.

A couple minutes after starting, I see two plotting operations happening. I'd expect one plot to start immediately, and the next to start in 60 minutes, etc. Am I misunderstanding how it should work?

This is on macOS in case it matters. Thank you!