Giter VIP home page Giter VIP logo

battlenet-lancache-prefill's Introduction

battlenet-lancache-prefill

view - Documentation ko-fi

GitHub all releases dockerhub

Automatically fills a Lancache with games from Battle.net, so that subsequent downloads for the same content will be served from the Lancache, improving speeds and reducing load on your internet connection.

Prefilling game

Features

  • Downloads specific games by product ID
  • High-performance! Downloads are significantly faster than using Battle.net, and can easily reach 10gbit/s or more!
  • Game install writes no data to disk, so there is no need to have enough free space available. This also means no unnecessary wear-and-tear to SSDs!
  • Multi-platform support (Windows, Linux, MacOS, Arm64)
  • No installation required! A completely self-contained, portable application.

Table of contents

Initial Setup

BattleNetPrefill is flexible and portable, and supports multiple platforms and configurations. It can be run on directly on the Lancache server itself, or on your gaming machine as an alternative Battlenet client. You should decide which one works better for your use case.

Detailed setup guides are available for the following platforms:

Linux   Docker   unRAID   Windows

Getting Started

Selecting what to prefill

Warning This guide was written with Linux in mind. If you are running BattleNetPrefill on Windows you will need to substitute ./BattleNetPrefill with .\BattleNetPrefill.exe instead.

Prior to prefilling for the first time, you will have to decide which apps should be prefilled. This will be done using an interactive menu, for selecting what to prefill from all of your currently owned apps. To display the interactive menu, run the following command

./BattleNetPrefill select-apps

All of your currently owned apps will be now displayed for selection. Navigating using the arrow keys, select any apps that you are interested in prefilling with space. Once you are satisfied with your selections, save them with enter.

Interactive app selection

These selections will be saved permanently, and can be freely updated at any time by simply rerunning select-apps again at any time.

Initial prefill

Now that a prefill app list has been created, we can now move onto our initial prefill run by using

./BattleNetPrefill prefill

The prefill command will automatically pickup the prefill app list, and begin downloading each app. During the initial run, it is likely that the Lancache is empty, so download speeds should be expected to be around your internet line speed (in the below example, a 300mbit/s connection was used). Once the prefill has completed, the Lancache should be fully ready to serve clients cached data.

Initial Prefill

Updating previously prefilled games

Updating any previously prefilled games can be done by simply re-running the prefill command, with the same games specified as before.

BattleNetPrefill keeps track of which version of each game was previously prefilled, and will only re-download if there is a newer version of the game available.
Any games that are currently up to date, will simply be skipped.

Prefilled game up to date

However, if there is a newer version of a game that is available, then BattleNetPrefill will re-download the game.
Due to how Lancache works, this subsequent run should complete much faster than the initial prefill (example below used a 10gbit connection). Any data that was previously downloaded, will be retrieved from the Lancache, while any new data from the update will be retrieved from the internet.

Prefill run when game has an update

Frequently Asked Questions

Can I run BattleNetPrefill on the Lancache server?

You certainly can! All you need to do is download BattleNetPrefill onto the server, and run it as you regularly would!

If everything works as expected, you should see a message saying it found the server at 127.0.0.1 Prefill running on Lancache Server

Running from a Docker container on the Lancache server is also supported! You should instead see a message saying the server was found at 172.17.0.1 Prefill running on Lancache Server in Docker

Running on the Lancache server itself can give you some advantages over running BattleNetPrefill on a client machine, primarily the speed at which you can prefill apps.
Since there is no network transfer happening, the prefill should only be limited by disk I/O and CPU throughput.
For example, using a SK hynix Gold P31 2TB NVME and running prefill --force on previously cached game yields the following performance Prefill running on Lancache Server in Docker

Detailed Command Usage

Note Detailed command documentation has been moved to the wiki : Detailed Command Usage

Updating

BattleNetPrefill will automatically check for updates, and notify you when an update is available :

Update available message

To update:

  1. Download the latest version for your OS from the Releases page.
  2. Unzip to the directory where BattleNetPrefill is currently installed, overwriting the previous executable.
  3. Thats it! You're all up to date!

Docker update:

sudo docker pull tpill90/battlenet-lancache-prefill:latest

Need Help?

If you are running into any issues, feel free to open up a Github issue on this repository.

You can also find us at the LanCache.NET Discord, in the #battlenet-prefill channel.

Additional Documentation

External Docs

Acknowledgements

battlenet-lancache-prefill's People

Contributors

comfuzio avatar justmaku avatar macwarrior avatar marlamin avatar matiaslgonzalez avatar nurtic-vibe avatar tpill90 avatar turboslayer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

battlenet-lancache-prefill's Issues

Feature Request - Add support to Download Games via their Productname

It would be awesome as an Option (before a Product is Implemented) to fill the cache when the Productname is known.

Os Product name i mean things like odin for Warzon or w3 for Warcraft 3 Reforged.

This would make it possible to fill the cache with products that are not already implemented into this software, but still can be used by more advanced Administrators

Linux: update.sh apply chmod +x to SteamPrefill on update

If update.sh is run, it would be helpful if chmod +x is re-applied to SteamPrefill.

If you wanted it to be more security conscious before the upgrade you get check if the existing older binary has the execute flag, and if it does, set chmod +x at the end of the upgrade. If not, then leave it be.

For me as a user, every time the binary is replaced with a new version, I have to re-apply chmod +x to SteamPrefill or my scripts will break.

Granted I could script the chmod +x myself as well, its a nice to have though ^^

Feature request - Total data downloaded

If you could, print this at the end of each run, then we could see what du -h shows and compare to see what's in the cache on disk vs just now downloaded.

libicu is required to run the app

The package libicu is required to run the app. This means users who choose to run the app in a docker container will have to build their own image from a Dockerfile.

The full error given when trying to run the app without libicu installed: Process terminated. Couldn't find a valid ICU package installed on the system. Please install libicu using your package manager and try again. Alternatively you can set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support. Please see https://aka.ms/dotnet-missing-libicu for more information.

ARM support?

Hi,
It would be really cool if i can run this script on my low powered raspberry pi,
so that i don't need my main pc to download games overnight,
it will be comparing 4w power consumption to 100w.

hardware:
pi4B

4GB of ram
64bit system

thanks

Speed drop after a few seconds

Describe the bug
I prefilled overwatch into lancache and started a download from my pc. It starts with 70 MBps and then drops dramatically after a few seconds to 2-3MBps.

Additional Details

  • Platform you are running BattleNetPrefill on: Linux Ubuntu server 23.04
  • I then tried filling a game into the lancache steam cache and downloading that. And that game downloaded full speed for the total download
  • Region : Amsterdam

!!!!!!!! Unknown buildconfig variable 'build-timestamp'

Sorry for the !s, its from the log....

Describe the bug
The command docker run --rm --name prefill-battlenet --net=host --volume /vmdisk/volumes/lancache/BattleNetPrefill:/Config tpill90/battlenet-lancache-prefill:latest prefill throws this error: !!!!!!!! Unknown buildconfig variable 'build-timestamp' and redownloads the whole 80GB every time I run the container.

docker run --rm --name prefill-battlenet --net=host --volume /vmdisk/volumes/lancache/BattleNetPrefill:/Config tpill90/battlenet-lancache-prefill:latest prefill
[11:01:23 AM] Prefilling 1 products 

[11:01:23 AM] Starting Diablo 4
Detecting Lancache server...
[11:01:23 AM] Detected Lancache server at level3.blizzard.com [192.168.1.202]

Getting latest config files...
!!!!!!!! Unknown buildconfig variable 'build-timestamp'
Building Archive Indexes...
Determining files to download...
[11:01:32 AM] Retrieved product metadata                         08.9542
[11:01:32 AM] Downloading 80.98 GiB
Downloading..: 0%

Additional Details

  • Ubuntu
  • Region : Germany

Screenshots
If applicable, add screenshots to help explain your problem.

prefill wow issue

root@lancache:/scripts/BattleNetPrefill-1.5.0-linux-x64# ./BattleNetPrefill prefill --products wow
[1:12:21 AM] Prefilling 1 products

[1:12:21 AM] Detected Lancache server at level3.blizzard.com [192.168.1.243]
[1:12:21 AM] Starting World Of Warcraft
System.IO.IOException: The response ended prematurely.
at async ValueTask System.Net.Http.HttpConnection.FillAsync(bool )
at async Task System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream , bool , ulong , int , CancellationToken )
at async Task System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task , CancellationToken )
at async Task<byte[[]]> BattleNetPrefill.Web.CdnRequestManager.GetRequestAsBytesAsync(Request request, ProgressTask task)
at async Task<byte[[]]> BattleNetPrefill.Web.CdnRequestManager.GetRequestAsBytesAsync(Request request, ProgressTask task)
at async Task<byte[[]]> BattleNetPrefill.Web.CdnRequestManager.GetRequestAsBytesAsync(Request request, ProgressTask task)
at void BattleNetPrefill.Web.CdnRequestManager.<>c__DisplayClass18_0.<b__0>d.MoveNext()
at async Task Polly.Retry.AsyncRetryEngine.ImplementationAsync(Func<Context, CancellationToken, Task> action, Context context, CancellationToken cancellationToken, ExceptionPredicates
shouldRetryExceptionPredicates, ResultPredicates shouldRetryResultPredicates, Func<DelegateResult, TimeSpan, int, Context, Task> onRetryAsync, int permittedRetryCount, IEnumerable sleepDurationsEnumerable,
Func<int, DelegateResult, Context, TimeSpan> sleepDurationProvider, bool continueOnCapturedContext)
at async Task Polly.AsyncPolicy.ExecuteAsync(Func<Context, CancellationToken, Task> action, Context context, CancellationToken cancellationToken, bool continueOnCapturedContext)
at async Task<byte[[]]> BattleNetPrefill.Web.CdnRequestManager.GetRequestAsBytesAsync(RootFolder rootPath, MD5Hash hash, bool isIndex, bool writeToDevNull, long? startBytes, long? endBytes)
at async Task<Dictionary<MD5Hash, ArchiveIndexEntry>> BattleNetPrefill.Handlers.ArchiveIndexHandler.ProcessArchiveAsync(CDNConfigFile cdnConfig, int start, int finish)
at async Task BattleNetPrefill.Handlers.ArchiveIndexHandler.BuildArchiveIndexesAsync(CDNConfigFile cdnConfig)
at void BattleNetPrefill.TactProductHandler.<>c__DisplayClass4_0.<b__0>d.MoveNext()
at void Spectre.Console.Status.<>c__DisplayClass16_0.<b__0>d.MoveNext() in Status.cs:79
at void Spectre.Console.Status.<>c__DisplayClass17_01.<<StartAsync>b__0>d.MoveNext() in Status.cs:120 at void Spectre.Console.Progress.<>c__DisplayClass28_01.<b__0>d.MoveNext() in Progress.cs:133
at async Task Spectre.Console.Internal.DefaultExclusivityMode.RunAsync(Func<Task> func) in DefaultExclusivityMode.cs:40
at async Task Spectre.Console.Progress.StartAsync(Func<ProgressContext, Task> action) in Progress.cs:116
at async Task Spectre.Console.Status.StartAsync(string status, Func<StatusContext, Task> func) in Status.cs:117
at async Task Spectre.Console.Status.StartAsync(string status, Func<StatusContext, Task> action) in Status.cs:77
at async Task BattleNetPrefill.TactProductHandler.ProcessProductAsync(bool skipDiskCache, bool forcePrefill)
at async ValueTask BattleNetPrefill.CliCommands.PrefillCommand.ExecuteAsync(IConsole console)

Can't complete prefill on slow internet connection

Describe the bug

comet424 on Discord

i have a like 3 megabit download speed.. and i find the prefill constantly failing... it took more then a week to download 80 gig for say world of warcraft and it didnt work right.. so i dont have pics and i find it the same issue with Epic and steam.. so what happens is it says downloading 0/80 and it will eventually get to 80 gig somethings not.. but then what happens is it says download retry 1 and then sometimes it does 0/80 but bit faster sometimes is like 0/56gb and then it downlaods for a few days.. and can fail and then download retry 2 3 .. and if it does get to 80/80 then does a retry and says 0/16gb and then u try to download and it gets close and retry.. and then after teh 3rd retry it will exit out and say like 320 files didnt download something like that i cant do pics as i not on my network today to post pics but i get this same errors for epic steam battlenet... it seems to be better just to download through like epic launcher etc.. it just seems to download through the prefill is slower then the launchers and it constnatly restarts it doesnt wanna resume.. where it left off.. and then kicks you out... id get pics tomorrow or so when i retry if needed.. but i figure maybe its due to slow internet connection it doesnt like it etc..

ah ok ya i gave up on prefill and was resuming my blizzard updates through launcher.. and it ya i dunno if i had fast internet wouldnt be a problem reason i like this lancache for epic games cuz you just cant copy the game files to a new install of windows.. so the lancache works just i cant seem to get it to work right on any of the games.. if the games are small on epic i found it worked those be the free games and are like under 1gb in size.. i not sure if its the size of the game that is the issue.. i cant remember the game on epic at the moment but its like game 1 of 10 2 of 10 its called and those worked.. but for the bigger games it craps out. not sure if that helps either.. i just remember that seems the small games it worked but nothing bigger then 1gb i think it was where i had issues

Steps to reproduce

  • Set QoS on router to throttle the Lancache server to less than 2mbit/s
  • Download WoW, ensuring that it is uncached.
  • You should see requests failing due to timeout

Additional Details

  • Platform you are running BattlenetPrefill on:
  • Discord Username : comet424
  • Region :

Screenshots
If applicable, add screenshots to help explain your problem.

Feature Request - Add Support for Diablo 4

Diablo 4 is coming/is downloadable
It is found under the Path of /tpr/fenris. Due to the nature of the Naming Scheme of other Activision-Blizzard Games it seemse like this is the Productname on their CDN

Blizzard games not selected, had to reinstall.

Describe the bug
Noticed this morning that all of my Blizzard games were not selected. Had to select them on prefill again and re install. Using all three prefills, Blizzard, Steam and Epic.

Apologies had opened a issue in Steam prefill.

Additional Details

  • Platform you are running SteamPrefill on: Ich777 prefill docker container
  • Discord Username (if you are part of the Lancache Discord Server) : SShizn
  • Region : US East Coast

Screenshots
If applicable, add screenshots to help explain your problem.

[Feature] Add a 'select-apps' interactive prompt, to choose which games to prefill

This should implement an select-apps command that displays an interactive prompt that allows users to choose which games to prefill. It should be implemented similarly to how it is done in SteamPrefill
This would improve the overall usability for users, as they will no longer need to know the relationship between game titles <-> product codes.
It would additionally bring BattleNetPrefill to have more consistency across all of the prefill apps.

image

Task List

  • Determine implementation details

[Docs] Add instructions on how to update app

Currently, the readme.md only has documentation for initial setup of the app.

image

This documentation should be updated to include a section on how to update the app if you have already previously downloaded it, and are currently using it.

Crash at download start

The program crashes with the following error (the same in Windows and Linux):

PS C:\Users\sascha\BattleNetPrefill-1.5.0-win-x64> .\BattleNetPrefill.exe prefill --products d3 wow wow_classic
[11:27:56 PM] Prefilling 3 products

[11:28:31 PM] Starting Diablo 3
System.UriFormatException: Invalid URI: The hostname could not be parsed.
  at void System.Uri.CreateThis(string , bool , UriKind , in UriCreationOptions )
  at System.Uri..ctor(string uriString)
  at async Task<byte[[]]> BattleNetPrefill.Web.CdnRequestManager.GetRequestAsBytesAsync(Request request, ProgressTask
     task)
  at void BattleNetPrefill.Web.CdnRequestManager.<>c__DisplayClass18_0.<<GetRequestAsBytesAsync>b__0>d.MoveNext()
  at async Task<TResult> Polly.Retry.AsyncRetryEngine.ImplementationAsync<TResult>(Func<Context, CancellationToken,
     Task<TResult>> action, Context context, CancellationToken cancellationToken, ExceptionPredicates
     shouldRetryExceptionPredicates, ResultPredicates<TResult> shouldRetryResultPredicates,
     Func<DelegateResult<TResult>, TimeSpan, int, Context, Task> onRetryAsync, int permittedRetryCount,
     IEnumerable<TimeSpan> sleepDurationsEnumerable, Func<int, DelegateResult<TResult>, Context, TimeSpan>
     sleepDurationProvider, bool continueOnCapturedContext)
  at async Task<TResult> Polly.AsyncPolicy.ExecuteAsync<TResult>(Func<Context, CancellationToken, Task<TResult>> action,     Context context, CancellationToken cancellationToken, bool continueOnCapturedContext)
  at async Task<byte[[]]> BattleNetPrefill.Web.CdnRequestManager.GetRequestAsBytesAsync(RootFolder rootPath, MD5Hash
     hash, bool isIndex, bool writeToDevNull, long? startBytes, long? endBytes)
  at async Task<BuildConfigFile> BattleNetPrefill.Parsers.BuildConfigParser.GetBuildConfigAsync(VersionsEntry
     versionsEntry, CdnRequestManager cdnRequestManager, TactProduct targetProduct)
  at void BattleNetPrefill.TactProductHandler.<>c__DisplayClass4_0.<<ProcessProductAsync>b__0>d.MoveNext()
  at void Spectre.Console.Status.<>c__DisplayClass16_0.<<StartAsync>b__0>d.MoveNext() in Status.cs:79
  at void Spectre.Console.Status.<>c__DisplayClass17_0`1.<<StartAsync>b__0>d.MoveNext() in Status.cs:120
  at void Spectre.Console.Progress.<>c__DisplayClass28_0`1.<<StartAsync>b__0>d.MoveNext() in Progress.cs:133
  at async Task<T> Spectre.Console.Internal.DefaultExclusivityMode.RunAsync<T>(Func<Task<T>> func) in
     DefaultExclusivityMode.cs:40
  at async Task<T> Spectre.Console.Progress.StartAsync<T>(Func<ProgressContext, Task<T>> action) in Progress.cs:116
  at async Task<T> Spectre.Console.Status.StartAsync<T>(string status, Func<StatusContext, Task<T>> func) in Status.cs:
     117
  at async Task Spectre.Console.Status.StartAsync(string status, Func<StatusContext, Task> action) in Status.cs:77
  at async Task<ComparisonResult> BattleNetPrefill.TactProductHandler.ProcessProductAsync(bool skipDiskCache, bool
     forcePrefill)
  at async ValueTask BattleNetPrefill.CliCommands.PrefillCommand.ExecuteAsync(IConsole console)

Unverified .dylib and ./BattleNetPrefill files

Seems like OSX is flagging each file (in some cases the same DLL multiple times) as coming from an unverified developer. Maybe there's a way to sign it so it doesn't get flagged. Not a blocker since I can go to System Preferences -> Security and Privacy -> General Tab and “Allow Anyway” each instance. This does happen once you can get the CLI running and actually try to download a cache additionally so just a bit of a speed bump.

BattleNetPrefill cannot be

Documentation Updates

  • Switch docs over to using variable replacement version of the documentation.
  • Add python script to generate .svg screenshots from .ansi files, and update readme with new images. See : tpill90/steam-lancache-prefill#308
  • Add copy button to MKdocs code snippets. See pull request tpill90/steam-lancache-prefill#309
  • Install section in readme should match SteamPrefill's install section.
  • FAQ
    • Can I download games that I don't own?
  • Write documentation for select-apps on detailed command usage page.
  • Re-record cast on prefill's detailed command usage page. Has Steam specific output in it.

Download hangs and won't finish

image

2022/08/12 11:38:15 [error] 1937#1937: *363188 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.168.200, server: , request: "GET /tpr/fore/data/83/2e/832e640c8add75816edaf21e452fd165 HTTP/1.1", subrequest: "/tpr/fore/data/83/2e/832e640c8add75816edaf21e452fd165", upstream: "[http://8.238.38.254:80/tpr/fore/data/83/2e/832e640c8add75816edaf21e452fd165](http://8.238.38.254/tpr/fore/data/83/2e/832e640c8add75816edaf21e452fd165)", host: "level3.blizzard.com"

OSX Release Folder Structure Bugged

Downloading the latest OSX release and extracting it yielded files that looked like these. Had to manually remove the folder name prefix to get it working!

May 8, 2022 at 118 PM

README update

README could benefit from a small note that instructs OSX and / or Linux users to run chmod +x .\BattleNetPrefix as users might not be familiar enough with Bash to know that off hand.

./BattlenetPrefill select-apps is not working

Describe the bug
According to the readme, when I enter ./BattlenetPrefill select-apps is not working. Shows unexpected parameters.
image

But if I run ./BattlenetPrefill list-products like before it works fine.
image
What is the right way to perform this?

Parallel request for downloading

When it's using with lancache project some CDN make drop down speed such as blizzard or origin.
If's stream is doubles speed up significally, so need some option like a:

--parallel-download 1-4 (1 by default)

to resolve issue with slow speed of this CDN's.

Feature Request - Include/Utilise XDG / Appdata Locations

G'day! Firstly, thanks for this package and the other lancache prefill packages 😄

I'd like to utilise the applications on my desktop but have the config and prefill payloads save to a standard location irrespective of when I run the application. I believe if the project is open to it, XDG for Linux/MacOS would make sense, and an Appdata folder on Windows.
On the inability to identify the correct directory, the fallback would be current design of the applications execution context.

My C# is rusty, but happy to give it a crack if the idea is considered suitable - it may be worth noting this is something I'd love to see across epic, steam and battlenet but the respective code for each is not within the lancache prefill module - keen to hear your thoughts on where the code would live if you're open to the above proposed feature @tpill90

Thanks!

[Tech Debt] Cleanup build warnings

This project's build is currently throwing various warnings

Task List

  • Resolve all build/analyzer warnings
  • Resolve SingleFile/Trim publish warnings
  • Configure Github Actions to fail the build when there are any warnings

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.