Giter VIP home page Giter VIP logo

epic-lancache-prefill's Introduction

epic-lancache-prefill

view - Documentation ko-fi

GitHub all releases dockerhub

Automatically fills a Lancache with games from Epic Games, so that subsequent downloads for the same content will be served from the Lancache, improving speeds and reducing load on your internet connection.

Overview

Features

  • Select apps to prefill through an interactive menu.
  • No installation required! A completely self-contained, portable application.
  • Multi-platform support (Windows, Linux, MacOS, Arm64)
  • High-performance! Downloads are significantly faster than using Epic Games, and can easily reach 10gbit/s or more!
  • Game downloads write no data to disk, so there is no need to have enough free space available. This also means no unnecessary wear-and-tear to SSDs!

Table of contents

Initial Setup

Note New to Linux? See this detailed tutorial : Install Guide For Linux Beginners

Note Interested in using Docker instead? See : Docker Setup Guide

Note Using Unraid? See : Unraid Setup Guide

  1. EpicPrefill can be run on both the Lancache server itself, or on your gaming machine as an alternative Epic client. You should decide which one works better for your use case.
  2. Download the latest version for your OS from the Releases page.
  3. Unzip to a directory of your choice

Warning Linux and macOS will require executable permissions to be granted with chmod +x ./EpicPrefill prior to running the app.

Note Alpine Linux requires additional dependencies to be installed for the .NET runtime : Alpine Linux Dependencies

Optional Windows Setup

Configuring your terminal to use Unicode will result in a much nicer experience with EpicPrefill, for much nicer looking UI output.

Initial Prefill

As the default console in Windows does not support UTF8, you should instead consider installing Windows Terminal from the Microsoft App Store, or Chocolatey.

Once Windows Terminal has been installed you will still need to enable Unicode, as it is not enabled by default. Running the following command in Powershell will enable it if it hasn't already been enabled.

if(!(Test-Path $profile))
{
    New-Item -Path $profile -Type File -Force
}
if(!(gc $profile).Contains("OutputEncoding")) 
{ 
    ac $profile "[console]::InputEncoding = [console]::OutputEncoding = [System.Text.UTF8Encoding]::new()";
    & $profile; 
}

Getting Started

Selecting what to prefill

Warning This guide was written with Linux in mind. If you are running EpicPrefill on Windows you will need to substitute ./EpicPrefill with .\EpicPrefill.exe instead.

Prior to prefilling for the first time, you will have to decide which apps should be prefilled. This will be done using an interactive menu, for selecting what to prefill from all of your currently owned apps. To display the interactive menu, run the following command

./EpicPrefill select-apps

Afterwards your Browser of choice will open to ask you to login to Epic Games. Simply login with your mail-address, your password and hopefully a mfa-token from your authenticator app of choice. Epic Login

Now copy the "authorizationCode" thats shown in your Browser and paste it in your terminal and press enter. Your login should be cached and automatically being refreshed everytime you use EpicPrefill.

Once logged into Epic Games, all of your currently owned apps will be displayed for selection. Navigating using the arrow keys, select any apps that you are interested in prefilling with space. Once you are satisfied with your selections, save them with enter.

Interactive app selection

These selections will be saved permanently, and can be freely updated at any time by simply rerunning select-apps again at any time.

Initial prefill

Now that a prefill app list has been created, we can now move onto our initial prefill run by using

./EpicPrefill prefill

The prefill command will automatically pickup the prefill app list, and begin downloading each app. During the initial run, it is likely that the Lancache is empty, so download speeds should be expected to be around your internet line speed (in the below example, a 300mbit/s connection was used). Once the prefill has completed, the Lancache should be fully ready to serve clients cached data.

Initial Prefill

Updating previously prefilled apps

Updating any previously prefilled apps can be done by simply re-running the prefill command, which will use same prefill app list as before.

EpicPrefill keeps track of which version of each app was previously prefilled, and will only re-download if there is a newer version of the app available. Any apps that are currently up to date, will simply be skipped.

Prefilled app up to date

However, if there is a newer version of an app that is available, then EpicPrefill will re-download the app. Due to how Lancache works, this subsequent run should complete much faster than the initial prefill (example below used a 10gbit connection). Any data that was previously downloaded, will be retrieved from the Lancache, while any new data from the update will be retrieved from the internet.

Prefill run when app has an update

Frequently Asked Questions

Can I run EpicPrefill on the Lancache server?

You certainly can! All you need to do is download EpicPrefill onto the server, and run it as you reguarly would!

If everything works as expected, you should see a message saying it found the server at 127.0.0.1

Prefill running on Lancache Server

Running from a Docker container on the Lancache server is also supported! You should instead see a message saying the server was found at 172.17.0.1

Prefill running on Lancache Server in Docker

Running on the Lancache server itself can give you some advantages over running EpicPrefill on a client machine, primarily the speed at which you can prefill apps.
Since there is no network transfer happening, the prefill should only be limited by disk I/O and CPU throughput.
For example, using a SK hynix Gold P31 2TB NVME and running prefill --force on previously cached game yields the following performance Prefill running on Lancache Server in Docker

Can EpicPrefill be run on a schedule?

Yes it can! Scheduled jobs can be easily setup on Linux using crontab, and can be flexibly configured to run on any schedule that you desire. Jobs are configured by specifying an "expression" that describes the schedule to run on. Some examples of cron expressions:

Schedule Cron Expression
Every day at 2am

0 2 * * * $PWD/EpicPrefill prefill

Every 4 hours

0 */4 * * * $PWD/EpicPrefill prefill

If the above examples don't cover your use case, crontab.guru is an online cron expression editor that can interactively edit cron expressions, and explain what they mean.

Once you have determined a cron expression, you can then create a job using the following:

Note This command should be run in the same directory where EpicPrefill is installed

job="cron expression here"; { crontab -l; echo "$job"; } | crontab -

After running the command, you can verify that the job was successfully created with crontab -l. If the output matches below, then your job is correctly configured!

Crontab jobs

Can I fill my cache using previously installed Epic games?

Unfortunately it is not possible to fill a Lancache using games that have been installed with Epic. The installed games are in a different format than what Lancache caches, as they are decrypted and unzipped from the raw request. The decryption/unzip process is not reversible. Thus, the only way to get games properly cached is to redownload them using either EpicPrefill or Epic

How can I limit download speeds?

You may want to limit the download speed of EpicPrefill to prevent it from potentially saturating your entire connection, causing other devices to suffer from massive latency and poor speeds. This issue is known as bufferbloat, and more detailed information on the issue can be found here: What is bufferbloat?

EpicPrefill does not currently contain any functionality to limit its own download speed, and due to the way that downloads are implemented will likely never be able to throttle its own download speed. Additionally, even if EpicPrefill was able to throttle itself, the same issue would persist with downloads through Epic.

One method to limit bandwidth would be to configure Quality of Service (QOS) on your router, limiting bandwidth to the Lancache server, or by prioritizing other network traffic. A general overview of QOS can be found here : Beginners guide to QOS

For more brand specific guides (non-exhaustive), see :

Detailed Command Usage

Note Detailed command documentation has been moved to the wiki : Detailed Command Usage

Updating

EpicPrefill will automatically check for updates, and notify you when an update is available :

Update available message

Automatically updating

  • Windows
    • Run the .\update.ps1 script in the executable directory
  • Linux
    • First time only : Grant executable permissions to the update script with chmod +x ./update.sh
    • Run the ./update.sh script in the executable directory

Manually updating:

  1. Download the latest version for your OS from the Releases page.
  2. Unzip to the directory where EpicPrefill is currently installed, overwriting the previous executable.
  3. Thats it! You're all up to date!

Docker updating:

  • sudo docker pull tpill90/epic-lancache-prefill:latest

Need Help?

If you are running into any issues, feel free to open up a Github issue on this repository.

You can also find us at the LanCache.NET Discord, in the #epic-prefill channel.

Additional Documentation

Acknowledgements

  • @Joly0 for all your help with testing!

epic-lancache-prefill's People

Contributors

comfuzio avatar joly0 avatar nurtic-vibe avatar tpill90 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

epic-lancache-prefill's Issues

Missing `--nocache`

Unlike BattleNetPrefill and SteamPrefill, EpicPrefill does not contain --nocache flag, and/or it is not working.

Consider a situation where one has a Lancache and for example deletes the cache, then runs ./EpicPrefill prefill, the prefill will just tell user "nothing to download", which is not true, as the actual cache is empty.

..or is the EpicPrefill so wise that it can deal with a situation as mentioned above? If so, then both BattleNetPrefill and SteamPrefill would need the same kind of brains.

image

Switch docs over to using variable replacement

Describe the bug
A clear and concise description of what the bug is.

Additional Details

  • Platform you are running EpicPrefill on: [e.g Windows, Linux, Docker]
  • Discord Username (if you are part of the Lancache Discord Server) : [e.g username#0000]
  • Region : [e.g US East Coast, Germany, Australia]

Screenshots
If applicable, add screenshots to help explain your problem.

download does not progress

Enviroment:

  • LanCache 192.168.1.30 running on Unraid
  • LanCacheDNS 192.168.1.33 running on Unraid
  • LanCachePrefill running on Unraid

What works:

  • Steam Prefill works fine
  • Battle.net Prefill works fine
  • All gamelaunchers (Epic, steam, EA, battlenet, GoG) work fine on my windows machines and use the LanCache. In the image below you can see the Epic Launcher download Fortnite which goes through the LanCache at the maximum speed of my internet connection (Fornite was not in the LanCache yet)

epic_launcher

What does not work:

While I can completelly configure EpicPrefill in both the Unraid Docker as well as the standanlone windows application, the prefill download does not progress and stays at 0% with no network traffic.

The bandwitdh value shown in the Unraid Docker version is "broken" as it shows several Gbps which my internet connection is not capable of - also there is no progress and no drive activity on my LanCache nvme.

Unraid_docker

The bandwitdh indicator in the windows build is more accurate as there is no traffic.

windows

LanCache Logs:
These are the logs from an attempt to prefill City Skylines

upstream-error.log
error.log
access.log
LanCache_access.log

Games tested with Unraid Prefill docker as well as the windows build:

  • Cities Skylines - fails/no progress
  • Fortnite - fails/no progress
  • Doom 64 - fails/no progress
  • Enter the Gungeon - fails/no progress
  • Rise of the Tomb Raider: 20 Year Celebration - fails/no progress

With the Epic Launcher I can download these games, even on the same machine where the EpicPrefill.exe does not produce any download traffic.

Use EpicPrefill on a game that is already in the LanCache

  1. I used the Epic Launcher to download Cities Skylines through the LanCache
  2. then I launched epicprefill.exe selected Cities Skylines and choose Yes to prefill

Result: EpicPrefill.exe "downloads" everything that the EpicLauncher put in the lancache.

cachedGame

System.ArgumentException: An item with the same key has already been added. Key: AnimStarterPack

[11:24:20 PM] If the web page did not open automatically, please manually open the following URL: https://legendary.gl/epiclogin
Please enter the authorizationCode from the JSON response: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[11:30:09 PM] Retrieving owned apps
System.ArgumentException: An item with the same key has already been added. Key: AnimStarterPack

Additional Details

  • Platform: Unraid Latest

Screenshots
If applicable, add screenshots to help explain your problem.

Getting 508 Response error

Describe the bug
Using the Docker image, fresh install today and showing the below error after running the select-apps option

[1:01:57 AM] Selected 13 apps to prefill!
[1:01:58 AM] Retrieving owned apps
[1:01:59 AM] Retrieved 41 apps                                   00.1535

[1:01:59 AM] Starting Destiny 2
[1:01:59 AM] Unexpected download error : Response status code does not indicate success: 508 ().
Skipping app...

Additional Details

  • Platform you are running EpicPrefill on: Docker
  • Discord Username:
  • Region :

Login fails with 400 (Bad request)

When attempting to initially login with an Epic Games account using the select-apps option, it produces the following output:
:~/EpicPrefill-1.1.0-linux-x64$ ./EpicPrefill select-apps
[1:28:40 PM] Please login into Epic via your browser
[1:28:40 PM] If the web page did not open automatically, please manually open the following URL: https://legendary.gl/epiclogin
Please enter the authorizationCode from the JSON response: [redacted]
System.Net.Http.HttpRequestException: Response status code does not indicate success: 400 (Bad Request).
at HttpResponseMessage System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at async Task EpicPrefill.Handlers.UserAccountManager.LoginAsync()
at async Task EpicPrefill.EpicGamesManager.InitializeAsync()
at async ValueTask EpicPrefill.CliCommands.SelectAppsCommand.ExecuteAsync(IConsole console)

I will update when I find out if the account has any additional MFA or other limitations that may be out of the ordinary.
When using a different account, login worked successfully.

Discord name: Ed/Megabeard

epic prefill missing # of apps completed

Other prefill apps show the following at the end of each run:

Prefilled 3 apps in 25.8501

Suggest this be added to EpicPrefill so we can compare one vs the other. :)

Thanks again for the awesome work on this solid solution!

"Cannot get the value of a token type 'Number' as a string"

Encountered this error upon first run of the select-apps command. Looks like the root error is "Cannot get the value of a token type 'Number' as a string"

System.Text.Json.JsonException: The JSON value could not be converted to System.Collections.Generic.List`1[
System.String]. Path: $[0] | LineNumber: 0 | BytePositionInLine: 8.
System.InvalidOperationException: Cannot get the value of a token type 'Number' as a string.
at string System.Text.Json.Utf8JsonReader.GetString()
at bool System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(ref Utf8JsonReader , Type ,
JsonSerializerOptions , ref ReadStack , out TCollection )
at bool System.Text.Json.Serialization.JsonConverter`1.TryRead(ref Utf8JsonReader , Type ,
JsonSerializerOptions , ref ReadStack , out T )
at T System.Text.Json.Serialization.JsonConverter`1.ReadCore(ref Utf8JsonReader , JsonSerializerOptions ,
ref ReadStack )
at void System.Text.Json.ThrowHelper.ReThrowWithPath(ref ReadStack , in Utf8JsonReader , Exception )
at T System.Text.Json.Serialization.JsonConverter`1.ReadCore(ref Utf8JsonReader , JsonSerializerOptions , ref
ReadStack )
at TValue System.Text.Json.JsonSerializer.ReadFromSpan(ReadOnlySpan , JsonTypeInfo , int? )
at TValue System.Text.Json.JsonSerializer.ReadFromSpan(ReadOnlySpan , JsonTypeInfo )
at TValue System.Text.Json.JsonSerializer.Deserialize(string , JsonTypeInfo )
at List EpicPrefill.EpicGamesManager.LoadPreviouslySelectedApps()
at async Task<List> EpicPrefill.CliCommands.SelectAppsCommand.BuildTuiAppModelsAsync(EpicGamesManager
epicManager)
at async ValueTask EpicPrefill.CliCommands.SelectAppsCommand.ExecuteAsync(IConsole console)
image

System.InvalidOperationException: Sequence contains no matching element

Describe the bug
When I run EpicPrefill prefill -f I get the error: "System.InvalidOperationException: Sequence contains no matching element"

Additional Details

  • Platform you are running EpicPrefill on:

Debian GNU/Linux 11

Linux lancache 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux on bare metal

  • Discord Username (if you are part of the Lancache Discord Server) : billcat
  • Region : Central US

Screenshots
user1@lancache:/EpicPrefill$ ./update.sh
Checking for latest version
Found latest version : 1.1.1
Downloading...
EpicPrefill-1.1.1-linux-x64.zip 100%[======================================================================================================================>] 12.67M --.-KB/s in 0.1s
Unzipping...
Complete!
user1@lancache:
/EpicPrefill$ ./EpicPrefill clear-cache
[10:25:49 AM] Found 3 cached files, totaling 2.61 MiB
[10:25:50 AM] Done!
user1@lancache:~/EpicPrefill$ ./EpicPrefill prefill -f
[10:26:06 AM] Reusing existing auth session...
[10:26:06 AM] Retrieving owned apps
[10:26:10 AM] Loaded new app metadata for 16 apps
[10:26:10 AM] Retrieved 16 apps 03.7645

[10:26:10 AM] Starting Satisfactory Early Access
[10:26:10 AM] Downloaded manifest 00.0498
[10:26:11 AM] Downloading 9.02 GiB
[10:26:11 AM] Detected Lancache server at localhost [127.0.0.1]
[10:26:17 AM] Finished in 06.3728 - 12.15 Gbit/s

[10:26:17 AM] Starting Satisfactory Experimental
[10:26:17 AM] Downloaded manifest 00.0437
[10:26:17 AM] Downloading 9.02 GiB
[10:26:23 AM] Finished in 05.9489 - 13.02 Gbit/s

ERROR
System.InvalidOperationException: Sequence contains no matching element
at System.Linq.ThrowHelper.ThrowNoMatchException()
at System.Linq.Enumerable.First[TSource](IEnumerable1 , Func2 )
at EpicPrefill.EpicGamesManager.DownloadMultipleAppsAsync(Boolean downloadAllOwnedGames, List1 manualIds) at EpicPrefill.CliCommands.PrefillCommand.ExecuteAsync(IConsole console) at CliFx.CliApplication.RunAsync(ApplicationSchema applicationSchema, CommandInput commandInput) in /D:\a\CliFx\CliFx\CliFx\CliApplication.cs:147 at CliFx.CliApplication.RunAsync(IReadOnlyList1 commandLineArguments, IReadOnlyDictionary`2 environmentVariables) in /D:\a\CliFx\CliFx\CliFx\CliApplication.cs:191
user1@lancache:~/EpicPrefill$

Unexpected download error : A connection attempt failed because the connected party did not properly

[10:02:12 AM] Starting A Short Hike [10:02:34 AM] Unexpected download error : A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (fastly-download.epicgames.com:443) Skipping app...

From a PCAP (172.16.1.40 is lancache instance):
36 3.755003 172.16.1.100 172.16.1.40 TCP 66 53437 → 443 [SYN] Seq=0 Win=62720 Len=0 MSS=8960 WS=256 SACK_PERM

Same instance works fine using epicprefill on OSX

Trying to see why its requesting port 443 on .exe but 80 on the OSX version

From OSX
104 10.082232 172.16.1.110 172.16.1.40 TCP 78 62416 → 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=2083870588 TSecr=0 SACK_PERM=1

Feature request - Total data downloaded

If you could, print this at the end of each run, then we could see what du -h shows and compare to see what's in the cache on disk vs just now downloaded.

Having this in all three programs would be great because then you can bash script running each of them and see what the total is.

Thanks!

update.sh always updates regardless of being up to date

Describe the bug
A clear and concise description of what the bug is.

Additional Details

  • Platform you are running EpicPrefill on: [e.g Windows, Linux, Docker]
  • Discord Username (if you are part of the Lancache Discord Server) : [e.g username#0000]
  • Region : [e.g US East Coast, Germany, Australia]

Screenshots
If applicable, add screenshots to help explain your problem.

Fornite can't be downloaded

I added Fornite and get this error:

[4:37:22 PM] Starting Fortnite
[4:37:23 PM] Unexpected download error : Index was outside the bounds of the array. Skipping app...

Other apps work fine.

I worked after I a
./EpicPrefill clear-cache

Documentation Updates

  • Switch docs over to using variable replacement version of the documentation.
  • Add python script to generate .svg screenshots from .ansi files, and update readme with new images. See : tpill90/steam-lancache-prefill#308
  • Add copy button to MKdocs code snippets. See pull request tpill90/steam-lancache-prefill#309
  • Install section in readme should match SteamPrefill's install section.
  • FAQ
    • Can I download games that I don't own?
  • Write documentation for select-apps on detailed command usage page.
  • Remove references to SteamPrefill from documentation

Fortnite Download From Cache Only Completes to 28%

Describe the bug
When going through the LAN-Cache, Fortnite only completes to 28%. It doesn't matter which computer in the house I use. Epic Games launcher shows the download rate drop to 0 instantly once it reachs 28% downloaded. Filesystem was XFS on a 6TB HDD.

Additional Details

  • unRAID Server - Lancache, Lancache-DNS, and Lancache-Prefill in Docker
  • ZurkeyDurkey
  • Region : US West Coast

Screenshots
If applicable, add screenshots to help explain your problem.
access.log

[Bug] Update script still looking for SteamPrefill

Hi, was trying to get this prefill working but before starting the program i started the update.sh script
I was presented with an error so i took a look into the script and found that the #unzip and below it is still looking for SteamPrefill instead off EpicPrefill:
# Unzip
echo -e "${Yellow} Unzipping... ${NC}"
unzip -q -j -o SteamPrefill-${LATEST_TAG}-linux-x64.zip

# Required so executable permissions don't get overwritten by unzip
chmod +x SteamPrefill update.sh

# Cleanup
rm SteamPrefill-${LATEST_TAG}-linux-x64.zip

When i change the SteamPrefill to EpicPrefill the script runs without errors :)

Unexpected download error : Response status code does not indicate success: 401 (Unauthorized). Skipping app...

After 8 hours of prefill. I get errors for the rest of the list.

[3:35:54 PM] Starting theHunter: Call of the WildT
[3:35:54 PM] Unexpected download error : Response status code does not indicate success: 401 (Unauthorized).  Skipping app...

[3:35:54 PM] Starting while True: learn()
[3:35:54 PM] Unexpected download error : Response status code does not indicate success: 401 (Unauthorized).  Skipping app...

[3:35:54 PM] Prefill complete!
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  Prefilled 269 apps in 8:15:52.46

   Updated │ Up To Date │ Failed
  ─────────┼────────────┼────────
     30    │     56     │  183

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.