kareadita / kavita Goto Github PK

ZipArchive doesn't support RAR/CBR files. Use this library instead:

https://github.com/adamhathcock/sharpcompress

This feature should support folder directory flattening till no nesting. This should allow reading zip, rar, cbz, cbr files.

The user should be able to setup logging level (which takes effect after reboot).

The user should be able to setup a backup db task.

I have rolling logs setup through logging solution.

A Volume consists of multiple Chapters in Manga. We need an ability to encapsulate and capture files that are spread either in volumes (single cbz) or chapters (multiple zip files representing one volume).

See Epic: Library Support - Series Entities for more information around representing these concepts.

For some reason, unit tests on PRs are failing, yet working locally. Investigate and fix this annoyance.

ZipArchive doesn't support RAR files. Use this library instead:

https://github.com/adamhathcock/sharpcompress

Given that a Library is represented by a collection of physical folders, Kavita needs the ability to scan said folders on both a scheduled and ad-hoc basic for new or modified content.

The goal of this task is to update the DB's entries for files and their ties to a volume, chapter, or library.

In future, this will trigger metadata resolving per series.

There is a bug in ScanLibrary for Chapters where multiple chapters get created and thus exceptions are caused by SingleOrDefault() causing whole series to be skipped.

In order to provide grouping of files to Series/Volumes, Kavita needs to be able to parse filenames and folder names in a way to group the data.

Implement a set of Parser methods to parse the following attributes:

• Series name
• Volume Number
• Chapter Number
• File type

In addition, a Match API that abstracts the underlying parser code would make integrating with the Scan Library code trivial.

The following must provide:

• Ability to store users in DB
• Ability to register new users
• Ability to login with a given username/password
• Users should have hashed/salted passwords
• Users should use JWT for authentication on future APIs

fail: API.Middleware.ExceptionMiddleware[0] Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'Could not place a lock on the resource 'HangFire:lock:recurring-job:ScannerService.ScanLibraries': The lock request timed out.' resource. Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'Could not place a lock on the resource 'HangFire:lock:recurring-job:ScannerService.ScanLibrar ies': The lock request timed out.' resource. at Hangfire.LiteDB.LiteDbDistributedLock.Acquire(TimeSpan timeout) at Hangfire.LiteDB.LiteDbDistributedLock..ctor(String resource, TimeSpan timeout, HangfireDbContext database, LiteDbStorageOptions storageOptions) at Hangfire.LiteDB.LiteDbConnection.AcquireDistributedLock(String resource, TimeSpan timeout) at Hangfire.RecurringJobExtensions.AcquireDistributedRecurringJobLock(IStorageConnection connection, String recurringJobId, TimeSpan timeout) at Hangfire.RecurringJobManager.AddOrUpdate(String recurringJobId, Job job, String cronExpression, RecurringJobOptions options) at Hangfire.RecurringJobManagerExtensions.AddOrUpdate(IRecurringJobManager manager, String recurringJobId, Job job, String cronExpression, TimeZoneInfo timeZone, String queue) at Hangfire.RecurringJob.AddOrUpdate(Expression1 methodCall, String cronExpression, TimeZoneInfo timeZone, String queue)
at Hangfire.RecurringJob.AddOrUpdate(Expression1 methodCall, Func1 cronExpression, TimeZoneInfo timeZone, String queue)
at API.Services.TaskScheduler..ctor(ICacheService cacheService, ILogger1 logger, IScannerService scannerService, IUnitOfWork unitOfWork) in C:\Users\Joe\Documents\Projects\KavitaOrg\Kavita\API\Services\TaskScheduler.cs:lin e 30 at ResolveService(ILEmitResolverBuilderRuntimeContext , ServiceProviderEngineScope ) at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp, Type type, Type requiredBy, Boolean isDefaultParameterRequired) at lambda_method327(Closure , IServiceProvider , Object[] ) at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.<CreateActivator>b__0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context) at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at API.Middleware.ExceptionMiddleware.InvokeAsync(HttpContext context) in C:\Users\Joe\Documents\Projects\KavitaOrg\Kavita\API\Middleware\ExceptionMiddleware.cs:line 30

A user should have their own settings for reading direction, page splitting, etc.

A user should be able to update them and have defaults.

The current GetDirectories from C# does not filter out System or Hidden folders.

Enhance the service to filter these folders out automatically.

Noticing a lot of issues with reading zips with ZipFile. In previous release, attempted to use SharpCompress but saw 30x speed reduction on zip operations.

In this story, fallback to another zip library (SharpZipLib) if ZipFile fails to read stream or contents inside stream.

Implement the ability to find the most appropriate cover image for a Series cover.

Generally the first image of the first volume/chapter is used, unless there is an image file named folder in the archive.

This will be overridden if there is a cover image file in the directory the archives are.

We need the ability to track for each user, how much a user has read of a given series.

The ability for a user to rate a series and give their own review for it (text). In future, this might be pulled from a linked account.

Things like automated scans, cover image calculation, etc.

Options will be:
Disabled
Daily
Every 3 days
Weekly

This is automated scans, ad-hoc will run immediately.

https://docs.hangfire.io/en/latest/deployment-to-production/making-aspnet-app-always-running.html

A user of the application should be able to configure preferences used throughout the web app.

Potential preferences could be:

• Split Options for reading
• Which side of screen is Next Page (left or right to emulate japanese book style flipping)
• Theme? (not sure if this is possible)
• Ordering of libraries on libraries page (Superhero Comics, Isekai Manga, Other Comics, Other Manga)

Currently, the rewritten ScanLibrary is completely bound to 1 thread. Now that it is battle tested, we can enhance by spreading it out onto multiple threads.

The data is already broken up in such a way that there should be minimal locking.

This task should ensure that scan library can be parallelized and that gains are significant to warrant the added complexity.

Parsing disk scans is cheap. For a decently sized library we are talking < 3 seconds.

Since it is so cheap to parse Disk IO and this doesn't have to be done in real time, we can move archive work (metadata) off into a separate task that executes after the initial DB update from Scan Library.

This will ensure the entities get entered into DB as fast as possible and metadata can follow.

This story is to investigate the performance benefit vs code complexity.

This may need to also calculate blurhashes if we go that route. Should set CoverImages on the respective entities.

A testing framework should be setup. The following need to be accomplished:

• Ability to create and run Unit Tests
• Ability to mock and run Integration tests (DB mocking)
• Ability for PRs to automatically execute all tests and fail merge check if tests fail.

A Series is the representation for a Manga, Comic, or a Book title (Each book is it's own Series (subject to change)).

A Series consists of a collection of Volumes or Chapters. Volumes/Chapters are what the user actually reads.

Most manga is downloaded as cbzs and either in volumes, but some in just chapters. Ideally, we'd want everything as volumes, but need to be prepared to either a) provide scripts to organize their media for them or b) provide a way to emulate as if they were all in volumes (cloning multiple chapters, recreating a cbz for one volume, then serving).

A Series is expected in future releases to present metadata around the manga, including genres tags, description, author, etc (See requirement doc for more). Design this with future metadata in mind.

A Series at the least needs:

• Name
• Thumb Image
• LibraryId (Foreign key)

Encourage users to use Reverse Proxy instead

During partial support implementation, many bugs have been found in ScanLibrary. Many of those bugs are fixed, but upon fixing them and refactoring the code to be more streamlined, it left me wanting a way to mock testing (have to use PrivateObjects or protected internal hacks) for the code.

I need to build out a test utility that generates fake archive files for testing and then cleans up after itself.

Implement the ability to Load a Volume/Manga file (cbz, cbr, zip, rar) and serve images to the Frontend.

The API needs to handle memory carefully and make sure all filestreams are closed when the user stops reading or closes a volume with timeouts for when a user connection severs.

When ScanLibrary is running, the main thread gets blocked and APIs and Hangfire cannot load properly.

Support for partial chapters (extras, double format)

This gives the user the ability to customize the UI visible Name/Sort Name but keep the parser still able to link the named parsed off disk to what the user sees.

Unable to restart, only shutdown. Putting on hold.

The user can search for items in libraries they have access to

A Library entity represents a ICollection of Series. A Library is bound to one of more physical folders on the disk that contain a given metadata type.

First pass implementation for a Library is:
Name,
Metadata Type (Manga, Comic, Book, Raw)
Folders
Thumb Image (image to show on the UI)

First pass implementation should only focus on MetadataType.Manga.

The following APIs are needed with DTO, Entities, and Migrations:

• GetLibraries
• GetLibrary(int libraryId)
• UpdateLibrary (update properties of it)
• DeleteLibrary
• CreateLibrary

N.B. Use Optimistic Concurency with EF (RowVersion) as libraries might be updated at same time and we need to prevent locking.

In the Kavita and Kavita-webui codebases, take care of some of the TODOs and NOTEs.

so what's happening is some volumes which are on disk and are being paresd correctly don't show up in the DB or DB entity creation code.

I traced it down to a piece of code that updates the concurent dictionary. What's happening is that I scan the directories using multiple threads for speed. Each thread finds files, parses them and tries to add them to the concurent dictionary.

Well for some reason, I just observed one thread add to the dictionary then another thread try to update the key (same key), but update failed, so it went to add to an empty dictionary. Meaning we lost the first object added.

GetLibrariesForUser is taking on average 200-300ms. Optimize this.