Disable loading meta info when open filder about audioserve HOT 27 CLOSED

Actually metadata scanning is now deeply build into the logic now - especially for transcoding (need to know bitrate before playing) . So this would be significant change to app, which I'm not planning to do now (possible solution would be to load metadata asynchronously). The intent was to keep it simple stupid, this is unfortunately more stupid part of it - it's also mentioned in readme - huge folders with over 200 files are bit slow.
My largest folder is now 373 audio files - and it took 15 secs to load initially. And I have to admit I also have problem in Android client - look like timeout there is around 10 secs for response to start.
However the second attempt is much faster (100ms) as OS already cached disk reads. So I just reload folder and it works.

I think probably bug should be opened against android client to extend the timeout there, 10 secs is too small.

For web client assume it works fine you, you just need to wait?

from audioserve.

Actually found one with 778 files - initial listing 29 secs, next one (when cached) 190 ms. Android client has also it's own caching. So reloading folder could work now as workaround.

from audioserve.

I am using a smb:// mounded folder in the AudioServe. That's properly why it is so slow every time (no filesystem cache at all) and double the time you mentioned.

from audioserve.

I see - unfortunately now audioserve is not accustomed for this scenario - and as I have written it now depends on metadata be part of folder listing. Still I guess the second listing of same folder should benefit on some caching on samba server side.

from audioserve.

Would adding a database be completely out of scope? Maybe have something like: open folder of files, is there a database entry, yes, display details, no, grab the details and add to database. This would let you query the database regardless of if you are using the web client or android, and there would be less penalty for using SMB.

I understand you are trying to KISS, but the addition of a database would allow to store things like books you are currently listening to, playback status (I know there is a current implementation using websockets, but you could update the database as well to make it more reliable).

If not a proper database, how about a JSON file that gets generated in the route? File exists, get data from it, if not, create one. You could also store in there groups and play position.

It would also be nice if you could get the total time of all the chapters, and the current position within that total time. So it would be possible to treat a book as a single book rather than a bunch of chapter files.

from audioserve.

@KodeStar
Thanks for ideas!
It's not completely out of scope. I understand that another level of functionality would be enabled by this.
Inital design was to use file system and see how you can get with that - and it provided fair level of functionality ( consider what git can do with just file system).
But database can provide another level of functionality and I think we did not need fully blown db, decent embedded key value store will do the job. Actually now I'm using custom KV for several tasks - playback position tracking, transcoding cache.

For JSON file caching directory structure - yes this can be viable solution. I still would prefer to have collections directory read only, but there can be shadow structure, but good KV store will probably make better job.

from audioserve.

@izderadicka I can understand where you are coming from, and I don't really disagree, the fact you are using the filesystem is what initially drew my interest.

I'm not even really affected by the issue because most of my files are single m4b's.

Also, as you say, collections being read only is a benefit really.

I'm really tempted to try making a nice front end for this (even if I'm the only one that uses it), my only hesitation is I've never used rust so making changes to the backend code base is going to be a big learning curve (plus I'd have to maintain a permanent fork if you weren't interested in the changes) and I have enough spinning software plates as it is.

If you are available and interested to have a chat about your thoughts and intentions for the project in the future ping me on the linuxserver.io discord at https://discord.gg/9guFVhmTcq or DM me details to chat with you at your preferred location. If you aren't interested in having a chat about it, no problem :)

from audioserve.

Actually now I started to looking to KV store (sled) to store couple of information.
If you noticed there is "search cache", which already stores collections directory structure in memory for quick searches.

Based on your input I though - what if this is changed and extended and also stored on disk and provide:

• directory structure (for search)

• media files metadata (so we can quickly display directory - original problem with this issue). We can also get more info from media tags (although I did not like tags first approaches, here we can get data almost for free, as we need to extract bitrate and duration anyhow).

• aggregated info - like total folder play time

• folder playback status - like already listened - % complete etc. There has been some requests to enable to mark/identify finished audiobooks/folders - I gave it some thoughts and was looking for place where to store it - and this could be the place

• and more

I'll be looking into it, do not have much spare time now, so we will see how it'll be going.

@KodeStar
"nice front end" - definitely! I'm not UI guy and my front ends are mediocriate at best (euphemism for saying they suck). Surely there is a lot of work on front end - and I hope API can isolate you from Rust server code.

In case you'll need extension of API, just open an issue. Both web client and Android will benefit from some upgrade and facelift. iOS client is missing.

API is pretty simplistic, I hope. The only thing, that requires bit more attention is playback of transcoded streams (as size is not know in advance). If player jumps out of already cached data special treatment is needed - using seek query parameter.

from audioserve.

from audioserve.

@izderadicka having that data would be awesome. I'm having a bit of an issue with CORS at the moment though. I tried using a CORS proxy to hit the authenticate endpoint on the demo server, and that worked for OPTIONS but the actual call returns a 401 (not sure if there is an issue with my generated salted shared secret or it just doesn't like the call), so I installed the docker version on my unraid server, but still got CORS issues, so I installed the docker on my macbook but running on :3333 as the dev environment I'm building the interface in is running on 3000, but still issues with CORS.

docker run -d --name audioserve -p 3333:3000 -v /Users/admin/AudioBooks:/audiobooks -e AUDIOSERVE_SHARED_SECRET=mypass izderadicka/audioserve /audiobooks that's the command I'm running.

General:
Request URL: http://localhost:3333/authenticate
Request Method: OPTIONS
Status Code: 401 Unauthorized
Remote Address: [::1]:3333
Referrer Policy: strict-origin-when-cross-origin

Request Headers:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Access-Control-Request-Method: POST
Connection: keep-alive
Host: localhost:3333
Origin: http://localhost:3000
Referer: http://localhost:3000/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36

Any ideas?

@wtanksleyjr In folders you can have additional metadata files - first available image (jpeg or png) is taken as a cover picture and first text file (html, txt, md) is taken as description of the folder. So maybe we could use that if it exists otherwise use the KV stored metadata?

from audioserve.

from audioserve.

"nice front end" - definitely! I'm not UI guy and my front ends are mediocriate at best (euphemism for saying they suck). Surely there is a lot of work on front end - and I hope API can isolate you from Rust server code.

https://www.youtube.com/watch?v=fK7ltyFSiiI - this is what it looks like so far, it's still very early and rough but you get the idea. I'm having to fake it for now though until I can get the CORS issues sorted, so I just copied some of the responses from the demo server and used those, and made up some bits so you can get an idea of what I was thinking.

from audioserve.

@wtanksleyjr - definitely folder structure and folder and file names will stay primary source on information. I was just thinking, if we will have possibility to cache key audio metadata (which is now duration and bitrate, which are needed for playback and transcoding), we can easily add more - so info from tags - but if this will be done, I'd add them as optional additional info (I do have same inconsistent experience with tags myself, so that's why I started with audioserve).
Initial design was to use plain file system only, but it has limitations for larger collections, mainly:
a) search on folder names takes long time - this is now resolved by caching folder structure to memory
b) folders with higher number of audiofiles (>200) take too long to read, because audio metadata are extracted from each file, which requires to read some part of file (I'm using libavformat from ffmpeg).

So I was thinking about some changes which will solve both of above mentioned and enable some other functionality. But as I have written changes will definitely take time, do not have much time now. And final design is not yet determined - just now started to think about it and experiment.

from audioserve.

@KodeStar I hope you noticed that you have to start server with --cors option to enable CORS (it did work for me for local development of web client, which was served from webpack serve. CORS logic in server is very simple - if --cors switch is enabled, than it returns AccessControlAllowOrigin with origin copied from request - so it should work for for all origins.

authenticate is the only POST endpoint - so it might be using pre - flight check with options. Maybe this can have some issue on the server? Can you check with other browser - I was testing with FF and Chrome. As temporary workaround you can run server with --no-authentication --cors - this will enable you to access freely and with CORS headers.

Design of web client looks nice. Just wondering how it'll behave on small screens - though I'm no UI expert I've tried to make current web client responsive - so it's usable on small screens.

from audioserve.

@KodeStar was looking quickly to https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
I think my CORS cases did not used preflight as they were considered "simple" - however in your case it might be different - then it's possible that preflight might not be properly working on server. If you'll find an issue with this pls let me know and I'll try to fix it.

from audioserve.

@izderadicka hey, there is no mention of --cors in the readme so no, I didn't run it with that :D I'll give that a try. As for the interface it will be responsive :)

from audioserve.

I changed the command to docker run -d --name audioserve -p 3000:3000 -v /mnt/user/AudioBooks:/audiobooks -e AUDIOSERVE_SHARED_SECRET=mypass izderadicka/audioserve --cors /audiobooks But I still get the same issues, this shows up in the console log:
Access to XMLHttpRequest at 'http://192.168.0.20:3000/authenticate' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.

from audioserve.

@KodeStar
I was looking into code, I think preflight OPTIONS request is not handled correctly - so it'll need to be fixed.

In mean time - use --no-authentication to skip authentication - hopefully GET requests should work with --cors.
Or have rev.proxy - mapping some requests to api, others to client files and thus serve all from same origin. But it is bit overkill.

I've opened new issue for problem with CORS preflight - #56.

from audioserve.

Turning off authentication does work though, so at least I can work on it a bit easier now. Though if we can get the preflight working as well that would be ideal.

That was still sitting in the textarea, forgot to send it, doh.

from audioserve.

@izderadicka https://youtu.be/d3ljM_a0gLA - This is how it currently looks on mobile. It doesn't currently bring the more info panels in on mobile. I'll probably do it so there is a button to show more info if you want and then show the more info panel by default when you get to an actual book, then clicking on the Listen button or clicking on one of the files will open up the player.

from audioserve.

@KodeStar - looks really nice.

from audioserve.

@KodeStar - added fix fro CORS preflight issue #56. Now code is master - you can try to build docker image locally and see it it works.

from audioserve.

I'll have a look at that tomorrow if I get a chance, thanks.

For reference you can find the code so far for this interface thing at https://github.com/KodeStar/audioserve-ui and there is a board at https://github.com/KodeStar/audioserve-ui/projects/1

I know you started building this over 3 years ago and your interest in the project has probably waned over time. I'm also aware that you have said you don't have much time to spend on this project, so I'm trying to curb my enthusiasm on this a bit as I'm also not overly abundant with time, but I'd love to get this to a proper working state.

from audioserve.

@KodeStar I opened new issue for web client - probably we can move our future discussions there. #58

Just one comment for todos on your project - hidden, dot files are already filter out on server - for security reasons.

audioserve is still my favored project - also because I'm listening to audiobooks every day. My limited resources are just due to other involvements, which just have increased recently. I definitely will not abandon it, just do not have much time left.

from audioserve.

collection cache is now available in master and pre-release image izderadicka/audioserve:unstable

It enables very fast loads even for directories with many audiofiles - directory with app. 700 audio files folder loads in 20-80 ms. Of course on first run of audioserve it takes some time to fill cache. but since cache is persistent, next runs are OK.

from audioserve.

Awesome :)

from audioserve.

New version with collection cache released as v0.16.5

from audioserve.