1thorsten / http-over-all Goto Github PK

At the moment the force update mechanism ensures that the process can only be called once, that's ok.
But after the first force update is processed, the waiting ones are processed as well.

This makes no sense, because the same update tasks will run again in no time.
It would be better if the waiting tasks are also finished with the termination of the initial call.

Windows clients can not connect.
https://www.webdavsystem.com/server/access/windows
https://webdav.io/webdav/webdav-operating-systems/webdav-windows/
https://decatec.de/home-server/owncloud-9-auf-ubuntu-server-16-04-lts-mit-nginx-mariadb-php-7-und-lets-encrypt/comment-page-3/

HEAD requests makes no sense !!
172.30.0.2 - - [10/Mar/2021:09:10:51 +0100] "HEAD /sce-scripts-aerownapi/softconex/sce-scripts/ibe-scripts/data/de/tourconex/geo/GeoObjectTranslation.ods HTTP/1.1" 200 0 "-" "Apache-HttpClient/4.5.12 (Java/11.0.9.1)"

172.30.0.2 - - [10/Mar/2021:09:10:51 +0100] "HEAD /sce-scripts-aerownapi/softconex/sce-scripts/ibe-scripts/data/de/tourconex/geo/GeoObjectTranslation.ods HTTP/1.1" 200 0 "-" "Apache-HttpClient/4.5.12 (Java/11.0.9.1)"

172.30.0.2 - - [10/Mar/2021:09:10:51 +0100] "GET /sce-scripts-aerownapi/softconex/sce-scripts/ibe-scripts/data/de/tourconex/geo/GeoObjectTranslation.ods HTTP/1.1" 200 130486 "-" "Apache-HttpClient/4.5.12 (Java/11.0.9.1)"

Add an option for a proxy that points directly to the source.
Especially for local Unix sockets

introduce some new options
RESOURCE_LOG_ACCESS
RESOURCE_LOG_ERROR

maybe some debug options as well
LOG_DEBUG_HEADER

http-over-all does not recognize changes to files and returns a cached file instead of the new file.

if a resource is complety removed that this resource should be deleted in filesystem.

umlauts are not working

Specified capabilities:
privileged: true
cap_add:
- SYS_ADMIN
- DAC_READ_SEARCH

I thought this was necessary because of mount.cifs (but removing cap_add has no effect on starting http-over-all with smb-shares). This must be checked with all protocols (especially nfs).

Restrict access to read operations per default

up to now:
dav_methods PUT DELETE MKCOL COPY MOVE;

perhaps we could disable dav at all in this case ?!?!

due to missing access rights executing force-update.sh from php does not work

for protocol git specified credentials are ignored

remove building part from docker-compose.yml otherwise standalone running is not possible

At the moment it is only possible to specify directories to include (DOCKER_[COUNT]_SRC_DIRS).

https://stackoverflow.com/questions/984204/shell-command-to-tar-directory-excluding-certain-files-folders

introduce a new option to set the port
SSH_[COUNT]_PORT -> default = 22

the filesystem is mounted only as read-only

For operation with connected servers, the call to force-update must also be executed on these servers.

View action is not working on dav resources

no-dav (is working):
http://localhost:8338/git_sds/README.md?view

dav (is not working):
http://localhost:8338/dav/git_sds/README.md?view

https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/#

Byte-Range Caching
Purging Data

https://github.com/awslabs/amazon-ecr-credential-helper

last modification is set to date when the file was checked out

use /usr/share/rsync/scripts/git-set-file-times (with option --quiet) to fix the date

there were a problem regarding a trailing @ character. This was solved with RELEASE 1.0.21

HTTP 401 is at the moment sign for accessibility but should not.

SMB_1_xxx
SMB_3_xxx

At the moment SMB_3 is not recognized because SMB_2 is missing.

root@ad0e958c9eca:/scripts# env | grep "^GIT_" | sort
GIT_1_NAME=git_ewake
GIT_1_REPO_BRANCH=main
GIT_1_REPO_URL=https://github.com/1thorsten/ewake.git
GIT_2_NAME=git_timeset
GIT_2_REPO_BRANCH=master
GIT_2_REPO_URL=https://github.com/1thorsten/timeset.git
GIT_3_NAME=git_sds
GIT_3_REPO_BRANCH=main
GIT_3_REPO_URL=https://github.com/1thorsten/http-over-all.git

root@ad0e958c9eca:/scripts# env | grep "^GIT_" | awk -F '_' '{print $2}' | sort | uniq 
1
2
3

improve host ranges and detection

if I try to use a DAV share from another http-over-all I get an error
/sbin/mount.davfs: mounting failed; the server does not support WebDAV

[RES]_CACHE | cache resources through proxy cache

seems to work almost out of the box. Only the PHP_VERSION has to be adjusted (7.3 -> 7.4)

checked with the following protocols:
git, smb, docker

max_input_time = -1 -> Unlimited

https://cetteup.com/36/cant-create-or-delete-directories-on-an-nginx-webdav-server-here-is-how-to-fix-that/
https://serverfault.com/questions/901325/how-to-rewrite-webdav-http-destination-request-header-on-nginx

https://www.robpeck.com/2020/06/making-webdav-actually-work-on-nginx/

Unix sockets are not currently supported. The http reachability check prevents working with Unix sockets.

SMB_1_SUB_DIR_PATH_2=Scanner
SMB_1_SUB_DIR_NAME_2=darkside
SMB_1_SUB_DIR_PERMITTED_RESOURCES_2=permitted.txt
SMB_1_SUB_DIR_PATH_3=/
SMB_1_SUB_DIR_NAME_3=root

At the moment SMB_1_SUB_DIR_PATH_2 is not recognized because SMB_1_SUB_DIR_PATH_1 is missing.

root@ad0e958c9eca:/scripts# env | grep "^SMB_1_SUB_DIR_" 
SMB_1_SUB_DIR_PATH_2=Scanner
SMB_1_SUB_DIR_NAME_2=darkside
SMB_1_SUB_DIR_PERMITTED_RESOURCES_2=permitted.txt
SMB_1_SUB_DIR_PATH_3=/
SMB_1_SUB_DIR_NAME_3=root
SMB_1_SUB_DIR_PERMITTED_RESOURCES_3=permitted_root.txt

Passwords must be accessible in plain text, because there is no control over the various sources. If necessary, the password must be sent in plain text.
However, this is no excuse for storing them in plain text in the configuration.

What is possible:

• save encrypted password (use CRYPT_KEY which is also used for link encryption)
• outsourcing the defining of CRYPT_KEY (separate config file; should be recommend as there is no obligation to do this)

Before:

CRYPT_KEY=2+oHyXHtwC6uxkliy4/ebM5aITT4eF9L2+Sf4YyIVUU=\n%
#-------------------
LOCAL_1_PATH=/local-data
LOCAL_1_NAME=local
LOCAL_1_DAV=true
LOCAL_1_DAV_AUTH=1thorsten:webdav

After (CRYPT_KEY is defined elsewhere)

#-------------------
LOCAL_1_PATH=/local-data
LOCAL_1_NAME=local
LOCAL_1_DAV=true
LOCAL_1_DAV_AUTH=1thorsten:{crypt:vaGcnaDY3T7YsFW9IQDKKIppJms68JVJ84ccg9c}