Example configuration for SSO about moodle-dl HOT 20 CLOSED

Yes of course the configuration file must be called config.json and is located in the folder where all files are to be downloaded.

{
    "mail": {
        "sender": "[email protected]",
        "server_host": "smtp.gmail.com",
        "server_port": "587",
        "username": "[email protected]",
        "password": "verySupersecurePassword",
        "target": "[email protected]",
        "send_error_msg": false
    },
    "sentry_dsn": "https://[email protected]/222228",
    "token": "c0000dc17yourTOKEN2e2e1111a942a0",
    "moodle_domain": "moodle.uni.de",
    "moodle_path": "/m/",
    "dont_download_course_ids":[],
    "download_submissions": true
}

moodle_path is the path after the domain. In my case Moodle is actually installed at moodle.uni.de/m/. All Urls of Moodle start with it.

The complete mail block can be omitted if you do not want to be notified by mail about changes.
Likewise, download_submissions, dont_download_course_ids and sentry_dsn can be omitted.

But the bigger problem is SSO, that was already a problem with the old version of the downloader, I had done a lot of tinkering there. That you don't get a token can also be caused by a wrong configured SSO plugin. But there is nothing you can do about that.

The normal way to get a token is with this URL https://www.yourmoodle.com/login/token.php?username=USERNAME&password=PASSWORD&service=SERVICESHORTNAME. Where for SERVICESHORTNAME normally only moodle_mobile_app is allowed.

But I read something here.
So you could generate a token by calling https://www.yourmoodle.com/m/admin/tool/mobile/launch.php?service=moodle_mobile_app&passport=12345.

You should be redirected to moodlemobile://token=$apptoken after your SSO login. Only $apptoken is not quite the token we need. According to the source code, the token is base64 encoded and includes the passed passport.
So if this really works, you have to decode the token, for example with this. Then the decoded one should consist of two or three sections divided with ':::'. The token we need is located in the second section.

You are welcome to contact me whether that works or not. I would also like to know what kind of error message comes up when you try to get a token normally, with /login/token.php?

from moodle-dl.

I have automated the process to get a token with SSO login. Check out the latest commit.

from moodle-dl.

I'm looking into it in the next days. I'll have a look at how it works with the mobile app (which works despite of sso).

Regarding the token.php API I'm a bit concerned my password could leak (don't know what came to their mind when deciding to pass passwords in URLs). There just have to be some very basic logging and all passwords are logged.
The purpose of sso is that moodle won't ever receive my password, to let it work the token.php API would have to contact e.g. an LDAP server at the backend and I doubt this is the case (then one wouldn't use sso in the first place).

Notes
php -r 'echo urldecode("string");'
https://github.com/moodle/moodle/blob/master/admin/tool/mobile/launch.php

from moodle-dl.

The mobile app uses the normal /login/token.php to get a token. Only if it detects that the moodle usese SSO then it uses /admin/tool/mobile/launch.php or other detected plugins.

The code of the app is here.

Yes logically the password could appear in logs of the server. But therefore you should use a separate password for each account (aka. use a password manager!). Normally you trust your provider to prevent this from happening. But SSO is not offered by every Moodle instance. For example, this is not an option for mine.
Btw. this is only done once to get a token, then not anymore.

Besides it can happen that the admin also logs post data. You can ask your admin about this and recommend censoring post and get data from token-login.

:D

from moodle-dl.

With the mobile/launch.php one is redirected to the SSO page and no credentials are transferred to moodle. So far decoding the token looked like binary data (at least not ASCII). Have a closer look into it tomorrow.

from moodle-dl.

I've just seen that the REST interface of Moodle also accepts post-requests, I'll make the changes so that all requests are sent by POST. This helps at least against basic logging.

from moodle-dl.

But I read something here.
So you could generate a token by calling https://www.yourmoodle.com/m/admin/tool/mobile/launch.php?service=moodle_mobile_app&passport=12345

Worked like a charm. I could now use the token to download everything from Moodle.
Thanks a lot for your help.

@shoeper First login and then change the url accordingly to @C0D3D3V's post

from moodle-dl.

Thank you very much @ndarr for testing it.

All Requests are now send as POST c96dc38.

from moodle-dl.

I also found the issue I ran into yesterday:

On my computer the token is not shown because xdg-open is registered as handler for all protocols. I first tried to pass http as protocol but then all characters will be lowercase and the whole string is urlencoded. Passing http://localhost shows the token in the url bar.

Example url: https://example.com/admin/tool/mobile/launch.php?service=moodle_mobile_app&passport=12345&urlscheme=http://localhost

from moodle-dl.

Using another moodle the urlscheme=http://localhost trick did not work. They seem to user proper sanitization. In that case one can open the developer console, open the network tab and the token will be visible there (as failed request). (And then just run echo -n "TOKEN" | base64 -d and copy the element after the first ::: - as you said).

Also we have multiple moodles. I don't know how common it is but in such a case it would be great when the config could contain multiple tokens and urls and they would be processed one by one.

And another thing is the file naming. At our university some course names are very long because they contain e.g. course numbers. Some courses also use too many folders (like one folder for one file or even folders without any files).

And personally I think about adding a whitelist because usually I only want to have the courses I currently hear and keep the others as they are.

Maybe I'll look into these some day.

Apart from these points (which are personal preferences anyway) the tools works great and it is quite fast.

Dockerfile

FROM python:3

ADD . /md

WORKDIR /md

RUN pip3 install -r requirements.txt

ENTRYPOINT ["python3", "main.py", "--path", "/files"]

.dockerignore

files/
files2/
config.json
Dockerfile
.dockerignore
.git/

build

podman build -t md .
# or
docker build -t md .

can be run like this:

# podman + fish with SELinux
podman run --rm -it -v (pwd)/files/:/files:Z md
# Docker + bash
docker run --rm -it -v $(pwd)/files:/files md
# config should be placed in files folder or run init on first run to create it
# the files end up in the folder `files` within the current directory.

In the next step WebDAV will be added to the Docker image. The files can then directly be uploaded to a WebDAV server and all devices can automatically fetch the files using e.g. Seafile or Nextcloud.

from moodle-dl.

I'm glad it worked out for you, too.

As soon as the exam phase is over, I will automate this so that you can easily add a Moodle account with SSO just like in the APP. But until then this issue is a good guide I think.

That the downloader downloads multiple Moodles is in itself easy to implement. But I'm not sure if it makes more sense to simply run multiple cron jobs. But I'll be happy to implement it, it doesn't hurt.

Yes, filenames are always a problem, I don't have a perfect solution yet. Especially files have two names in Moodle, the real filename I am currently using and the name used to display them on the platform.

Course names are really sometimes very long. Although you can give a short name as a course creator, I have not seen anyone use this feature yet. I could, however, change the implementation and use the short name instead (although this is most likely identical to the long name). And then I could cut the names to a certain length.

Do you mean real folders or sections in which a course is divided. I have not seen empty folders yet, and the downloader should not download them at all.

Yes, I can implement a whitelist already tomorrow. In the end it is the same as the blacklist I already implented.

I would be very happy if you implement some of this and make a Pull Request. Any support is welcome, I don't have to implement everything myself.

Thanks for your appreciation, I am very pleased that the tool is useful.

And many thanks for the Docker guide!

It might be better to split all these different topics into different issues, so that you can find the topics easier in the future...

from moodle-dl.

Do you mean real folders or sections in which a course is divided.

We have quite some courses where files and up in folders named by date or weak number. Such folders are really inconvenient, because they do not say anything about the content and make it more "complex" to find and open the files.

Yes, I can implement a whitelist already tomorrow. In the end it is the same as the blacklist I already implented.

I could also imagine some further options in combination with that. Or rather just as a new option where one can define such things per course id. Specifically:

• configurable course name (instead of taking the one by moodle one sets a preferred name)
• whether to create folder locally or not (if not put all files in one directory)
• select which of the two files names one want to use
• or even further "rewrite rules"

Obviously that would be some additional configuration to perform, but given that it would only have to be done once and saves all the manual file lookups and downloads on moodle (without the tool) it would be worth it.

Regarding pull requests: I'll look into this after my exams.

from moodle-dl.

That's nice. I think there are still cases where manual configuration could be preferred, though. E.g. when running it on a headless machine.

from moodle-dl.

Yes good idea i plan to create a Wiki page for the repo where i can document this. Documenting everything in the readme would be too much.

from moodle-dl.

@shoeper Have you by chance created a ready-/working Dockerfile? I would like to include it in the project or at least offer it in the wiki.

from moodle-dl.

I use this simple one. davfs2 and bash aren't required for most users so the RUN statement could be removed.

Dockerfile

FROM python:3

RUN apt-get update \
  && DEBIAN_FRONTEND=noninteractive apt-get install -y bash davfs2 \
  && mkdir /files \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

ADD . /md

WORKDIR /md

RUN pip3 install -r requirements.txt

ENTRYPOINT ["./docker-entrypoint.sh", "python3", "main.py", "--path", "/files"]

.dockerignore

config.json
Dockerfile
.dockerignore
.git/

# Build
podman build -t md .
# Run in the mounted dir should be the config
# Multiple moodles can be downloaded by running the container for each folder
podman run --rm -it --net=host -v $(pwd):/files:Z md

podman can be replaced with docker command.

from moodle-dl.

Ok perfect, I tested it, apart from the docker-entrypoint.sh it works great. But I think those are just customizations of yours. I will add the dockerfile to the repo and will explain in the wiki how to use it.

from moodle-dl.

I could also setup an automated build but don't have time, currently.

from moodle-dl.

This is the entrypoint. I'm not using it so far, so not sure whether it really works. The idea is to allow placing the files on a WebDAV remote. That way the job can run on a server or also services like Gitlab CI and upload the files to e.g. Nextcloud or Seafile where the clients automatically fetch the new files for all devices.

#!/bin/bash
#
# variables to be passed
# WEBDAV_URL e.g. https://example.com/
# WEBDAV_USER
# WEBDAV_PASSWORD
# 
# set mountpoint if not already set
WEBDAV_MOUNTPOINT="${WEBDAV_MOUNTPOINT:-/files/}"

# fail script on error
set -e

if [ ! -z ${WEBDAV_URL+x} ]; then
    echo "${WEBDAV_MOUNTPOINT} ${WEBDAV_USER} ${WEBDAV_PASSWORD}" > "/etc/davfs2/secrets"
    echo "${WEBDAV_URL} ${WEBDAV_USER} ${WEBDAV_PASSWORD}" >> "/etc/davfs2/secrets"
    chmod 600 "/etc/davfs2/secrets"
    /sbin/mount.davfs "$WEBDAV_URL" "$WEBDAV_MOUNTPOINT" && echo "[`date -d today +%Y-%m-%d\ %_H:%M:%S`] $WEBDAV_URL mounted"
fi

"$@"

if [ ! -z ${WEBDAV_URL+x} ]; then
    /sbin/umount.davfs "$WEBDAV_MOUNTPOINT" && echo "[`date -d today +%Y-%m-%d\ %_H:%M:%S`] $WEBDAV_URL unmounted"
    rm /etc/davfs2/secrets
fi

from moodle-dl.

This is very cool, I will test it and then add it to the wiki. Thanks a lot.

from moodle-dl.