felixauringer / moodle-course-downloader Goto Github PK

It would be nice to be able to configure the crawler to crawl multiple courses / multiple moodle instances. Instead of command line arguments this would be easier using actual configuration files.

Most moodle courses are only visible when logged in. Therefore, the crawler needs a valid session to do its work. I would think setting a single cookie should work. The cookie should not be visible in the cli history.

The crawler should do the following things:

• save which pages were already fetched
• keep a queue of resources to be fetched
• include the configuration and http client (less global state)

Go is nice for concurrency. That should be used.

Not all code should be in main.go

• parse a moodle page and remove all the unnecessary html (header, footer, ...)
• collect all links

• deduce unique filename from resource url
• save data in the right format based on the content type
• check that webpage can be opened locally afterwards
• make the path where the files are saved configurable

Right now, I give most errors to log.Fatal. That is not very error resilient and causes the application to crash on every error. Some errors can probably just be logged or handled somehow else and do not need the application to crash.

In order to learn the basics, I do not want to use some fancy framework but just the net/http library