berntpopp / screen-scout Goto Github PK

currently the README is empty. Describe this script in detail.

Concurrency control for how many pages are processed at the same time could improve performance, especially when working with a large number of pages.

Consider including concurrent page processing, rate limiting, or request-to-server delay mechanisms in the script in order to control load and adhere to server limitations if you intend to scale it.

relates to #2

The crawler does not wait fpor all responses and sometimes takes screenshots of pages still loading.

For example:

Solution:

• set a parameter to handle max wait time
• check if pupateer can wait for whole page to laod

There should be a possibility to provide login credentials to browse and generate screenshots of pages with authentication.

More detailed logging, such as the current depth level and number of pages processed, can aid in progress tracking and debugging.

Making feedback available in the command line for each step (e.g., "Launching browser", "Processing URL:...", "Closing browser") would improve the script's usability.

Instead of input from the command line allow input through a list of URLs.
Specify if these should be crawled through their links or not.

The screenshots could be processed by ChatGPT to generate descriptions of the website.
This implementation will require:

• a config file
• likely a second, independent script
• tailored API requests to produce for example markdown formatted files

Research if it is possible to take screenshots of elements like drop-down menus or modals.

Each URL currently launches a new browser instance. This can be time-consuming, especially with deep recursion or a large number of pages. Use the same browser instance for all pages if possible. This may necessitate reorganizing how browser and page instances are managed.
Ensure that resources are released properly. Close the page after taking a screenshot, but keep the browser open if you are going to reuse it.

Add command-line argument validation, such as checking to see if the provided output directory exists (and possibly creating it if it doesn't).
Validate the page response and log any errors that occur.

It might be useful to add a setting to configure the timeout for page loading, as some pages may take longer to load than others.

For example the output "sysndd.dbmr.unibe.ch-Genes-HGNC".
This seems to be an issue with certain characters like ":" in the URL when they are parsed with regex.

Add semantic versioning to the CLI output including last commit.

For reproducibility it would be helpful to implement WayBack machine snapshots using their API.
This will need a config file holding the user's API key.

The screenshot file naming should be robust with long urls including parameters and hashes.
A possible solution is to make a meta file that holds the full urls and the file names with shortened und unique names.
Carefully implement to follow filename conventions for different operating systems.

This could be useful to test page load and potential layout shifts.
Allow users to set the interval in milliseconds.
Name the files accordingly with the interval time.

For some use cases it would be great to add a SVG or PNG overlay to the screenshots, including watermarks or highlights.

Investigate possible solutions.

After all browser tabs/pages have been processed, ensure that they are properly closed. This aids in the efficient management of resources.