swarthmore / filescan Goto Github PK

Technically plugin settings scoped for the config_plugins table should be named block_filescan/<settingname> rather than just filescan/<settingname>.

To prevent timeouts

After remediated files are added would be nice to have a "Refresh Results" button so users can update the results after they've added new documents or fixed old ones.

Might be nice to give a before and after number for sense of progress/satisfaction - but don't think this is essential

• Convert text image to text, provide user with example of text so they can judge if acceptable
  • would need some method of quality assurance or some method of human review
• provide link to ocr'd document to allow user to replace existing doc
• keep and tag the old doc for period of time to
  a. be able to check original if needed
  b. find the old docs and remove them after time period ends
• allow users to indicate the doc is a legit image during the review process

Alternately, have the fix button trigger email to faculty and/or document remediation team that includes moodle course and file information

"Pass" does not indicate the file is definitely good. Come up with better language to explain the status of a file.

As part of developing the admin report. Not yet available on the master branch.

We could use this to mark which items should be exported for remediation, which are okay, and which have accessible online alternatives

For some reason I'm not able to add a label to this - should be an enhancement

I don't have an example of this but wanted to add it as a ticket as was raised in related conversation

The path is set to block/filescan/db, but should be blocks/filescan/db. This causes installing on at east 3.5.1 to explode. Here is a related MDL-Tracker issue.

Error:

Errors found in XMLDB file: PATH attribute does not match file directory: blocks/filescan/db
Error code: ddlxmlfileerror

line 29: “Unkown” is missing letter "n", should be “Unknown”

Change block title to "PDF Scan"

Scan only looks at PDF files and if we add other types of files it may lead to some confusion

So actually what I suggested in #6 about the permissions was too simple. Defining the capability is still correct, but I think you will also want to set the 'applicable formats.' This is assuming that the filescan doesn't make sense on a "My" page or on a "Site home". I'll post a merge request.

Not sure how best to require Guzzle for all systems. Currently just include files in the vendor directory.

We're contemplating using this block with the Poster module. Right now that requires a tedious workaround:

1. Add a poster module to a course
2. Add the Quickmail block
3. In the block configuration, change the value of "where this block appears" to "any page".
4. Navigate to the poster; the block will appear on the page.
5. Change the configuration again, from "any page" to "Poster module main page".

If mod is added as a supported applicable format we can add the block directly to the poster in the first instance.

It would be helpful to have total pages and total documents by semester for reporting and to document progress

A file is linked to within a course section. Added by editing the topic and uploading into the topic description.

Swat example: https://moodle.swarthmore.edu/course/view.php?id=20001

There are some PDFs that are not getting picked up by the Filescan report.

Looking at this course:

https://moodle.swarthmore.edu/course/view.php?id=19701#section-2

The PDF files linked in the Week 2 and Week 3 sections should be picked up, but are not.

The timemodified field in the files table does not get updated when a file is copied to a new course. Use the id field in the files table to sort instead. Ideally, we want a specific number of unique contenthashes that reflect the most recently used files. The largest files.id value should be the most recently updated file.

FROM:
$query = 'SELECT distinct f.contenthash, f.timemodified FROM {files} f, {context} c WHERE c.id = f.contextid AND c.contextlevel = 70 AND f.filesize <> 0 AND f.mimetype = "application/pdf" AND f.component != "assignfeedback_editpdf" AND f.filearea != "stamps" AND f.contenthash NOT IN (SELECT contenthash FROM {block_filescan_files} where checked=True or (checked=False and status="error" and statuscode >=' . $max_retries . ')) ORDER BY f.timemodified DESC LIMIT ' . $max_files_to_check;

To: (maybe)
$query = 'SELECT f.contenthash, f.id FROM {files} f, {context} c WHERE c.id = f.contextid AND c.contextlevel = 70 AND f.filesize <> 0 AND f.mimetype = "application/pdf" AND f.component != "assignfeedback_editpdf" AND f.filearea != "stamps" AND f.contenthash NOT IN (SELECT contenthash FROM {block_filescan_files} where checked=True or (checked=False and status="error" and statuscode >=' . $max_retries . ')) ORDER BY f.id DESC LIMIT ' . $max_files_to_check;

The file scan summary information in the block is cached so doesn't need to recalculate on every page load. Currently, only clicking on "View File Details" will force a cache refresh.

Ideally the cache would be invalidated each time a PDF file was uploaded to the course, but perhaps it could be invalidated on a time cycle or other event. According to the Moodle Dev documentation, setting a time to live (ttl) is not recommended.

In db/access.php the block_filescan:scan permission is defined this way:

  'block/filescan:scan' => array(
    'riskbitmask' => RISK_PERSONAL,
    'captype' => 'read',
    'contextlevel' => CONTEXT_COURSE,
    'archetypes' => array(
      'guest'           => CAP_PREVENT,
      'student'         => CAP_PREVENT,
      'teacher'         => CAP_PREVENT,
      'coursecreator'   => CAP_ALLOW,
      'editingteacher'  => CAP_ALLOW,
      'manager'         => CAP_ALLOW
    ),
    'clonepermissionsfrom' => 'moodle/course:update',
  ),

The settings in archetypes suggest that guests, students and teachers should be set to CAP_PREVENT for this capability, but if that is the design then it is not happening as planned because the permissions are being cloned from moodle/course:update where the roles in question typically have different or unset permissions. On a brand new installation right after installing block_filescan I find that block_filescan:scan is unset for guests and students and set to CAP_ALLOW for teachers.

The archetypes element and the clonepermissionsfrom element seem to be in conflict. I'm not sure which permissions you want, but I think it would be good to resolve the conflict.

Most 3rd-party vendors charge by the page (as well as complexity). It would be very helpful to have the total number of pages for documents found by the file scan so that we can estimate potential costs.

Total pages by course would also be useful

Add a task that checks to see if the existing results still have a match in the Moodle file table. If not, delete the filescan result.

our standard right now is if any part of the document has text. Ran into unexpected scenario where a document was marked as having text but about 98% of it was images of text with the last paragraph on the last page being text.

In addition to identifying the presence of text, would be nice if we could identify what percentage of the doc has text, or how may words out of the total are in the lang dictionary, or some other way of discerning the quality of the document.

Request to add an additional rich text box configuration option for information to show in the block that students see. If there is content in the rich text box configuration option, it would show up in the block for students. If there box is empty, students wouldn't see the block (same as the current situation).

Develop an admin view to allow site-wide summary statistics as well as searching for all files within specific courses and categories.

This view would be useful for people working to remediate courses.

Possible idea: check to see if the block is being added to a system-wide context and display summary data. Only allow system-level roles such as admins to see the block.

Running Moodle 3.6.4 and PostgreSQL 9.6.13 on Debian 9.9, when I put the files into blocks/ and click "Upgrade Moodle database now" in the web UI, I get an error that blocks installation:

DDL sql execution error

More information about this error

Debug info: ERROR: column "timechecked" is of type timestamp without time zone but default expression is of type integer
HINT: You will need to rewrite or cast the expression.
BEGIN ISOLATION LEVEL SERIALIZABLE;
CREATE TABLE mdl_block_filescan_files (
id BIGSERIAL,
contenthash VARCHAR(40) NOT NULL DEFAULT '',
checked SMALLINT NOT NULL,
pagecount BIGINT,
status VARCHAR(20),
statuscode BIGINT DEFAULT 0,
hastext SMALLINT,
hastitle SMALLINT,
hasoutline SMALLINT,
haslanguage SMALLINT,
timechecked TIMESTAMP NOT NULL DEFAULT 0,
courseinfo TEXT,
CONSTRAINT mdl_blocfilefile_id_pk PRIMARY KEY (id)
)
;
COMMENT ON TABLE mdl_block_filescan_files IS 'Filescan results'
; COMMIT
Error code: ddlexecuteerror

Stack trace:

• line 492 of /lib/dml/moodle_database.php: ddl_change_structure_exception thrown
• line 248 of /lib/dml/pgsql_native_moodle_database.php: call to moodle_database->query_end()
• line 676 of /lib/dml/pgsql_native_moodle_database.php: call to pgsql_native_moodle_database->query_end()
• line 77 of /lib/ddl/database_manager.php: call to pgsql_native_moodle_database->change_database_structure()
• line 425 of /lib/ddl/database_manager.php: call to database_manager->execute_sql_arr()
• line 370 of /lib/ddl/database_manager.php: call to database_manager->install_from_xmldb_structure()
• line 969 of /lib/upgradelib.php: call to database_manager->install_from_xmldb_file()
• line 520 of /lib/upgradelib.php: call to upgrade_plugins_blocks()
• line 1852 of /lib/upgradelib.php: call to upgrade_plugins()
• line 694 of /admin/index.php: call to upgrade_noncore()

Best guess based on the error message is that PSQL won’t take integer 0 as its zero-value timestamp, so now I’m going to poke around and see if I can find an alternative.

The word "unknown" is misspelled in the Moodle block.

I'm running into this exception when I try to install the current master:

!!! Exception - syntax error, unexpected '"', expecting '-' or identifier (T_STRING) or variable (T_VARIABLE) or number (T_NUM_STRING) !!!
!! 
Error code: generalexceptionmessage !!
!! Stack trace: * line 209 of /blocks/filescan/classes/task/scan_files.php: ParseError thrown
* line ? of unknownfile: call to core_component::classloader()
* line ? of unknownfile: call to spl_autoload_call()
* line 290 of /lib/classes/task/manager.php: call to class_exists()
* line 68 of /lib/classes/task/manager.php: call to core\task\manager::scheduled_task_from_record()
* line 88 of /lib/classes/task/manager.php: call to core\task\manager::load_default_scheduled_tasks_for_component()
* line 989 of /lib/upgradelib.php: call to core\task\manager::reset_scheduled_tasks_for_component()
* line 520 of /lib/upgradelib.php: call to upgrade_plugins_blocks()
* line 1852 of /lib/upgradelib.php: call to upgrade_plugins()
* line 491 of /lib/installlib.php: call to upgrade_noncore()
* line 461 of /lib/phpunit/classes/util.php: call to install_cli_database()
* line 150 of /admin/tool/phpunit/cli/util.php: call to phpunit_util::install_site()
 !!

I've had this problem on a number of installations and it also causes PHPUnit tests to fail. The cause of the error was not immediately apparent to me, so I'm just going to hold off upgrading for now.

At the top of the details page, the sum of each file status type is shown. The unknown or error value is not shown.

export that maintains weeks/topics would be great

I just installed on a fresh site. When I go to the summary report in the admin block I get warnings like this:

Warning: Division by zero in <path/to/moodle>/blocks/filescan/admin.php on line 155

Greatly speeds up database queries used to generate content for the block.

Need to update the db/install.xml file using <INDEX>
https://docs.moodle.org/dev/XMLDB_defining_an_XML_structure#The_INDEX_element
and db/upgrade.php file with $dbman->add_index https://docs.moodle.org/dev/Data_definition_API#Handling_indexes

With debugging on and the block installed I see the following on the dashboard:

The block filescan does not define the standard capability block/filescan:myaddinstance

    line 595 of /blocks/moodleblock.class.php: call to debugging()
    line 567 of /blocks/moodleblock.class.php: call to block_base->has_add_block_capability()
    line 228 of /lib/blocklib.php: call to block_base->user_can_addto()
    line 2213 of /lib/blocklib.php: call to block_manager->get_addable_blocks()
    line 1236 of /lib/blocklib.php: call to block_add_block_ui()
    line 538 of /lib/outputrenderers.php: call to block_manager->ensure_content_created()
    line 39 of /theme/bootstrapbase/renderers/core_renderer.php: call to core_renderer->standard_head_html()
    line 56 of /theme/hampshire/layout/columns2.php: call to theme_bootstrapbase_core_renderer->standard_head_html()
    line 1110 of /lib/outputrenderers.php: call to include()
    line 1040 of /lib/outputrenderers.php: call to core_renderer->render_page_layout()
    line 164 of /my/index.php: call to core_renderer->header()

Maybe this is related to #5? At any rate, I'm using Moodle 3.2 with a Clean-based theme. When I install the plugin the 'Site administration' menu in the administration block stops working. When I remove the plugin, that menu starts working again. Because it is large the menu loads only when called for. I can see the loading spinner, but then it comes back empty. If I go to /admin manually everything works fine.

If document is being used in multiple courses, develop mechanism to notify faculty in other courses that an accessible version of a document is available and give them easy way to replace the inaccessible doc

To see the problem visit the "capabilities overview" page with debugging on:

Invalid get_string() identifier: 'filescan:myaddinstance' or component 'block_filescan'. Perhaps you are missing $string['filescan:myaddinstance'] = ''; in /Users/kwiliarty/Sites/moodles/hc32/blocks/filescan/lang/en/block_filescan.php?

    line 349 of /lib/classes/string_manager_standard.php: call to debugging()
    line 7086 of /lib/moodlelib.php: call to core_string_manager_standard->get_string()
    line 2693 of /lib/accesslib.php: call to get_string()
    line 44 of /admin/tool/capability/index.php: call to get_capability_string()

Invalid get_string() identifier: 'filescan:managepages' or component 'block_filescan'. Perhaps you are missing $string['filescan:managepages'] = ''; in /Users/kwiliarty/Sites/moodles/hc32/blocks/filescan/lang/en/block_filescan.php?

    line 349 of /lib/classes/string_manager_standard.php: call to debugging()
    line 7086 of /lib/moodlelib.php: call to core_string_manager_standard->get_string()
    line 2693 of /lib/accesslib.php: call to get_string()
    line 44 of /admin/tool/capability/index.php: call to get_capability_string()

Invalid get_string() identifier: 'filescan:scan' or component 'block_filescan'. Perhaps you are missing $string['filescan:scan'] = ''; in /Users/kwiliarty/Sites/moodles/hc32/blocks/filescan/lang/en/block_filescan.php?

    line 349 of /lib/classes/string_manager_standard.php: call to debugging()
    line 7086 of /lib/moodlelib.php: call to core_string_manager_standard->get_string()
    line 2693 of /lib/accesslib.php: call to get_string()
    line 44 of /admin/tool/capability/index.php: call to get_capability_string()

Invalid get_string() identifier: 'filescan:viewpages' or component 'block_filescan'. Perhaps you are missing $string['filescan:viewpages'] = ''; in /Users/kwiliarty/Sites/moodles/hc32/blocks/filescan/lang/en/block_filescan.php?

    line 349 of /lib/classes/string_manager_standard.php: call to debugging()
    line 7086 of /lib/moodlelib.php: call to core_string_manager_standard->get_string()
    line 2693 of /lib/accesslib.php: call to get_string()
    line 44 of /admin/tool/capability/index.php: call to get_capability_string()

Invalid get_string() identifier: 'filescan:addinstance' or component 'block_filescan'. Perhaps you are missing $string['filescan:addinstance'] = ''; in /Users/kwiliarty/Sites/moodles/hc32/blocks/filescan/lang/en/block_filescan.php?

    line 349 of /lib/classes/string_manager_standard.php: call to debugging()
    line 7086 of /lib/moodlelib.php: call to core_string_manager_standard->get_string()
    line 2693 of /lib/accesslib.php: call to get_string()
    line 44 of /admin/tool/capability/index.php: call to get_capability_string()

I haven't seen anything like this with MySQL/MariaDB but when I try to install a PostgresQL database I'm getting this error:

block_filescan

DDL sql execution error

More information about this error

It is usually not possible to recover from errors triggered during installation, you may need to create a new database or use a different database prefix if you want to retry the installation.
Debug info: ERROR: column "timechecked" is of type timestamp without time zone but default expression is of type integer
HINT: You will need to rewrite or cast the expression.
BEGIN ISOLATION LEVEL SERIALIZABLE;
CREATE TABLE m_block_filescan_files (
id BIGSERIAL,
contenthash VARCHAR(40) NOT NULL DEFAULT '',
checked SMALLINT NOT NULL,
pagecount BIGINT,
status VARCHAR(20),
statuscode BIGINT DEFAULT 0,
hastext SMALLINT,
hastitle SMALLINT,
hasoutline SMALLINT,
haslanguage SMALLINT,
timechecked TIMESTAMP NOT NULL DEFAULT 0,
courseinfo TEXT,
CONSTRAINT m_blocfilefile_id_pk PRIMARY KEY (id)
)
;
COMMENT ON TABLE m_block_filescan_files IS 'Filescan results'
; COMMIT
Error code: ddlexecuteerror
Stack trace:

line 492 of /lib/dml/moodle_database.php: ddl_change_structure_exception thrown
line 248 of /lib/dml/pgsql_native_moodle_database.php: call to moodle_database->query_end()
line 676 of /lib/dml/pgsql_native_moodle_database.php: call to pgsql_native_moodle_database->query_end()
line 77 of /lib/ddl/database_manager.php: call to pgsql_native_moodle_database->change_database_structure()
line 425 of /lib/ddl/database_manager.php: call to database_manager->execute_sql_arr()
line 370 of /lib/ddl/database_manager.php: call to database_manager->install_from_xmldb_structure()
line 969 of /lib/upgradelib.php: call to database_manager->install_from_xmldb_file()
line 520 of /lib/upgradelib.php: call to upgrade_plugins_blocks()
line 1852 of /lib/upgradelib.php: call to upgrade_plugins()
line 694 of /admin/index.php: call to upgrade_noncore()

identify which videos do not have captions
identify which audio does not have transcripts