exponential-decay / demystify Goto Github PK

List all extension mismatches found in a file.

Figure out a way to do this nicely.

Workarounds are currently happening in the tests folder... for example.

Based on sscompare work this will be useful. It makes the CSV input more flexible.

Additionally, revisit OPF blog to correct language usage there.

Sorting results based on the issues that put a collection/ingest at most risk, first, might be a more useful output than a mixture of summary and risk based information.

An example might be seeing unidentified files and extension only identifications output before frequency charts.

Similarly, identical content could be output immediately after those two as well.

Initial list with biggest priority:

• Unidentified list
• Total files in container objects
• Extension only identifications
• Extension mismatches
• Multiple identifications
• Duplicate file names
• Duplicate content

e.g. Z:\transfers-working\18-Jan-2016\dispositions\disp_20160115143947_334133\Docs_WIN1648_108_12\88800.1.doc'

MAX_PATH is 255 characters in NTFS. If we see a path longer, raise it as a digital preservation risk in the script.

https://msdn.microsoft.com/en-us/library/aa365247.aspx#maxpath

A report that can produce various statistics relating to file-size.

Note: Could be configured to produce a list larger than top five.
Note: May not always be useful, but might help users to understand their collections better.
Note: Configure via external config file.

Total files Files inside containersArchive Objects:
The number of files inside a container archival objects

This would enable users to better use the empty-folders script in the exponential-decay repositories.

• https://github.com/exponential-decay/detect-empty-folders

By popular opinion!

Poor IF logic currently used, line 150 droid2sqlite.py onwards.

Because a split ingest could be wildly impacted by this feature, and would require quite a bit of effort.

Used to be able to output DROID CSV again from SQLITE DB... do we re-add this functionality?

One idea might be to create a dump per namespace? https://github.com/exponential-decay/droid-sqlite-analysis/blob/master/libs/ExportDBClass.py

Currently for each character issue discovered we do something a bit verbose:

def reportIssue(self, s, msg, value=''):
    self.report = self.report + "File: " + s + " " + msg + " " + value + "\n"

E.g.

self.reportIssue(s, "contains, non-printable character:", hex(c) + ", " + self.unicodename(c))

We might be able to create a list of value pairs, e.g. (enum, char), e.g. ('non-ascii-char', '{char}) we can then process this more intelligently at the end of the analysis and perhaps in different ways too.

Mechanism to blacklist certain PUIDs to place them into a separate listing in some useful manner.

Examples might be fmt/111 - OLE2 Compound Object Format, or x-fmt/411 - Windows Portable Executable.

These are files identified by DROID but are likely to require action before ingest, e.g. the more specific identification of fmt/111 as a child format, e.g. Microsoft Word, San Serif Page Plus.

This will be institution specific.

Note: Likely to be configured by external config file for usability.

This should be pretty easy to return, and might be quite nice for users.

E.g. https://en.wikipedia.org/wiki/PRONOM#The_PRONOM_Persistent_Unique_Identifier_.28PUID.29_scheme

Removing top IDs for now because it does not make sense in a multi-identifier environment.

Use a blacklist of file extensions to generate special listings. E.g. for the recognition of Thumbs.db files, .tmp files, .bat files etc.

Note: May keep listings separate depending on how this works.
Note: Likely to refer to an external config file.

Sort the identified PUID listing by format name to make it more useful to spot trends. There is no relationship between PUID numbers and so a sort is useless. PDF 1.1 and PDF 1.2 could be separated by 100 other PUIDs. As such, sort by format name to highlight trends across a collection.

Find a test case.

e.g. for creating a distilled set of objects during transfer. (the opposite being a distilled set of objects for destruction/passing back to an agency to make a decision on.)

As a result of making the code more generic we may need more pointers as to the hash mechanism used in the code. Output this as a value in the report.

To provide more robust input and output.

ref: http://ndsr.nycdigital.org/trojan-dots-and-diy-solutions/

C:\Working\git\droid-siegfried-sqlite-analysis-engine>python droidsqliteanalysis.py --export sf-working-copies.json > working-copies.htm
Traceback (most recent call last):
  File "droidsqliteanalysis.py", line 129, in 
    main()
  File "droidsqliteanalysis.py", line 112, in main
    handleDROIDCSV(args.export, True, args.txt, blacklist, args.rogues, args.heroes)
  File "droidsqliteanalysis.py", line 73, in handleDROIDCSV
    dbfilename = droid2sqlite.identifyinput(droidcsv)
  File "C:\Working\git\droid-siegfried-sqlite-analysis-engine\droid2sqlite.py", line 22, in identifyinput
    return handleSFYAML(export)
  File "C:\Working\git\droid-siegfried-sqlite-analysis-engine\droid2sqlite.py", line 41, in handleSFYAML
    loader.sfDBSetup(sfexport, basedb.getcursor())
  File "C:\Working\git\droid-siegfried-sqlite-analysis-engine\libs\SFLoaderClass.py", line 152, in sfDBSetup
    cursor.execute(i)
sqlite3.OperationalError: near "s": syntax error

E.g. with Poetry https://hackersandslackers.com/python-poetry-package-manager/

Hi,

I just tried to run the analysis on a Siegfried CSV export file, and I got the following error message:

akb@debian:~$ python git-repos/droid-siegfried-sqlite-analysis-engine-master/droidsqliteanalysis.py --export ResurectionMen.csv
Traceback (most recent call last):
File "git-repos/droid-siegfried-sqlite-analysis-engine-master/droidsqliteanalysis.py", line 10, in
from libs.DroidAnalysisClass import DROIDAnalysis
File "/home/akb/git-repos/droid-siegfried-sqlite-analysis-engine-master/libs/DroidAnalysisClass.py", line 7, in
import MsoftFnameAnalysis
File "/home/akb/git-repos/droid-siegfried-sqlite-analysis-engine-master/libs/MsoftFnameAnalysis.py", line 8, in
from internationalstrings import AnalysisStringsEN as IN_EN
ImportError: No module named internationalstrings

It looks to me like the MsoftFnameAnalysis.py module is trying to import a module that doesn't exist in the repo?

Thank you!

E.G. For stat; Frequency of Extension Only Identification - We don't filter on distinct file ID and so a result of 22 extension only, might present as:

ns:pronom 22
ns:freedesktop 22
ns:tika 22

Related to #46 if folks are to engage, folks need to do so knowing there is a code of conduct to help ensure everyone is looked after.

For SF with any number of namespaces - always allow output of a simplified PRONOM only mode.

Make README international.

Hi Ross,

Loving the new version of the analysis tool, but I seem to have hit a snag in trying it out. I tried to run droid2sqlite.py against a YAML export from Siegfried 1.5, using both PRONOM and tika namespaces. It's unsuccessful, seemingly due to the parsing of years in the SFHandlerClass.

STDERR is copied below:

mfmmessier:droid-sqlite-analysis-0.4.0 twalsh$ python droid2sqlite.py --export kolmactest.yaml
Traceback (most recent call last):
  File "droid2sqlite.py", line 79, in <module>
    main()
  File "droid2sqlite.py", line 72, in main
    identifyinput(args.export)
  File "droid2sqlite.py", line 21, in identifyinput
    return handleSFYAML(export)
  File "droid2sqlite.py", line 40, in handleSFYAML
    loader.sfDBSetup(sfexport, basedb.getcursor())
  File "/Users/twalsh/droid-sqlite-analysis-0.4.0/libs/SFLoaderClass.py", line 108, in sfDBSetup
    sf.addYear(sfdata)
  File "/Users/twalsh/droid-sqlite-analysis-0.4.0/libs/SFHandlerClass.py", line 284, in addYear
    row[self.FIELDYEAR] = self.getYear(year)
  File "/Users/twalsh/droid-sqlite-analysis-0.4.0/libs/SFHandlerClass.py", line 290, in getYear
    dt = datetime.datetime.strptime(datestring.split('+', 1)[0], '%Y-%m-%dT%H:%M:%S')
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_strptime.py", line 328, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: -04:00

And the relevant lines from _strptime.py:

if len(data_string) != found.end():
        raise ValueError("unconverted data remains: %s" %
                          data_string[found.end():])

It appears that the issue may be in the time zone parsing but I wasn't able to figure out exactly what in the limited time I had to tinker with getYear in SFHandlerClass. Any ideas? (I'm running Python 2.7.10, if that makes any difference)

Thanks!

Because it might be considered unwieldy right now...

Via @joshuatj, inspured by @anjackson and his work on ident-o-matic

This is likely an epic, so epic related tasks will appear below:

• Renaming
• Python 3
• Molularizing (pathlesstaken, sqlitfid, etc.)

Tasks on this are likely to begin early July.

• Google Doc to help organize things...

Remove hard coded indexes from the script. Proposal: read CSV header, index. Hard code the column names we are interested in, but not their index. We are more interested in the content than the index of the header. Hypothesis is that the index is more likely to change than the column name.

Two column indexes that are currently hard coded are:

URI_COL = 2
PATH_COL = 3

Via: https://twitter.com/euanc/status/793108956021489664

In any given context, a large number of Unicode characters is very likely. It may be useful to understand groupings, or themes, but not necessarily all. I'm considering a filter for certain characters so that they don't explore the report, but I am also considering other ways of presenting this information, e.g. organize by category and providing a collapsable file-list underneath.

Connected to #49

These variables need translating to other languages! We've started with French and Czech, but need to keep going :D

https://docs.google.com/spreadsheets/d/1dVsRsXgD9V2GarNHHpf6Tzhrfx99_MXt3LjSSDrNLOY/edit#gid=0

Contents section to link to specific anchor sections of the document so it is easier to browse.

Because right now only rogues can be used to access the unidentified listing...

Since creation so need to have a look at how to better do these counts.

https://pythonclock.org/

This app shouldn't use any external dependencies. It was designed to work in an environment that doesn't have the ability to use PIP.

Lxml.etree should be replaced by etree pretty easily. Lxml.html, I am not sure about.

Black-list of container file types that we can identify DROID won't analyse. There will need to be further analysis done on these files in any collection. E.g. RAR.

Have removed getContentHash() and getTimeStamp()

Original idea was to prevent a DB from generating again, but also to try and begin generating cumulative results per improved signature release to understand the evolution of a collection.

Will an add function for the DB prove useful? - Namespace could include a timedatestamp also.

See stat in HTML: Archive File Types in Accession/Extract

They're redundant but do not interfere with results. Probably slows things down.