Same as with atc0005/elbow#44

See atc0005/todo#9. Pending completion of #59

I've gotten distracted by other projects and haven't looped back and released the latest changes in master as a new release. Need to refresh where I left off and get a new release cut.

See also #24.

As with the file size threshold, the duplicate file count threshold (DuplicatesThreshold) is a constant defining how many 1:1 files are required in order to be considered duplicates. As of this writing this number is hard-coded to 2, but there may be use cases where the number would be specified higher to filter out a small number of matches.

Note: This may be better handled by adding a new flag that acts as a filter, leaving the existing DuplicatesThreshold value as-is.

See also atc0005/todo#6.

Nearly done with the prototype per my notes in #1. The more I work with this codebase the more I consider using logrus for it's leveled logging support.

Perhaps it is worth trying one of the other popular third-party logging libraries?

While refreshing my memory where I left off, I noticed that the examples were completely lacking the csvfile flag.

Example:

./bridge.exe -recurse -path "/tmp/path1"

https://github.com/kisielk/errcheck

Note: golangci-lint bundles this package and makes the functionality available via a flag option, so it might not be worth implementing this separately long-term. It may be worth enabling at least initially in order to compare the results.

refs:

• atc0005/send2teams#13
• #47
• atc0005/check-mail#19
• atc0005/elbow#218

see atc0005/todo#8

See atc0005/todo#4

From #1:

A later revision could recursively perform this task and move duplicates into a subfolder alongside the image? What if the images (or files) are in entirely different folders? Perhaps create a log instead? Or, an option to choose which of those steps are performed?

The initial v0.1 release will focus on creating a CSV file for manual review, but it would be useful to perform some sort of cleanup option either automatically or based on a column entry for the CSV file.

For example, the generated CSV file could create a column with prefilled "keep" action entries that the user could replace with delete. This could involve adding new flags:

• simulate
• prune

see atc0005/todo#6

As of b6fb543 not specifying the desired Excel file results in a barely deciperable error message:

$ ./bridge.exe -recurse -path . -csvfile "pics.csv"
2020/02/08 18:55:54 Configuration: {Paths:[.] RecursiveSearch:true ConsoleReport:false IgnoreErrors:false FileSizeThreshold:1 FileDuplicatesThreshold:2 CSVFile:pics.csv ExcelFile:}
2020/02/08 18:55:54 Path exists: .
2020/02/08 18:56:00 105 evaluated files in specified paths
2020/02/08 18:56:00 14 potential duplicate file sets found using file size
2020/02/08 18:56:00 14 confirmed duplicate file sets found using file hash
2020/02/08 18:56:00 28 files with identical file size
2020/02/08 18:56:00 28 files with identical file hash
2020/02/08 18:56:00 14 duplicate files
2020/02/08 18:56:00 188.1 MiB wasted space for duplicate file sets
2020/02/08 18:56:00 Successfully created CSV file: "pics.csv"
2020/02/08 18:56:00 open : The system cannot find the file specified.

Seeing the empty ExcelFile config value provided a clue, but glancing at the code this is what I see:

	// Generate Excel workbook for review
	// TODO: Implement better error handling
	if err := fileChecksumIndex.WriteFileMatchesWorkbook(config.ExcelFile, duplicateFiles); err != nil {
		log.Fatal(err)
	}
	log.Printf("Successfully created workbook file: %q", config.ExcelFile)

Evidently I didn't finish adding guards around the Excel file generation step to skip creation if the user didn't specify a value.

This application groups detected duplicate files into "sets". Each set contains all files that are duplicates of each other, including what could be considered the "original" file that is duplicated by others in the set.

The current implementation for file removal support (see #4) will honor a user-supplied "flag" in in the input CSV file to indicate that the file should be removed, even if the result is removing ALL files in a set. This issue is to extend the file removal support to accept a CLI-flag option to help prevent removal of all files in a set. Instead, when the application can determine that all files are marked for removal that set would be skipped (if config.IgnoreErrors is set perhaps?) or cause the application to immediately fail.

refs #4

While reading over Fun with Flags I learned that the stdlib flag package supports displaying custom help text as a header or lead-in to the auto-generated list of options available to the user. This can be used to display branding, version details and a link back to the main project repo.

For example, here is a snippet from a v0.4.2 generated CSV file:

directory,file,size,size_in_bytes,checksum,remove_file
C:\Users\adam\Desktop\upload_me\Camera,0825161607a.jpg,7.8 MiB,8199899,1da9b7e355d64f57acf5854e746fff01bb5ab0291bcbad857e63607fcf8d03c7,
C:\Users\adam\Desktop\upload_me\Camera,IMG_20160825_160705412.jpg,7.8 MiB,8199899,1da9b7e355d64f57acf5854e746fff01bb5ab0291bcbad857e63607fcf8d03c7,

If for example we wanted to automatically remove any duplicate that contains a IMG_ prefix, we could use a flag like --mark-dupe-prefix (not a fan of the name) like so:

--mark-dupe-prefix 'IMG_'

This would mark the remove_file column for any duplicate files that have that prefix.

Lines:

Options under review

• Task

  • https://taskfile.dev/#/installation
  • https://github.com/go-task/task

• taskctl

  • https://github.com/taskctl/taskctl
  • https://github.com/taskctl/taskctl/blob/master/docs/example.yaml

• Tusk

  • https://github.com/rliebz/tusk
  • https://github.com/rliebz/tusk/blob/master/example/example.yml
  • https://github.com/rliebz/tusk/blob/master/example/github-scripts.yml
  • https://github.com/rliebz/tusk/blob/master/tusk.yml

Goals

• Ease of extension, maintenance
• Readability for anyone else reviewing this project's build process

Same as with atc0005/elbow#46

With the release of Go v1.14 today, v1.12 is no longer supported. Need to update the GitHub Actions Workflow to use 1.13 and 1.14 instead of v1.12 and v1.13.

While testing on a Windows system from a non-privileged account, the application was unable to descend recursively (as requested) into $RECYCLE.BIN and System Volume Information and immediately aborted.

After digging further, I found at least three places in the code where I was failing to implement support for ignoring the error so that the application could continue as requested:

• bridge/paths.go

  Lines 55 to 61 in 611391d

  	// If an error is received, return it. If we return a non-nil error, this
  	// will stop the filepath.Walk() function from continuing to walk the
  	// path, and your main function will immediately move to the next line.
  	if err != nil {
  	return err
  	}

• bridge/matches.go

  Lines 167 to 170 in 611391d

  	fm[index].Checksum, err = GenerateCheckSum(file.FullPath)
  	if err != nil {
  	return err
  	}

• bridge/main.go

  Lines 43 to 46 in 611391d

  	fileSizeIndex, err := ProcessPath(config.RecursiveSearch, path)
  	if err != nil {
  	log.Fatal(err)
  	}

There may be additional locations that I've missed.

Currently we are both deferring file closure and returning the results of file closure, but we are not explicitly syncing before doing so.

Explicitly syncing and using the error return value as our error return value should hopefully better expose write issues?

• https://golangcode.com/writing-to-file/
• https://www.joeshaw.org/dont-defer-close-on-writable-files/
• https://golang.org/pkg/os/#File.Sync

Currently the threshold is set via a constant in order to weed out 0 byte files, but allowing the value to be specified by command-line would allow the user to focus just on files large enough for them to care about.

Perhaps later (if desired) support could be added to specify both lower and upper threshold values.

I "borrowed" the Makefile from this project for a new project I'm working on. While updating the Makefile to reflect the new project I noticed that the version string is not being set as expected.

While useful, the current status badges link out to the SVG used to provide the current status, not the workflow results themselves.

Update all badges to use a syntax similar to 'GoDoc' or 'Latest Release' so that the badge displays, but also functions as a "GoTo" button as well.

Swap out the GitHub Actions used for CI work with custom Docker containers from the atc0005/go-ci project.

refs atc0005/todo#22

See also atc0005/elbow#234.

Verbatim from atc0005/send2teams#41:

As I've begun doing with my other projects, I'd like to replace content like this:

linting:
	@echo "Calling wrapper script: $(LINTINGCMD)"
	@$(LINTINGCMD)
	@echo "Finished running linting checks"

with this:

.PHONY: linting
## linting: runs common linting checks
# https://stackoverflow.com/a/42510278/903870
linting:
	@echo "Running linting tools ..."

	@echo "Running gofmt ..."

	@test -z $(shell gofmt -l -e .) || (echo "WARNING: gofmt linting errors found" \
		&& gofmt -l -e -d . \
		&& exit 1 )

	@echo "Running go vet ..."
	@go vet ./...

	@echo "Running golint ..."
	@golint -set_exit_status ./...

	@echo "Running golangci-lint ..."
	@golangci-lint run \
		-E goimports \
		-E gosec \
		-E stylecheck \
		-E goconst \
		-E depguard \
		-E prealloc \
		-E misspell \
		-E maligned \
		-E dupl \
		-E unconvert \
		-E golint \
		-E gocritic

	@echo "Running staticcheck ..."
	@staticcheck ./...

	@echo "Finished running linting checks"

It's more verbose (though the golantci-lint options will be moved to an external config file before long), but hopefully more consolidated and clearer what is going on.

As noted on atc0005/todo#3:

1. go mod tidy
   • correct as necessary
2. go mod vendor
3. git add vendor
4. git commit -m "go mod tidy, go mod vendor"

• examples

  • Windows
  • Linux (WSL)

• license

Prototype: md5sum * | sort -k1

This sorts by the md5sum and places the duplicates next to each other. In a real setup, the app can move the duplicates to another folder or delete them outright.

See also atc0005/todo#5

From https://github.com/karrick/godirwalk:

godirwalk is a library for traversing a directory tree on a file system.

In short, why do I use this library?

• It's faster than filepath.Walk.
• It's more correct on Windows than filepath.Walk.
• It's more easy to use than filepath.Walk.
• It's more flexible than filepath.Walk.

The initial concept is to save the crawl status for later file hash comparison in order to confirm duplicates. Depending on how this is implemented, this could also provide a useful "audit" mechanism for discovery purposes.

If there are permission issues, it would probably be good to know this before you wait for the app to range over what could be well over an hour for a large collection of content to learn there is a problem.

Perhaps just something simple like an open/close on a temp file in the specified location?

Same here as with our other Go-based projects:

• Docs linting
• Code linting, building and testing

Although we don't have any tests just yet we can go ahead and setup the framework for that work.

refs atc0005/todo#1

Found this package (haven't tested yet): https://github.com/360EntSecGroup-Skylar/excelize

Idea:

1. create one workbook
2. each set of duplicate files is added to a new sheet in the workbook

Not sure how easy this would be to work in alongside the idea on #4 to support file removal based on a keep or delete column value in a flat CSV structure. I expect that (at least initially) the two output files would be separate and a modified copy of the CSV file would be read back in for further action.

See atc0005/todo#20 for details.

This is primarily intended for confirming the version of Go used, but it could also be beneficial to echo out the versions of the various linting tools used as well before running linting checks.

$ go list -m -versions github.com/360EntSecGroup-Skylar/excelize
github.com/360EntSecGroup-Skylar/excelize v1.1.0 v1.2.0 v1.3.0 v1.4.0 v1.4.1

$ go list -m -versions github.com/360EntSecGroup-Skylar/excelize/v2
go: finding github.com/360EntSecGroup-Skylar/excelize/v2 v2.1.0
github.com/360EntSecGroup-Skylar/excelize/v2 v2.0.0 v2.0.1 v2.0.2 v2.1.0

The import port path in the go.mod file currently (and unintentionally) locks our use to the v1.4.1 release of the 360EntSecGroup-Skylar/excelize package. We'll need to explicitly request the v2 series in order to pull in those updates.

From a CI run for #74:

##[error]matches/matches.go:595:16: Error return value of `f.SetCellValue` is not checked (errcheck)
	f.SetCellValue(summarySheet, "A1", "Evaluated Files")
	              ^
##[error]matches/matches.go:596:16: Error return value of `f.SetCellValue` is not checked (errcheck)
	f.SetCellValue(summarySheet, "A2", "Sets of files with identical size")
	              ^
##[error]matches/matches.go:597:16: Error return value of `f.SetCellValue` is not checked (errcheck)
	f.SetCellValue(summarySheet, "A3", "Sets of files with identical fingerprint")

Existing problem not related to #74, exposed (evidently) due to the work on #70 or #69.

Thankfully all are minor; some are from me being new to Go and some are from quickly hacking this together without following common design patterns.

$ make linting
Calling wrapper script: bash testing/run_linting_checks.sh
matches.go:105:6: `InList` is unused (deadcode)
func InList(needle string, haystack []string) bool {
     ^
units.go:13:6: `ByteCountSI` is unused (deadcode)
func ByteCountSI(b int64) string {
     ^
matches.go:275:6: sloppyLen: len(fileMatches) <= 0 can be len(fileMatches) == 0 (gocritic)
                if len(fileMatches) <= 0 {
                   ^
matches.go:314:9: Error return value of `w.Write` is not checked (errcheck)
        w.Write(csvHeader)
               ^
checksums.go:28:9: S1025: the argument is already a string, there's no need to use fmt.Sprintf (gosimple)
        return fmt.Sprintf("%s", string(cs))
               ^
matches.go:150:4: S1011: should replace loop with `mergedFileSizeIndex[fileSize] = append(mergedFileSizeIndex[fileSize], fileMatches...)` (gosimple)
                        for _, fileMatch := range fileMatches {
                        ^
Non-zero exit code from golangci-lint: 1
checksums.go:28:9: the argument is already a string, there's no need to use fmt.Sprintf (S1025)
matches.go:105:6: func InList is unused (U1000)
matches.go:150:4: should replace loop with mergedFileSizeIndex[fileSize] = append(mergedFileSizeIndex[fileSize], fileMatches...) (S1011)
units.go:13:6: func ByteCountSI is unused (U1000)
Non-zero exit code from staticcheck: 1
Linting failed, most recent failure: staticcheck
Makefile:61: recipe for target 'linting' failed
make: *** [linting] Error 1

The unused linting failures are because I'm treating those functions as part of a library/package without actually having them in one. The quick fix here is to pull them out, but a better approach (at least for now) would be to create a sub-package for them. I'll likely go that direction.

Same as with:

• atc0005/elbow#123
• atc0005/elbow#144
• atc0005/elbow#145

https://github.com/360EntSecGroup-Skylar/excelize/releases/tag/v2.2.0

What I got (tossing unrelated rows):

directory,file,size,size_in_bytes,checksum,remove_file
,,,,,
.,prune-input-testing.csv,25.2 KiB,25832,1d0c2cbe8fc192c57774fb39b258860bc66427ba2914ceaf46493112c35165ae,
.,report.csv,25.2 KiB,25832,1d0c2cbe8fc192c57774fb39b258860bc66427ba2914ceaf46493112c35165ae,

Actual:

$ sha256sum *.csv
1d0c2cbe8fc192c57774fb39b258860bc66427ba2914ceaf46493112c35165ae *prune-input-testing.csv
d942dce529d17457f5f42b386256da6c3d6b43118ec9219c6f0a5c2270a6c740 *report.csv

While working on something else I spot checked the doc.go file and noticed that the flags help output was outdated. Need to refresh it to match current help output.

Same thing as with atc0005/elbow#8.

Two packages I'm considering:

• https://github.com/cheggaaa/pb
• https://github.com/schollz/progressbar

• Review README
• Review GoDoc coverage
• Update changelog
• Create tag
• Create new release

Both the main README and GoDoc coverage may need to be updated to reflect recent changes to the Help output.

see atc0005/todo#10