Comments (5)
Yep I agree. This is something that should be added in. I probably won't have time to look at this till next week though sorry. From memory this should be fairly straight forward to implement though and I can start looking at what you have done as a start.
from scc.
Implemented in #69
I didn't use your code too much due to some other changes that had happened but will totally credit you when this makes the next release after it passes PR.
from scc.
Done in this release https://github.com/boyter/scc/releases/tag/v2.4.0
from scc.
I've slapped together an unpolished and mostly untested version of this which seems to work fine;
diff --git a/processor/file.go b/processor/file.go
index 70a3419..527513f 100644
--- a/processor/file.go
+++ b/processor/file.go
@@ -201,7 +201,6 @@ func walkDirectoryParallel(root string, output chan *FileJob) {
}
wg.Wait()
- close(output)
if Debug {
printDebug(fmt.Sprintf("milliseconds to walk directory: %d", makeTimestampMilli()-startTime))
}
diff --git a/processor/processor.go b/processor/processor.go
index 1fed4fc..0e72a25 100644
--- a/processor/processor.go
+++ b/processor/processor.go
@@ -311,11 +311,18 @@ func Process() {
if len(DirFilePaths) == 0 {
DirFilePaths = append(DirFilePaths, ".")
}
- fpath := filepath.Clean(DirFilePaths[0])
- if _, err := os.Stat(fpath); os.IsNotExist(err) {
- fmt.Println("file or directory does not exists: " + fpath)
- return
+ fpaths := make([]string, 0, len(DirFilePaths))
+
+ for _, fpath := range DirFilePaths {
+ cfpath := filepath.Clean(fpath)
+
+ if _, err := os.Stat(cfpath); os.IsNotExist(err) {
+ fmt.Println("file or directory does not exists: " + cfpath)
+ return
+ }
+
+ fpaths = append(fpaths, cfpath)
}
SortBy = strings.ToLower(SortBy)
@@ -330,7 +337,22 @@ func Process() {
fileReadContentJobQueue := make(chan *FileJob, FileReadContentJobQueueSize) // Files ready to be processed
fileSummaryJobQueue := make(chan *FileJob, FileSummaryJobQueueSize) // Files ready to be summarised
- go walkDirectoryParallel(fpath, fileListQueue)
+ var wg sync.WaitGroup
+
+ wg.Add(len(fpaths))
+
+ for _, fpath := range fpaths {
+ go func(fpath string) {
+ walkDirectoryParallel(fpath, fileListQueue)
+ wg.Done()
+ }(fpath)
+ }
+
+ go func() {
+ wg.Wait()
+ close(fileListQueue)
+ }()
+
go fileReaderWorker(fileListQueue, fileReadContentJobQueue)
go fileProcessorWorker(fileReadContentJobQueue, fileSummaryJobQueue)
Although it now causes the unit tests to fail with this error;
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000cac00, 0x11cecd3, 0x19, 0x11f4650, 0x106ea01)
/usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:879 +0x383
testing.runTests.func1(0xc0000ca100)
/usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:1119 +0x78
testing.tRunner(0xc0000ca100, 0xc0000ade08)
/usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:827 +0xbf
testing.runTests(0xc000094160, 0x1342d40, 0x4e, 0x4e, 0x100b51f)
/usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:1117 +0x2aa
testing.(*M).Run(0xc0000c8080, 0x0)
/usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:1034 +0x165
main.main()
_testmain.go:258 +0x13d
goroutine 28 [chan receive]:
github.com/boyter/scc/processor.TestWalkDirectoryParallel(0xc0000cac00)
/Users/jack/Development/Go/src/github.com/boyter/scc/processor/file_test.go:88 +0x1af
testing.tRunner(0xc0000cac00, 0x11f4650)
/usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
/usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:878 +0x35c
FAIL github.com/boyter/scc/processor 0.050s
Which is understandable, as I removed the close
call from walkDirectoryParallel
.
from scc.
Merged in. Going to close this down, but it should be turned into a binary release sometime soon.
from scc.
Related Issues (20)
- How can I include all _test.go files by regex. HOT 2
- Question: Use wildcards with excludes? HOT 1
- Unable to use multiple count-as flags HOT 5
- GNUmakefile is not recognized as Makefile HOT 1
- Recognize file type base on mime type HOT 8
- --exclude-dir doesn't exclude directories HOT 4
- Shields like style HOT 2
- Different Results on Windows and Linux HOT 5
- Cut new release? HOT 2
- Why CUDA files are not supported? HOT 2
- scc handle .gitignore are unexpected HOT 1
- error in Ubuntu 18.06: file or directory does not exist HOT 4
- sourcehut on sloc.xyz HOT 8
- Wrong language Complexity/Lines calculation when using -wide
- scc version 3.2.0 fails to compile with 32 bit golang toolchain in Windows HOT 3
- "Arvo" should be "Avro" HOT 1
- Add Templ Support
- Let badge exclude/include some folders HOT 4
- scc includes the .ignore file as part of the project counting HOT 4
- Feature Request Azure Bicep Support HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scc.