scalameta / metabrowse Goto Github PK

$ metadoc --help
metadoc 0.1.0-SNAPSHOT
Usage: metadoc [options]
  --usage
        Print usage and exit
  --help | -h
        Print help message and exit

Note that it doesn't mention that a classpath is expected.

To unblock the Scala.js v1.0 upgrade, we disabled JsonSuite. This test suite is for a feature that nobody is using to my best knowledge. Metabrowse supports reading JSON files instead of protobuf files. This feature was added to see if there was interest outside of the Scala community to emit SemanticDB, but it was never publicly advertised (only in one private DM). It's OK to close this issue by removing JsonSuite.

https://microsoft.github.io/monaco-editor/api/interfaces/monaco.languages.referenceprovider.html

The reference provider interface defines the contract between extensions and the find references-feature.

.... to keep the readme focused.

As discussed elsewhere, we want to load content for each implemented language provider using futures. Some of the providers needs to access "global" state such as the editor instance(s). We can either try to implement this using services similar to Monaco which are injected and wired at start up or adopt something like Diode which provides several patterns for managing state, such as lenses and actions.

The corpus in the current demo is quite uninteresting code. It would be cool if the demo was a larger corpus such as for example the Scala community build, including semantic support for .sbt files, sbt sources and the Scala standard library.

A lot of features related with search are supported by vscode. We should investigate if it is possible to implement the platform specific parts of vscode to run completely in the browser.

• #36 - Workspace symbol provider
• #20 - Search by filename
• #29 - basic UI and start page
• Tabs
• File browser

A good starting point seems to be https://github.com/spiffcode/ghedit

We should add a license file before the first release.

Here are some findings from using the ZIP file of semantic DBs mentioned in #22 (comment) to generate a metadoc site.

• Duplicate semantic DBs: the Scalameta community build contains several duplicates semantic DBs for the same path under META-INF. Not sure if this can be sanely handled by metadoc, but the problem of duplicate definitions could still apply if metadoc where to support multiple versions of the same project/file, e.g. Akka 2.4 and Akka 2.5.

• Memory usage: The CLI currently loads all semantic DBs into memory and generates additional datasets for files and symbols in memory before writing anything out to disk. I was not able to generate a metadoc site for a significant subset of semantic DBs from the community build:

    > du -sh community/classes/
    581M	community/classes
  

• File name too long: The current symbol name encoding scheme does not work on file systems with modest path lengths.

See https://github.com/scalameta/metadoc/pull/33/files#r125193716

Currently, only the filename is included in the url. It would be nice if it was possible to create more fine-grained links, by for example

• line number
• range selection
• symbol definition (e.g. #/Foo.scala:MyFoo links to MyFoo definition)

From https://github.com/olafurpg/metadoc/pull/4#issuecomment-310866379:
It seems npm and yarn is installed on every ci run and node_modules is not cached. It would be nice to cache those too, but I don't know what's the best way to achieve that. The yarn.lock and node_modules are located in nested directories.

"See our demo page to try it out!" says readme.md, but the link is 404

Based on #27 (comment), we should show an empty start page with instructions on how to open file, symbol, or the editor's context menu.

At the same time we should also add some basic "chrome" for managing settings (e.g. theme), link to GitHub. I suggest starting out with Material Design Lite which is declarative and integrates well with webpack.

We should probably just pick either the type or term symbol.

The JsonTest file contains a pretty standard JSON ADT

sealed abstract class Json
case class JString(str: String) extends Json
// ...

No definition symbols seems to be generated for them. For example line 56 doesn't allow to "peek definition":

Metadoc is kind of useful now that #19 has been solved in #33. It would be nice to provide an sbt-metadoc plugin that generates a static site on some metadoc command. The command should compile all main and test sources and generate a site under some configured directory.

The reference provider seems to do a lot of redundant fetches

It might be a good idea to change the schema to store a map<string, Ranges> instead of List[(Filename, Range)]. That would mean we only store the filename once for all references in that file and we solve the redundant semanticdb fetching problem.

Currently, "Go to symbol..." shows raw symbols IDs as containerName

It would be nice if we instead showed the method signature such as the parameter types and result types.

It would be nice if it was possible to open a particular file by searching its name. Ideally, this should be possible in a similar way as vscode does it with Cmd+P

Also, the url hash should ideally follow the active filename so that it's possible to share a link to a specific filename (+ line number)

When following the "CONTRIBUTING" section in the README, I get the following error:

sbt @ metadoc > [error] module.js:491
[error]     throw err;
[error]     ^
[error] 
[error] Error: Cannot find module '/data/rw/code/scala/scala-profiling/metadoc/metadoc-js/target/scalajs-bundler-webpack-dev-server/node_modules/webpack-dev-server/bin/webpack-dev-server.js'
[error]     at Function.Module._resolveFilename (module.js:489:15)
[error]     at Function.Module._load (module.js:439:25)
[error]     at Function.Module.runMain (module.js:609:10)
[error]     at startup (bootstrap_node.js:158:16)
[error]     at bootstrap_node.js:598:3

Executing the next command in the guide ~js/fastOptJS succeeds but the server is not spinned up, so I cannot open the Metadoc website.

It would be nice if it was possible run scalafix rewrites on the code being browsed. In particular, it would be nice to see the sugars such as inferred type annotations and implicits expanded. Features such as jump to definition and see references should continue work on the rewritten source.

The README says: npm install -g yarn. I think it'll be better to prefer local installation. The build has to be adapted for npm install --local yarn to work. Currently it fails with the following error:

sbt @ metadoc > js/fastOptJS::startWebpackDevServer
[info] Installing webpack-dev-server in /data/rw/code/scala/scala-profiling/metadoc/metadoc-js/target/scalajs-bundler-webpack-dev-server
[info] Updating NPM dependencies
[info] Fast optimizing /data/rw/code/scala/scala-profiling/metadoc/metadoc-js/target/scala-2.12/scalajs-bundler/main/metadoc-js-fastopt.js
[trace] Stack trace suppressed: run last js/*:installWebpackDevServer for the full output.
[trace] Stack trace suppressed: run last js/compile:npmUpdate for the full output.
[error] (js/*:installWebpackDevServer) java.io.IOException: Cannot run program "yarn" (in directory "/data/rw/code/scala/scala-profiling/metadoc/metadoc-js/target/scalajs-bundler-webpack-dev-server"): error=2, No such file or directory
[error] (js/compile:npmUpdate) java.io.IOException: Cannot run program "yarn" (in directory "/data/rw/code/scala/scala-profiling/metadoc/metadoc-js/target/scala-2.12/scalajs-bundler/main"): error=2, No such file or directory

This is just a nice to have. Local npm packages are much superior in every case I can think of.

Related to #20. It would be nice if it was possible to search by "symbol", where a symbol is a definition such as a class/trait/object and maybe val/var/def.

If sbt-metabrowse has some kind of metabrowseForceCompile command that inserts the semanticdb-scalac compiler plugin and -Yrangepos scalacOptions, then we can add the plugin globally and run that command in any build without fiddling with the build settings.

Some builds do libraryDependencies := ... and scalacOptions := ... which overrides addCompilerPlugin from the plugin.

For the CLI to be able to generate a complete metadoc site it needs to be able to extract the webpack assets from somwhere. Options are:

• package (and publish) webpack assets in a separate org.scalameta:metadoc-js:version:assets jar (note the classifier)
• ship assets in the main metadoc-js jar for example inside a META-INF/metadoc-assets folder

Packaging can be achieved using sbt's Defaults.packageTaskSettings however version 0.8.0 of scalajs-bundler no longer support custom output paths so we need to find another long term way to resolve all assets generated by webpack.

Follow up on #55 (comment)

An idea testing infrastructure makes it easy to

• add new tests
• update tests when expected behavior changes
• review diffs from test

I'm personally a big fan of "expect tests" (see https://blog.janestreet.com/testing-with-expectations/), which are similar to partests in scala/scala.

Example expect tests in scalameta: https://github.com/scalameta/scalameta/blob/15a0187501dc8e5a6fe06d0e002ce1d3340968dc/scalameta/semanticdb-scalac/src/test/scala/scala/meta/tests/SemanticSuite.scala#L6

I'd love to setup something similar for metadoc once I finally get through with this scalafix 0.5.0 release.

The editor uses the Monarch language tokenizer defined in metadoc-js/src/main/resources/scala.ts which still lacks support for a lot of language constructs. A non-exhaustive list:

• Symbols defined with backticks

   val `number-42` = 42

• Strings defined with triple quotes

   val a = """
     select * from table;
   """

• String contexts

   val a = s"$serverName resolves to ${resolve(serverName)}"

More info on Monarch: https://microsoft.github.io/monaco-editor/monarch.html
Inspiration from the Scala vscode integration: scala.tmLanguage and scala.configuration.json

Monaco 0.9.0 (merged in #49) added a new TypeDefinitionProvider service
https://microsoft.github.io/monaco-editor/api/interfaces/monaco.languages.typedefinitionprovider.html

It would be nice to implement it in metadoc. The service provides a way to jump to the type definition when hovering over a non-type symbol (e.g., in term or pat position).
Example motivation microsoft/TypeScript#10394

olafurpg@c17d2d9 shows a skeleton of how to implement this service in metadoc.

Hi,

This is more of a question than an issues; so it it would be better to raise it elsewhere; please point me in the direction.

When I Find All References of a method that overrides from a Trait - "No Results" is reported.

Would there be options to explore a feature similar to Intelij's where it prompts: "Do you want to find usages for the base member?"

The goal is to go through a code set and find all unused methods. So any other approaches that could achieve that would also be of interest.

Thanks

We should mention how to incrementally compile changes after the webpack dev server is started.

See #32 (comment)

We currently use file:$filename but it might be a good idea to use a custom scheme such as semanticdb:$filename. It might get more tricky once we have multiple repositories.

I added a larger corpus to the build and found cases where reference provider returns too many results, including to positions inside comments etc.

Currently, metadoc only works for a single source file. It should be possible to switch between files when jumping to symbols defined in other compilation units.

By default, no information is shown when hovering a symbol. It would be nice to show at least type information by implementing the HoverProvider.

In addition, the default "show implementation" pop-up displayed when hovering and pressing Ctrl or ⌘ key on Mac could also be improved by implementing the ImplementationProvider:

@olafurpg It seems there was an issue with the release of 0.2.4. The gpg keys seems to be spurious or missing.

From #100 (comment)

Grouping by top-level package is problematic for widely referenced packages, for example the org.scalatest symbol index in the megasite is 11mb. I was thinking of chunking down those files in ~500kb sizes and the client needs to "guess" the first package index and from there we can link to a secondary package index. That would mean users you never download more than ~1mb to resolve a symbol (and we can lower 1mb at the cost of producing more files).

Either way, I believe even 11mb is an acceptable solution considering the megasite is for a corpus of 3.5 million loc corpus. We can further improve the situation in followup PRs.

Changelog: https://github.com/Microsoft/monaco-editor/blob/master/CHANGELOG.md#0100-17082017

https://code.visualstudio.com/docs/editor/editingevolved#_open-symbol-by-name

The api exists in vscode but not monaco https://github.com/Microsoft/vscode/blob/478365bce508cc391e4cfc5c9e1eeee7c87c39c9/src/vs/vscode.d.ts#L2034

We have all the data to implement this.

I expected DB to appear in the search list.

To unblock the Scala.js v1.0 upgrade, we updated the build to use only fastOptJS instead of fullOptJS. Full optimization was failing with the following error from com.google.javascript.jscomp

sbt:metabrowse> last js / Compile / fullLinkJS
[info] Full optimizing /Users/olafurpg/dev/scalameta/metabrowse/metabrowse-js/target/scala-2.13/metabrowse-js-opt
[debug] Linker: Compute reachability: 1020002 us
[debug] Linker: Assemble LinkedClasses: 97469 us
[debug] Linker: Check IR: 672343 us
[debug] Linker: 1867273 us
[debug] Optimizer: Batch mode: true
[debug] Optimizer: Incremental part: 258541 us
[debug] Optimizer: Optimizing 26648 methods.
[debug] Optimizer: Optimizer part: 2344534 us
[debug] Optimizer: 2616057 us
[debug] Refiner: Compute reachability: 254856 us
[debug] Refiner: Assemble LinkedClasses: 18135 us
[debug] Refiner: 299668 us
[debug] Module Splitter: Calculate Dependency Info: 4730 us
[debug] Module Splitter: Analyze Modules: 1701 us
[debug] Module Splitter: Assemble Modules: 14870 us
[debug] Module Splitter: 22834 us
[debug] Emitter: Generate Classes: 1528297 us
[debug] Emitter: Write trees: 41128 us
[debug] Emitter: Class tree cache stats: reused: 0 -- invalidated: 2685
[debug] Emitter: Method tree cache stats: reused: 0 -- invalidated: 16066
[debug] Emitter: 1658313 us
[debug] Closure: Create trees): 1101883 us
[info] Closure: 0 error(s), 0 warning(s)
[error] java.lang.IllegalStateException
[error] 	at com.google.javascript.jscomp.jarjar.com.google.common.base.Preconditions.checkState(Preconditions.java:492)
[error] 	at com.google.javascript.jscomp.OptimizeParameters.addVariableToFunction(OptimizeParameters.java:995)
[error] 	at com.google.javascript.jscomp.OptimizeParameters.optimizeFunctionDefinition(OptimizeParameters.java:903)
[error] 	at com.google.javascript.jscomp.OptimizeParameters.tryEliminateConstantArgs(OptimizeParameters.java:614)
[error] 	at com.google.javascript.jscomp.OptimizeParameters.process(OptimizeParameters.java:128)
[error] 	at com.google.javascript.jscomp.OptimizeCalls.process(OptimizeCalls.java:125)
[error] 	at com.google.javascript.jscomp.PhaseOptimizer$NamedPass.process(PhaseOptimizer.java:317)
[error] 	at com.google.javascript.jscomp.PhaseOptimizer$Loop.process(PhaseOptimizer.java:462)
[error] 	at com.google.javascript.jscomp.PhaseOptimizer.process(PhaseOptimizer.java:232)
[error] 	at com.google.javascript.jscomp.Compiler.performOptimizations(Compiler.java:2438)
[error] 	at com.google.javascript.jscomp.Compiler.lambda$stage2Passes$1(Compiler.java:808)
[error] 	at com.google.javascript.jscomp.CompilerExecutor.lambda$runInCompilerThread$0(CompilerExecutor.java:101)
[error] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] 	at java.lang.Thread.run(Thread.java:748)
[error] (js / Compile / fullLinkJS) java.lang.IllegalStateException

We should figure out what's causing this error and start using fullOptJS again

Model markers provide a way to annotate the code with compiler warnings. Should probably be optional.

https://microsoft.github.io/monaco-editor/api/modules/monaco.editor.html#setmodelmarkers