treetide / underhood Goto Github PK

UnderHood is a code browsing interface backed by Kythe indices.

License: Apache License 2.0

Nix 78.17% Haskell 7.61% Vue 8.28% HTML 0.13% JavaScript 1.70% Starlark 1.29% CSS 2.28% Dockerfile 0.12% Shell 0.07% Awk 0.36%

underhood's Introduction

UnderHood

Mono-repo for TreeTide UnderHood.

Note for UnderHood: treetide-workspace refers to the repository root with the bazel WORKSPACE file. Reason: UnderHood was forked out from the treetide monorepo.

Note: the Kythe backend's API is now out of sync slightly with what the UI expects. Zoekt-underhood backend matches it. If keen to revive Kythe API, ping.

Getting started super-quickly

Using docker-compose and the prebuilt JS distributables.

For a quick demo with some pre-baked source repos:

$ cd docker
$ docker-compose -f docker-compose-demo.yml up

For your own Zoekt index:

$ cd docker
$ ZOEKT_INDEX_DIR=/path/to/zoekt/indexdir/ docker-compose up

Then navigate to http://localhost:8080. Quick usage:

Click around in the filetree
Use top search bar just like you would for Zoekt searcher, for example r:under Option f:go$

Source code navigation (could be better, but for now):

Ctrl+click to find lax matches for word under cursor
Hover with mouse + X: same lax matches as Ctrl+click
Hover with mouse + C: exact-word matches
Select text + V: lax matches on selection
Select text + B: exact-word matches on selection

Matchlist abbreviations:

SNIP means line is same as in other hit, but file differs
DUP means whole file is same as for some other hit

Getting started quickly

Building backend

Get a nix-shell in repo root. Get a coffee when you do this the first time.

Execute bazel build -c opt //treetide/....

Building frontend.

Get a nix-shell in treetide/underhood/ui.

For the first time, do npm ci to get node_modules pulled.

Running underhood locally.

Getting a Kythe index

TODO(robinp): automate. In the mean time use https://github.com/TreeTide/kythe/tree/nix (see https://github.com/TreeTide/kythe/blob/nix/README.treetide.md) to compile Kythe.

Once Kythe tools are compiled, TLDR of getting an index (on Kythe itself): https://gist.github.com/robinp/a7a10db5b38630a9a0ffc805fc402500

Running underhood

With backend and frontend built like above, use bazel run -c opt //treetide/underhood/frontend_server to start the frontend API server (from a repo-rooted nix-shell, as always).

In treetide/underhood/ui, execute npm run start:dev to start up the hot-reloading UI server.

Visit http://localhost:9000.

Deploying underhood using NixOps

To a VirtualBox VM

First, build the backend using above instructions. This step is not yet hermetic in the deploy process. Then get a nix-shell in production. Now you have the ttops command, which is just an alias to a version-pinned nixops.

The ttops create -d my-uh-vm underhood/local/vbox.ops.nix command will set up the deployment metadata, which you can verify with ttops list.

Use ttops deploy -d my-uh-vm to deploy a new version to the VM.

Finally execute ttops ssh -d my-uh-vm underhood-main -L 0.0.0.0:9001:127.0.0.1:80 to tunnel the VM's web server to your port.

Use ttops scp -d my-uh-vm ... to copy kythe binaries and indices to the VM for the time being. Start the kythe api server in the vm.

Bring up frontend_server in the VM using systemctl start underhood_frontend_server, check its status using systemctl status and journalctl.

Visit http://localhost:9001.

File structure

treetide-workspace
  |
  +- nix                -- Pinned nixpkgs versions used throughout.
  |
  +- treetide           -- Source and data root, along with build rules.
  |    |
  |    +- experimental  -- Throw-away code.
  |    |
  |    +- thirdparty   -- Checked in third party code, with licenses.
  |    |
  |    +- ...           -- Various projects, libs.
  |
  +- production         -- Files related to running services in various envs.
       |
       +- ...           -- Bringing up projects, grouped by project.

Git workflow

Follow https://nvie.com/posts/a-successful-git-branching-model/.

Nix

Most parts of the repo use Nix / Nixpkgs: Development stuff uses nix-shell to bring dev tools quickly in scope. Bazel is sourced through nix, so bringing it up on workstations (or build envs) is painless and reproducible. Inside Bazel, rules_nixpkgs is used to bring in tools. Production stuff uses nixos configs to describe and compose services.

Currently nixops is used to deploy some of these nixos systems. Note however that the general consensus is that nixops is convoluted and buggy, which has two implications:

Non-machine resources (queues, etc) might not be the best idea to manage with nixops.
Large-scale machine deployments might better be done without nixops, just using a thin layer upon nix-copy-closure. See http://www.haskellforall.com/2018/08/nixos-in-production.html.

You might need to define the tt path alias pointing to treetide-workspace. This is needed so nix expressions don't need relative imports and so are easier to move. Define it using:

export NIX_PATH=tt=/path/to/treetide-workspace:$NIX_PATH

or pass using -I tt=... on nix invocations.

Build tooling

Use Bazel (http://bazel.io/) for projects if it has language support. This includes most server-side languages.

Don't use Bazel for JS - as of early 2019, it doesn't give a good tradeoff for smaller projects.

Third-party code

Normally third-party code should go in treetide/thirdparty.

Random links

About mono-repos: http://danluu.com/monorepo/.

underhood's People

Contributors

Stargazers

Watchers

Forkers

kamahen ayazhafiz mbrukman robinp-tw c0ns0le

underhood's Issues

Use branch or commit metadata for operations

Now the UI operates with the assumption of a static background dataset. But the indices can updated, and filetrees becoming stale. Think through implications and take appropriate steps in UI to invalidate stale model parts.

Browser bottom panel doesn't work with Kythe v0.0.31 (and later)

With Kythe v0.0.31, the browser bottom panel doesn't show anything when an anchor is clicked or moused-over. It works for Kythe v0.0.30.

To reproduce:

Download pykythe.
Edit the Makefile's ENTRYSTREAM_EXE, WRITE_ENTRIES_EXE, etc.
- The Makefile points VERIFIER_EXE to the executable generated from the Kythe source tree - this is only for getting Kythe PR 4199, which is a cosmetic change to the verifier "?" output -- you can use /opt/kythe/tools/verifier if you wish.
Run make target run-underhood-all.
You may prefer running the individual commands in separate windows, to make clean-up easier.

Here is an example URL for browising:
http://localhost:9000/#/file/kythe:%2F%2Ftest-corpus%3Froot=test-root%3Fpath=tmp%2Fpykythe_test%2FSUBST%2Fhome%2Fpeter%2Fsrc%2Fpykythe%2Ftest_data%2Fbindings.py

Explain "nix-shell" etc.

The README talks about "nix-shell in repo root". Please provide a pointer for those of us who aren't familiar with this tool.

UI tests

We need to keep regressions from happening, and UI tests are great for that.

Should we mock server responses? Or should we test end-to-end? Let's start with former for first.

Handle multiple targets at a given span

A given source code span can contain multiple Kythe anchors, which in turn can target different semantic entities (maybe via different edge types).

On the UI now we have chooseDecorStrategy, that currently chooses the tightest span, or in case of equal lengths, an arbitrary (first) one. (This confusing behavior almost lead to a false bug report in kythe/kythe#4129).

Examples

Definitions

According to the Kythe schema, the definition anchor has (at least) two outgoing edges: defines/binding to the definition entitiy, and completes to the declaration.

Expectation: when selecting for purposes of getting backrefs, the defines/binding target should take precedence automatically.

Local binding aliasing global thing

As described in kythe/kythe#3934, some languages want both defines/binding to a local thing and a ref to a global thing from the same anchor. For example, TypeScript or Python for from foo import bar imports, or Haskell for f Bar{bar} record puns [1] [2] or positional deconstruction 3.

Expectation: alternative behavior. UI should offer to show backrefs for the global ref, bringing repository-wide backrefs (including, module paging, the localized usage), or for the defines/binding, bringing backrefs of only the localized usage.

[1]: sidenote, there are separate-anchored variants such as from foo import bar as baz or f Bar{bar = baz} as well, here bar having the ref and baz the defines/binding, the.

[2]: Actually Haskell also has similar-ish import syntax (qualified etc), but on the language-level (GHC level?) there is no separate local binding, it is just sugar. Might make sense to emulate though.

Multi-ref due to ellipsis

For example in Haskell's RecordWildCards, the .. in f Bar{..} brings into scope all of Bar's fields. [1]

Cool expectation: A really cool UI would expand the actually used fields in-place, thus reducing the case to the above "aliasing" case. A really cool programmer would not use RecordWildCards in the first place though (but the temptation is so hard to resist, and we are not here to judge).
Alternative expectation: alternative behavior. Give chance to the user to select which reference they want to see. Not as cool though.

[1] Combined with above "aliasing" case, there would actually be multiple refs and defines/bindings. This happens also for blanket-import syntax (see #9).

Subcase: Record-update with punning (or god forbid RecordWildCards)

Assuming data Bar = Bar { foo :: Int, bar :: Int}, the RecordWildCards syntax let foo = 1; bar = 2; in Bar{..} will have (assuming aliasing implemented) at the ellipsis refs to the global fields and refs to the local bindings.

Cool expectation: again, expanding in-place. The - not so large - difference is that differentiation is between refs of different kind, not between a ref and a defines/binding. A bit more (server-side?) preprocessing would be needed so sensible differentiation can be made to the user about the refs.

Positional constructors

In Haskell, data Bar = Bar Int should - after implementing google/haskell-indexer#104 - at the Int span both have a defines/binding towards the field def, and a ref towards the Int type.

Multiple refs due to ambiguity

For example in Python's case, there could be multiple refs to alternative bindings: see kythe/kythe#3815.

Expectation: alternative behavior, should let the user choose.

Notes

When we talk about a choice, the UI might also be free to take some smart behavior - if there are not too many choices (say 2), it could sideload both of them and present in parallel? And highlight with different colors.

Weinberg code shuffling (tracking issue)

Tracking issue to implement various code shufflers, inspired by Gerald M. Weinberg's the Psychology of Computer Programming (Part 4, Chapter 13, Documentation).

The observation is that once a programmer is stuck on debugging, he is stuck since he already tried all rational approaches. So irrational / unconventional methods are needed to shake the programmer out of stuckness - the "set". In Underhood we constrain the methods to code display methods.

Examples from the book (pdf page 442), with some additional remarks:

(Note: many items from the list, which was published in 1971 originally, is clearly a forerunner of many modern niceties. Absolutely stunning, respect to the elders.)

Inspired custom additions:

Nicer filenames in bottom backref panel

New we print the full kythe uri - rather should only print the file name.

Bonus points for small icons corresponding the language.

Backref autoload behavior is odd

Now we autoload crossrefs on hover. Sometimes this is convenient, but sometimes not.

Maybe make it load on click only, or configurable. Maybe only load if click happens on definition site, otherwise jump directly to definition.

The thing is, sometimes we don't want to lose the backrefs panel since it is some important context. Sometimes we don't care though. Maybe we should let the user pin an important backref listing as a tab?

Make tree-nav load subtrees on-demand.

Now the full tree is loaded upfront, which is a slow operation for large trees.

We could split the load in two parts:

The backend should cache the full tree. So only the first invocation is slow.
The UI should request only one (yet invisible) level in advance.

TreeNav should jump to displayed file.

Expand and focus on the file shown in the editor.

Link out to hosting repos

It should be possible to open any referred resource on the hosting repo (like github) easiely.

The currently viewed file or its parent dirs (from viewer header).
Specific line of the viewed file.
Reference results

For this, the backend should pipe through the templates usable, or maybe pre-render the templates already to spare the UI from specifics.

Source colorization customization?

Is there a way to colorize the sources displayed by Underhood?
When I display Python source, it correctly colorizes (for example) print, if, for; incorrectly colorizes map (but does allow it as an anchor); and doesn't colorize def, not, in.

Also, it would be nice if there were a way to customize display of comments and to turn off colorization within them.

=== Edit by @robinp below

Option to underline xrefs or not by default
Pass language to CodeMirror
Selectable CodeMirror theme css

Slow loading of codemirror editor

Likely due to variable height codemirror container, see codemirror/codemirror5#4076.

Highlight token in local viewer after click

With loose xref search backend, we don't have decors for the file. We should still highlight a given token when clicked?

Echo back search

In free-form query modes, it is not always clear what token the search results correspond to, and other settings (like lax or boundary-strict search). Should feature these somewhere at top of the reference results pane.

Imported-via feature

For languages that have blanket-import-everything-from-module [1], the UI could indicate what are the actually used bindings at a blanket-import.

Reverse, given a usage, the UI could indicate (via highlight, and/or in backrefs) which import was used to bring it in scope.

See inspiration in google/haskell-indexer#100.

[1]: Haskell for sure. ML's open structure syntax? Java's import static foo.*?

Indicate if more results are available for free-form query

Now if not all results are returned due to limits, there's no clear indication that there are more (or possibility to fetch them).

At least we should indicate and ask to restrict the query further.

Constant filters

Now a top-bar search or a reference search doesn't have any fixed constraints, making it harder to do repeated search subject to same constraints.

There could be a combobox for additional constant query expression for top-bar. (Should this also affect ref results by default? Probably, so this would be an ultimate global filter).

And there could be an other combobox for the results filter - this can be used to quickly filter away certain file types, or test patterns.

Expandable call chain in backref panel.

It is often useful to see callers of a suspicious use site.

It seems only the columnar beam pipeline serves callers - see https://groups.google.com/forum/#!topic/kythe/hbo_hFkBuXE.

codeunderhood.com still online?

Hi! I wanted to check out codeunderhood.com but it's just infinitely spinning (and I suspect will time out). Is this a temporary thing or is it permanently offline? If the latter, do you have a container build that would make it easy to preview locally, or someone using it that I could look at? Thanks!

Cancel in-progress search with canceller if needed

So resources are not wasted.

Router-based top search

Now we can perform a top-bar search, but it is not reflected in the route. This way search results can't be easily shared.

If we had a route for this, next question is: what do we do with the filetree and file viewer? Could we omit showing these? That opens a can of worms of questions...

Server should return file types

Now we infer on the client side (see LangIcon.vue). Instead the server should send them.

Loading race between source text and decorations

On slowly loading files when clicking around, a file might get the decorations (hyperlinks) of an other one. Likely JS callback scheduling on UI side.

Build and publish Docker image

For one-step (or at least few-step) usage.

Use custom search to speed up loading of large sources

Now we have CodeMirror viewportMargin set to Infinity for quick searches on the page (with Ctrl-F), but this makes loading the full doc upfront and thus slow.

We should limit to some small size, and use custom search (or a search plugin already present for CM).

History navigation doesn't keep filetree in sync

When navigating history, the filetree should stay in sync and show the file currently in the viewer.

Rethink focus

Now focus is ad-hoc: needs initial click for viewer pane to get focus, and after that we do best effort to give back focus after operations in different panes (like after click in filetree or refs).

We should maybe give visual indication of where the focus is? Or make key combinations work regardless of where the focus is - for example, a boundary-respecting mouse-pointer search could work regardless of focus, as long the pointer is at a sensible place.

Sounds like mouse events should work globally, while key-only events (we don't have such now) would need to take focus in account.