githubocto / repo-visualizer Goto Github PK

the visualization does only include folder structures up to a certain maximum depth. details that are deeper in the directory tree are completely omitted. could the maximum depth be a configurable parameter?

Thanks for this nice action!

When adding it to my GitHub action file it turns out it's unable to push it because the master branch is protected.

/usr/bin/git push
remote: error: GH006: Protected branch update failed for refs/heads/master.        
remote: error: 2 of 2 required status checks are expected. At least 1 approving review is required by reviewers with write access.        
To https://github.com/nextflow-io/nextflow
 ! [remote rejected] master -> master (protected branch hook declined)
error: failed to push some refs to 'https://github.com/nextflow-io/nextflow'
(node:1610) UnhandledPromiseRejectionWarning: Error: The process '/usr/bin/git' failed with exit code 1
    at ExecState._setResult (/home/runner/work/_actions/githubocto/repo-visualizer/main/index.js:935:21)
    at ExecState.CheckComplete (/home/runner/work/_actions/githubocto/repo-visualizer/main/index.js:921:16)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/githubocto/repo-visualizer/main/index.js:824:21)
    at ChildProcess.emit (events.js:210:5)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)

I've managed to make it work by pushing to a different branch. However, it would be desirable to have some kind of option to allow pushing also to a protected branch if the committing user is allowed to.

i hope the feature of "What files are connected " can be ues,now I only can use repo Visualizing

This is a follow up to #20.

The Interactive Repo Visualizer tool is superior to the static SVG generation option.

It will be great to have it either open-sourced, released as part of the repo Insights (tab) on GitHub, or at least add official support for accessing private repos.

In the mean time, here are the instructions to get the current tool working with private repos:

1. Go to https://github.com/settings/tokens

2. Generate a new PAT with repo permissions

3. Go to https://mango-dune-07a8b7110.1.azurestaticapps.net/

4. Add the following code to the JS console replacing TOKEN with your GitHub PAT

   window.originalFetch = window.fetch;
   const modifiedFetch = (resource, init) => window.originalFetch(resource, { ...init, headers: { "Authorization": "Bearer 
   TOKEN" } });
   window.fetch = modifiedFetch;

5. In the repository input, type any repository (public or private)

Kudos to @filipegiusti for posting the original instructions for private repo support.

repo-visualizer is failing to push changes back to private repo:

/usr/bin/git commit -m Repo visualizer: updated diagram
[main fd9d4bb] Repo visualizer: updated diagram
 1 file changed, 1 insertion(+), 11 deletions(-)
 rewrite docs/repviz.svg (100%)
/usr/bin/git push
fatal: could not read Username for 'https://github.com': No such device or address
(node:2291) UnhandledPromiseRejectionWarning: Error: The process '/usr/bin/git' failed with exit code 128
    at ExecState._setResult (/home/runner/work/_actions/githubocto/repo-visualizer/main/index.js:935:21)
    at ExecState.CheckComplete (/home/runner/work/_actions/githubocto/repo-visualizer/main/index.js:921:16)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/githubocto/repo-visualizer/main/index.js:824:21)
    at ChildProcess.emit (events.js:210:5)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
(node:2291) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:2291) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Currently, the SVG background is set to white, which doesn't provide optimal compatibility with dark themes when used in README files. I suggest allowing the SVG background to be transparent, so it can seamlessly adapt to both light and dark themes.

A transparent background would improve the overall user experience and visual appeal of the output across various themes. Thanks for considering this enhancement!

It would be great to be able to resize the svg and have more information represented within each directory as a result of the resize - especially good for the big repos!

Hello,
The link in the readme for default colors takes to this link instead of taking to the actual file of colors, which I suppose should be this one.

Environment: This is run inside a docker container.

It seems the action fails to verify if a branch exists or not:

/usr/bin/git fetch
/usr/bin/git rev-parse --verify project-tree
fatal: Needed a single revision
Branch project-tree does not yet exist, creating project-tree.
/usr/bin/git checkout -b project-tree
Switched to a new branch 'project-tree'

In the above case, it says that project-tree doesn't exist, when it exists on remote. Furthermore, this matters because it later fails to push as the branch already exists on remote and the local history isn't updated:

[project-tree REDACTED] Update Project Tree (REDACTED).
   1 file changed, 1 insertion(+)
   create mode 100644 diagram.svg
  /usr/bin/git push --set-upstream origin project-tree
  To REDACTED
   ! [rejected]        project-tree -> project-tree (non-fast-forward)
  error: failed to push some refs to 'REDACTED'
  hint: Updates were rejected because the tip of your current branch is behind
  hint: its remote counterpart. Integrate the remote changes (e.g.
  hint: 'git pull ...') before pushing again.
  hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Right now there's no need for us to waste quite a lot of CI resources after diagram.svg is commited.

Ideally I would want to have [ci skip] tag in a diagram commit message.
But, I understand that this might not work for every project, so this looks like good candidate for an option, something like with: skip_ci: true

What do you think?

Btw, our diagram: https://wemake-python-stylegui.de/en/latest/pages/api/index.html#overview
🙂

Good day,

I found out about this project today and I'm very impressed! It would be a great tool for people on my team (myself included) to get to know quickly code bases we are not familiar with.

But we don't use GitHub in my company.

Is there a way to run this repo-visualizer locally? A bit like what was done here: https://octo-repo-visualization.vercel.app

Thank you!

Other utilities like cloc follow the .gitignore file. It would be great for the repo-visualizer to also follow the .gitignore file, rather than specifying what to ignore again with globs.

Here is an example on how to get the list of files that the repo-visualizer should use:

import { execa } from "execa";
import { getMonorepoPath } from "@nodejs/list-workspaces";

export async function gitLsFiles() {
  const { stdout } = await execa("git", ["ls-files"], {
    maxBuffer: 500_000_000,
    cwd: await getMonorepoPath(),
  });
  return stdout;
}

I'm working on an Elixir + Phoenix project. There are two files I wish to ignore:

• assets/package-lock.json
• priv/repo/structure.sql

Given this config:

name: Create diagram

on:
  workflow_dispatch: {}
  push:
    branches:
      - "AP-47"
jobs:
  get_data:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@master

      - name: Update diagram
        uses: githubocto/repo-visualizer@main
        with:
          commit_message: "skip ci"
          excluded_paths: "mix.lock,.github,_build,priv/repo/structure.sql,assets/package-lock.json,nix"

I see this:

Why is that?

Hi both githubocto/[email protected] and githubocto/repo-visualizer@main fail with the same git error.
Do you know what might be the issue? The git version perhaps?

Commit and push diagram
  /usr/bin/git commit -m Repo visualizer: updated diagram
  [detached HEAD b4c04e1] Repo visualizer: updated diagram
   1 file changed, 1 insertion(+), 1 deletion(-)
  /usr/bin/git push
  fatal: You are not currently on a branch.
  To push the history leading to the current (detached HEAD)
  state now, use
  
      git push origin HEAD:<name-of-remote-branch>
  
  Error: Error: The process '/usr/bin/git' failed with exit code 128

I was thinking to change the colour scheme of particular files, background and circles, would be that possible?

Happy to learn a new trick if you tell me how it could be done.

Oh found hints how to do it probably https://github.com/githubocto/repo-visualizer/blob/main/src/Tree.tsx#L85

I really like repo-visualizer, thanks for the action!

But it would be really cool if we could create multiple svgs with an array of objects where the properties of the object are the existing inputs

...And a bonus feature would be to allow for regex matching so if you want multiple diagrams in the immediate children directories of a particular directory it is easy to do!
(instead of copying:

uses: githubocto/[email protected]
  with:
    output_file: "images/diagram.svg"
    root_path: "/"

for each svg)

I see excluded_paths is supported, but it would be fantastic to be able to exclude filenames (especially with wildcard support for partial matches) using something like excluded_filenames.

Perhaps this logic could be squeezed in here:

An example use case is excluded_filenames=*.Designer.cs,*.resx to exclude automatically-generated .NET files from large Windows Forms applications.

PS: consider adding a CONTRIBUTING.md to this repo

to reproduce

• check the link to my repo visualization
  • https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=LangLangBart%2FboonGUI

This could be the same as #2 - but quieter?

We're seeing something unusual where it looks like the commit and push is working fine, and yet nothing appears in our git repo when using the GitHub Action.

Our output from the action looks as follows:

/usr/bin/git add ./diagram.svg
/usr/bin/git status --porcelain ./diagram.svg
A  diagram.svg

A  diagram.svg
diff: A  diagram.svg

/usr/bin/git commit -m Repo visualizer: updated diagram
/usr/bin/git push
[development 274d2128] Repo visualizer: updated diagram
 1 file changed, 1 insertion(+)
 create mode 100644 diagram.svg
Everything up-to-date
All set!

Yet commit 274d2128 does not exist on the development branch anywhere to be seen!

I'm wondering where it has gone?

We don't see any other error messages, and I cannot determine any setting that looks to forbid pushing to the branch. We're not getting an error like #2, although we do have the protected branches on. So I'm wondering if we should be seeing the same issue?

We resorted to cat'ing the file out and copying it out - we're just playing anyway, so it's not critical :)

P.s. Thanks for this tool - it's really interesting to see our code visualised in this way.

Steps to reproduce this issue:

Set excluded_globs: "**/node_modules/**,**/dist/**,**/lib/**,**/built/**,**/public/**,**/.cache/**" to exclude nested files in a monorepo

Like this

      - name: Update diagram
        uses: githubocto/repo-visualizer@main
        with:
          commit_message: "chore(core): update diagram"
          output_file: "docs/assets/images/core-repo-diagram.svg"
          excluded_globs: "**/node_modules/*,**/dist/*,**/lib/*,**/built/*,**/public/*,**/.cache/*"
          excluded_paths: "node_modules,dist,lib,built,.cache,public,out,build,.yarn,.vscode,package-lock.json,yarn.lock"

Expected

E.g. files inside /packages/*/.cache are ignored

Actual behavior

files inside /packages/*/.cache are not ignored by minimatch

There is a bunch of bugs with the default minimatch options

Minimatch options:

• https://github.com/micromatch/micromatch#options

Related issues:

• isaacs/minimatch#30 (comment)
• micromatch/micromatch#199

This is kind of follow up on #2 - since in my repository master branch is protected I don't really see any way to use the GH action currently. It could be nice if I could push the diagram to another branch, and then merge it on my own if I want.

This could be also useful for folks keeping documentation on docs branch or for GH Pages hosted on gh-pages branch.

The writeup has a section "Potential future directions" that say:

To find these connections, I scanned the contents of each file for import statements, then linked that file with the one it imports from.

However I cannot find docs of the feature in README or code of the function in this repo.

It would be wonderful for the repo-visualizer to have a CLI version, so that it can used locally or in other environments.

Simply put, all the code is already written in https://github.com/githubocto/repo-visualizer/blob/main/src/index.jsx

Just make another version without the actions code, bundle, and publish. It is fairly straightforward with https://www.npmjs.com/package/tsup and https://www.npmjs.com/package/commander

For example:

import fs from "fs-extra";
import React from "react";
import ReactDOMServer from "react-dom/server.js";
import { Tree } from "../vendor/repo-visualizer/src/Tree";
import { processDir } from "../vendor/repo-visualizer/src/process-dir.js";

export const main = async () => {
  const rootPath = ""; // Micro and minimatch do not support paths starting with ./
  const maxDepth = 9;
  const colorEncoding = "type";
  const excludedPathsString =
    "node_modules,bower_components,dist,out,build,eject,.next,.netlify,.yarn,.git,.vscode,package-lock.json,yarn.lock";
  const excludedPaths = excludedPathsString.split(",").map((str) => str.trim());

  // Split on semicolons instead of commas since ',' are allowed in globs, but ';' are not + are not permitted in file/folder names.

  // const excludedGlobsString =
  //   ""
  //   "frontend/*.spec.js;**/*.{png,jpg};**/!(*.module).ts";
  // const excludedGlobs = excludedGlobsString.split(";");
  const excludedGlobs = [
    "**/node_modules/**",
    "**/dist/**",
    "**/out/**",
    "**/build/**",
    "**/.cache/**",
    "**/.next/**",
    "**/.keystone/**",
    "**/.vercel/**",
    "**/package-lock.json",
    "**/*.bundled_*.mjs",
    "**/tmp/**",
    "**/vendor/**",
    "**/.DS_Store/**",
  ];

  const data = await processDir(rootPath, excludedPaths, excludedGlobs);

  const componentCodeString = ReactDOMServer.renderToStaticMarkup(
    // @ts-ignore
    <Tree data={data} maxDepth={+maxDepth} colorEncoding={colorEncoding} />,
  );

  const outputFile = "./repo-visualization.svg";

  await fs.outputFile(outputFile, componentCodeString);
};

and

import { Command } from "commander";
import { readPackageUpSync } from "read-pkg-up";
import { main } from "../core";

const pkg = readPackageUpSync({ cwd: __dirname })!.packageJson;
export const program = new Command()
  .name(pkg.name)
  .description(pkg.description || pkg.name)
  .version(pkg.version, "-v, --version")
  .action((options: { version?: boolean }) => {
    if (options.version) {
      console.log(pkg.version);
    } else {
      main();
    }
  });

You may need to debug the react bundling following tips from egoist/tsup#579 and egoist/tsup#589 and replace lodash with lodash-es

I didn’t link together the options, but commanderjs can support them

Anyway, my rough proof of concept works, but I would much rather see the cli live here in the original, than in a fork.

With large repos not only does the graph hang and slow down, but the visualization itself only shows folders as the files are too small to render. It's largely a bunch of grey circles.

If there was a way to whitelist paths vs blacklisting that may make is somewhat better?

Playing around with it locally, I very much didn't expect it to automatically try to push my changes.

Using @main is not exactly great in an action – could be v0.1.0 maybe?

The cloud is awesome almost as AI is, but I'm pretty sure installing and running this locally (on the already supported Ubuntu platform as I understand) could work without requiring too many tweaks or need for a docker container. This would also provide a simple and clean workflow for updating the diagram without implying a dedicated commit per real code revision.

Hey it would be great if you could open source the interactive visualizer https://octo-repo-visualization.vercel.app/?repo=paperjs%2Fpaper.js

If that is not possible, at least adding support for private repos with Access tokens would be nice

Another idea here is this repo could deploy an interactive version of the visualizer on gh pages so we can get that on each PR rather than the static svg

Grey circles are R scripts

I think it would be useful to have a flag where we can set it to open a PR instead of pushing to the main branch

I had a rule in place to prevent from pushing to the main repo and the action failed

A  diagram.svg

A  diagram.svg
diff: A  diagram.svg

Commit and push diagram
  /usr/bin/git commit -m Repo visualizer: update diagram
  [main a6487aa] Repo visualizer: update diagram
   1 file changed, 1 insertion(+)
   create mode 100644 diagram.svg
  /usr/bin/git push
  remote: error: GH006: Protected branch update failed for refs/heads/main.        
  remote: error: Changes must be made through a pull request.        
  To https://github.com/************************
   ! [remote rejected] main -> main (protected branch hook declined)
  error: failed to push some refs to 'https://github.com/************************'
  Error: Error: The process '/usr/bin/git' failed with exit code 1

My action:

name: Create diagram
on:
workflow_dispatch: {}
push:
  branches:
    - main
jobs:
get_data:
  runs-on: ubuntu-latest
  steps:
    - name: Checkout code
      uses: actions/checkout@master
    - name: Update diagram
      uses: githubocto/repo-visualizer@main
      with:
        excluded_paths: "ignore,.github"

In my case pushing diagram failed because master branch is protected - however, the GH Action had success status, and I was a bit confused for a moment why I don't see a diagram in the repo ;-)

https://github.com/fsharp/FsAutoComplete/runs/3253469374?check_suite_focus=true#step:3:281

I was wondering if this would work for GitLab. Thanks

Committing the file back to the repo automatically is very handy, but it would be nice if this could be disabled.

We have a large mono-repo and we would only use this diagram for our main branch, so we would be happy storing this (potentially large and frequently changing) file outside of git, at the expense that when we link to it from our markdown docs it would only ever show the latest version. I imagine this would be preferable for most people but that's just speculation.

Would you accept a patch for this?

It appears that, at least at the front-end side of things, repo-visualizer has trouble with generating images live. From scrolling down the page associated with the demo, it would "freeze up". I would like to help if there was a branch showing the source code for the page.

If that is not possible, here are my suggestions:

• Reduce the height of the page
• Have a loading phase
• Make a download button/option
• Minify the code

Currently when depth is reached, the hidden files still affect the size of the top directories.

It would be nice to have size of last circle the same as any other (as it has reached the depth limit)

A easy to implement yet HUGE suggestion that would help this project would be using the colors defined in the languages.yml file in github/linguist. This way developers will be familiar with what they're looking at.

since the maximum display depth is limited, a significant part of the folder hierarchy is omitted. perhaps this can be improved by making it possible to zoom in on subfolders.