Giter VIP home page Giter VIP logo

facebookincubator / glean Goto Github PK

View Code? Open in Web Editor NEW
914.0 25.0 44.0 34.88 MB

System for collecting, deriving and working with facts about source code.

Home Page: https://glean.software/

License: Other

Makefile 0.17% Haskell 33.84% C++ 19.86% Python 0.38% Thrift 1.40% C 0.12% Lex 0.09% Yacc 0.16% Shell 0.18% Objective-C 0.12% Perl 0.08% JavaScript 0.12% CSS 0.02% Dockerfile 0.03% Objective-C++ 0.02% Hack 40.73% Rust 1.21% Go 0.02% TypeScript 0.01% Java 1.45%

glean's Introduction

CI Glean demo Docker image

Glean

Glean home | Documentation

Glean is a system for working with facts about source code. You can use it for:

  • Collecting and storing detailed information about code structure. Glean is designed around an efficient storage model that enables storing information about code at scale.

  • Querying information about code, to power tools and experiences from online IDE features to offline code analysis.

Status

Glean is pre-release software. There are many rough edges; there are limited language indexers available initially; the build system is not as smooth as we would like. However, we want to make it available for you to experiment with and contribute to.

We'd love to hear feedback! If you run into problems or have suggestions please file an issue.

Language coverage

There is currently full support for:

We also support the SCIP or LSIF code indexing formats, for:

Custom indexers for these languages exist but aren't in the open source release yet; we hope to make these available in the future:

  • Python
  • Java
  • Kotlin
  • Erlang
  • Thrift
  • Buck and Bazel
  • C#

License

Glean is licensed under a BSD LICENSE.

Contributing

We'd love to have your help developing Glean. Please submit pull requests on github, and note that we need contributors to sign the Contributor license agreement before we can accept your pull request.

Style guide: for Haskell code, we use hlint to check for style issues. Lines wrap at 80 columns. Code should be -Wall clean. For C++, code should compile with Clang or GCC.

How to contact the Glean team

Building

See Building Glean.

Docker demo

For demo of the react codebase with hyperlinks powered by glean run docker run -ti -p8888:8888 ghcr.io/facebookincubator/glean/demo

Try out on your own codebase with a .flowconfig by running docker run -ti -p8888:8888 -v __YOUR_CODE_DIR__:/glean_demo/code ghcr.io/facebookincubator/glean/demo

Play round using the glean binaries in a shell by running docker run -ti -p8888:8888 ghcr.io/facebookincubator/glean/demo shell

glean's People

Contributors

0x1eaf avatar abhishektiwari23 avatar adelavais avatar ahilger avatar alpmestan avatar antonk52 avatar bochko avatar donsbot avatar facebook-github-bot avatar gownta avatar iahs avatar jknoxville avatar kathryngraymeta avatar lpetre avatar malanka avatar mpark avatar nhawkes avatar pepeiborra avatar podtserkovskiy avatar r-barnes avatar rleshchinskiy avatar ruslanla avatar shodoco avatar simonmar avatar tdupes avatar williamtwilson avatar yfeldblum avatar yoney avatar zertosh avatar zsol avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

glean's Issues

[__1] unknown package: thrift-server (dependency of glean)

Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: glean-0.1.0.0 (user goal)
[__1] unknown package: thrift-server (dependency of glean)
[__1] fail (backjumping, conflict set: glean, thrift-server)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: glean, thrift-server

Provide make install

At the moment the Makefile does not provide a make install as it's customary for Makefiles. In particular, this would allow much easier packaging for most packaging systems (as they require install to a chroot before packaging). In addition, it helps to quickly move the right binaries to an easy accessible place.

C++ indexer support

This ticket will help us track and discuss the progress on getting the C++ indexer building, running and fully integrated, including the additional layer that derives more information from the "raw" facts that the indexer provides and the testing side.

WIP branch at: https://github.com/alpmestan/Glean/tree/cpp-indexer - not cleaned up at all, lots of superfluous changes in there, but this is where I'm going to push and extract reviewable patches from.

Binary distributions

Are there any plans of providing binary distributions? If so, what's the status there?

Support for C++.

README says:

Indexers for these languages exist but aren't in the open source release yet; we hope to make these available in the future:
...
C++

I'd like to know when C++ is available. Thanks!

Test failure on arm64 : SIGABRT raised in facebook::glean::rts::executeQuery() on runQuery for fact with wrong type

Around Feb 3, 'cabal test angle' started failing deterministically on arm64 builds. All other angle tests pass.

The query asks for facts of the wrong type, using a fact literal id. The id is a valid fact for a Cxx.Name, but we query for it as a Cxx.Type. This is exported to raise a catchable exception in Haskell with "fact has the wrong type" mentioned somewhere, by throwing a std::runtime_exception via https://github.com/facebookincubator/Glean/blob/main/glean/rts/error.cpp#L24

On ARM, now, we get a SIGABRT in the following runQuery:

  -- Literal fact ID with the wrong type
  r <- try $ runQuery_ env repo $ modify $ angle @Cxx.Type $
      "$cxx1.Type " <> factId (head names)
  assertBool "angle - fact id with wrong type" $
    case r of
      Left (SomeException x) -> "fact has the wrong type" `isInfixOf` show x
        -- it's actually a GleanFFIError when running locally,
        -- and probably a Thrift ApplicationError when running
        -- remotely. It should really be a BadQuery though.
      _ -> False

or as a standalone example:

  (r :: Either Glean.Types.Exception [Cxx.Type]) <- try $ runQuery_ env repo $ angle @Cxx.Type "$cxx1.Type 1026"

Reproduces in the shell on ARM too using the example.angle schema:

facts>  $1026 : example.Parent
fact has the wrong typeterminate called after throwing an instance of 'std::runtime_error'
terminate called recursively
*** Aborted at 1645421878 (Unix time, try 'date -d @1645421878') ***
*** Signal 6 (SIGABRT) (0x45f8) received by PID 17912 (pthread TID 0xffffa0dc04f0) (linux TID 17912) (maybe from PID 17912, UID 0) (code: -6), stack trace: ***
/root/.hsthrift/lib/libfolly.so.0.58.0-dev(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x47)[0xffffa4c4a4c7]
/root/.hsthrift/lib/libfolly.so.0.58.0-dev(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x47)[0xffffa4c4a4c7]
/lib/aarch64-linux-gnu/libc.so.6(+0xe7)[0xffffa14340e7]
(safe mode, symbolizer not available)
Aborted (core dumped)

Minimal test case:
https://gist.github.com/donsbot/08302e0ff4ba3c3d580129c04df67584

Correct execution (e.g. on x86) generates a raise / raiseError call which is then caught.

fact has the wrong type
Stack trace:
  facebook::glean::rts::raiseError(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > facebook::glean::rts::error<>(folly::Range<char const*>)
  facebook::glean::rts::Subroutine::execute(unsigned long const*) const

On ARM, though, our raiseError isn't caught, instead it gets rethrown.

terminate called after throwing an instance of 'std::runtime_error'
terminate called recursively

In gdb, we can see the trace, suggesting something is throwing when the first exception is raised. It's still the same execution path.

Thread 1 "barebones" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

And folly.git/folly/experimental/exception_tracer/ExceptionTracerLib.cpp gets entered twice.

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000ffffab3f0d68 in __GI_abort () at abort.c:79
#2  0x0000ffffac2af868 in __gnu_cxx::__verbose_terminate_handler() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#3  0x0000ffffac2ad21c in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x0000ffffac2ad280 in std::terminate() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#5  0x0000ffffac2ad5e0 in __cxa_rethrow () from /lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x0000ffffaeda3164 in __cxxabiv1::__cxa_rethrow ()
    at /tmp/fbcode_builder_getdeps-ZrootZgleanZGleanZhsthriftZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/experimental/exception_tracer/ExceptionTracerLib.cpp:119
#7  0x0000ffffac2af804 in __gnu_cxx::__verbose_terminate_handler() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#8  0x0000ffffac2ad21c in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#9  0x0000ffffac2ad280 in std::terminate() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#10 0x0000ffffac2ad574 in __cxa_throw () from /lib/aarch64-linux-gnu/libstdc++.so.6
#11 0x0000ffffaeda2e70 in __cxxabiv1::__cxa_throw (thrownException=0x41010cd0, type=0x297cac0 <typeinfo for std::runtime_error>, destructor=<optimized out>)
    at /tmp/fbcode_builder_getdeps-ZrootZgleanZGleanZhsthriftZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/experimental/exception_tracer/ExceptionTracerLib.cpp:106
#12 0x0000000001135bec in facebook::glean::rts::raiseError(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()

#13 0x0000000001129f64 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > facebook::glean::rts::error<>(folly::Range<char const*>) ()
#14 0x00000000011690e8 in facebook::glean::rts::Subroutine::execute(unsigned long const*) const ()
#15 0x000000000115d6cc in facebook::glean::rts::executeQuery(facebook::glean::rts::Inventory&, facebook::glean::rts::Define&, facebook::glean::rts::DefineOwnership*, facebook::glean::rts::Subroutine&, facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag>, std::shared_ptr<facebook::glean::rts::Subroutine>, folly::Optional<unsigned long>, folly::Optional<unsigned long>, folly::Optional<unsigned long>, facebook::glean::rts::Depth, std::unordered_set<facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag>, folly::hasher<facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag>, void>, std::equal_to<facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag> >, std::allocator<facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag> > >&, bool, folly::Optional<facebook::glean::thrift::internal::QueryCont>) ()
#16 0x0000000001142cb0 in glean_query_execute_compiled ()
#17 0x0000000000fb3dfc in c6RWQ_info$def ()

So something about how we catch this exception, or set it up. Or possibly clang/gcc differences? Thoughts?

Recursive Queries in Angle?

Is it possible to do recursive queries and if so what is the recommended way? For instance, in the schema in the example, is there an easy query that could list all the ancestor classes of "Goldfish" (in this case, it should give back "Fish" and "Pet" for example)?

New programming languages

Exist any map or checklist for adding support for new programming languages ? Example: I want to test it for Ruby and I would be interested in adding support.

glean throws error on basic usage with haskell-hie indexer

I am trying to use the haskell-hie indexer. I have a local build of glean and if I do:

../github/Glean-copy/dist-newstyle/build/x86_64-linux/ghc-8.10.2/glean-0.1.0.0/x/glean/build/glean/glean index --db temp-db/ haskell-hie dist-newstyle/

in a local project, then I get the following error:

glean: basic-thriftservice does not support Tier
CallStack (from HasCallStack):
  error, called at glean/util/Glean/Impl/ThriftService.hs:98:25 in glean-0.1.0.0-inplace-util:Glean.Impl.ThriftService

Any ideas what I might be doing wrong?

cabal test test:dbjanitor failing

Current ci failure is in test:dbjanitor. trace below

dons@debian:~/Glean$ cabal test test:dbjanitor
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - glean-0.1.0.0 (test:dbjanitor) (ephemeral targets)
Preprocessing test suite 'dbjanitor' for glean-0.1.0.0..
Building test suite 'dbjanitor' for glean-0.1.0.0..
Running 1 test suites...
Test suite dbjanitor: RUNNING...
Cases: 14  Tried: 0  Errors: 0  Failures: 0WARNING: Logging before InitGoogleLogging() is written to STDERR
I0801 10:16:11.352203 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.452033 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.549765 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.614281 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.686482 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.746145 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.829859 194470 Janitor.hs:87] running database janitor
I0801 10:16:11.829896 194470 Janitor.hs:88] Assigned shards: [NoSharding]
I0801 10:16:11.829903 194470 Janitor.hs:337] fetching restorable databases list
I0801 10:16:11.830024 194474 Delete.hs:82] test/0005: deleting
I0801 10:16:11.830471 194576 Delete.hs:82] test2/0006: deleting
I0801 10:16:11.831920 194472 Delete.hs:82] test/0002: deleting
I0801 10:16:11.832860 194474 Delete.hs:98] test/0005: deleted
I0801 10:16:11.832888 194474 Delete.hs:82] test/0004: deleting
I0801 10:16:11.833371 194576 Delete.hs:98] test2/0006: deleted
I0801 10:16:11.833611 194472 Delete.hs:98] test/0002: deleted
I0801 10:16:11.833714 194472 Delete.hs:82] test/0003: deleting
I0801 10:16:11.834358 194474 Delete.hs:98] test/0004: deleted
E0801 10:16:11.834650 194470 Exception.hs:134] test2/0006: UnknownDatabase {unknownDatabase_repo = Repo {repo_name = "test2", repo_hash = "0006"}, unknownDatabase_callStack = "CallStack (from HasCallStack):\n  throwSTM, called at glean/db/Glean/Database/Open.hs:218:29 in glean-0.1.0.0-inplace-db:Glean.Database.Open\n  withActiveDatabase, called at glean/db/Glean/Database/Open.hs:70:3 in glean-0.1.0.0-inplace-db:Glean.Database.Open\n  withOpenDatabase, called at glean/db/Glean/Database/Janitor.hs:233:11 in glean-0.1.0.0-inplace-db:Glean.Database.Janitor"}

thrift annotations include path changes broke build : thrift-compiler: ./thrift/annotation/cpp.thrift: openFile: does not exist

mkdir -p .build/def/codegen/thrift-glean-hs
for f in glean/github/if/fb303.thrift glean/github/if/fb303_core.thrift glean/if/glean.thrift glean/config/recipes/recipes.thrift glean/config/recipes/recipes.thrift glean/config/server/server_config.thrift glean/config/service.thrift glean/config/client/client_config.thrift; do
/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/thrift-compiler-0.1.0.0/x/thrift-compiler/build/thrift-compiler/thrift-compiler --hs $f -o .build/def/codegen/thrift-glean-hs/$(dirname $f);
done
thrift-compiler: ./thrift/annotation/cpp.thrift: openFile: does not exist (No such file or directory)

internal goes in a subdir, so do it separately

/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/thrift-compiler-0.1.0.0/x/thrift-compiler/build/thrift-compiler/thrift-compiler --hs glean/if/internal.thrift
-o .build/def/codegen/thrift-glean-hs/glean/if/internal
thrift-compiler: ./thrift/annotation/cpp.thrift: openFile: does not exist (No such file or directory)
make: *** [Makefile:249: thrift-glean-hs] Error 1

Accessing AST?

Do Angel or any toolkit in Glean provide ways to view the full AST (e.g of a function) after I get a queried fact?

What's the best way to build such pipeline? (e.g. accessing AST, walking it to mutate parts of existing code)

Fresh install seems broken

If I checkout Glean on a fresh machine and run ./install-deps.sh then I get the following

Using pinned rev 2459e172518a7b5d55292b0d8b741ff14def87ba for https://github.com/facebook/folly.git
Assessing folly...
Using pinned rev 2459e172518a7b5d55292b0d8b741ff14def87ba for https://github.com/facebook/folly.git
fatal: not a git repository (or any of the parent directories): .git
Command '['git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
!! Failed

Any ideas why this might be happening?

Question about batched writes

In the docs here you explain how to do batched writes:
https://glean.software/docs/write/#creating-a-database-using-the-command-line

but earlier in the document it says: "It is only possible to refer to ids from facts in the same file, if you are writing multiple files using glean write or via the sendJsonBatch API."

I am wondering then how does one refer to earlier facts in a later file or if it is no longer possible to refer to such facts (perhaps it is possible by something other than the id)? What is the currently recommended way to handle a case like this?

Clarifications on src.angle

I am trying to understand Glean's requirements for representing source locations. There are a couple of points that are not clear to me after reading src.angle:

  1. What's a column? Is it
    • a visual concept (based on the display column), which would take into account expansion of tabs and Unicode combining characters, or
    • an offset into the string content of the line, in which case is the offset counted in UTF-8 bytes, UTF-16 code units, or code points?
  2. How do byte spans work if a file uses \r\n as a line terminator? Do they assume \r\n has been normalized into \n or are they counted as two bytes?

Motivation: Document what kinds of facts you'd analyze.

The introductory documentation says that Glean is for analyzing "facts about source code", but does not give examples of those facts. Front-loading the introduction with examples of =questions it can usefully answer would help motivate a user. (I dug ten pages into the documentation, still don't know why I'd use it, and have lost interest.)

free(): invalid pointer in glean test:angle on clang-11

The clang ci job started failing a couple of days ago in cabal test glean:angle with a core dump. clang-11 on ubuntu 20. (previously we were using ubuntu 22).

Stack trace below:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff39f1859 in __GI_abort () at abort.c:79
#2  0x00007ffff3a5c29e in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff3b86298 "%s\n")
    at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007ffff3a6432c in malloc_printerr (str=str@entry=0x7ffff3b844c1 "free(): invalid pointer") at malloc.c:5347
#4  0x00007ffff3a65b5c in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:4173
#5  0x000000000101a1ea in facebook::glean::thrift::internal::QueryCont::~QueryCont() ()
#6  0x0000000000fe0501 in glean_query_execute_compiled ()
#7  0x00000000025cb058 in MainCapability ()
#8  0x00000000025cb000 in last_free_capability ()

Fixing the demo

Hey folks, Glean seems really cool. I want to use this as a core component of my project but am unable to even try it out. The demo isn't working and attempting to build this locally also fails. Are there plans to keep the version of Glean available publicly fully functional?

Mac OS build

Hey folks,

Glean is looking like a very interesting project, is anybody have a instruction for compilation for mac os?

currently, I stuck on the compilation of fbthrift

[ 15%] Linking CXX shared library ../../lib/libcompiler_base.dylib
Undefined symbols for architecture x86_64:
  "apache::thrift::compiler::gen::cpp::find_ref_type(apache::thrift::compiler::t_field const&)", referenced from:
      apache::thrift::compiler::(anonymous namespace)::validate_boxed_field_attributes(apache::thrift::compiler::diagnostic_context&, apache::thrift::compiler::t_field const&) in standard_validator.cc.o
  "apache::thrift::compiler::cpp2::has_ref_annotation(apache::thrift::compiler::t_field const&)", referenced from:
      apache::thrift::compiler::(anonymous namespace)::validate_ref_field_attributes(apache::thrift::compiler::diagnostic_context&, apache::thrift::compiler::t_field const&) in standard_validator.cc.o
  "apache::thrift::compiler::cpp2::is_mixin(apache::thrift::compiler::t_field const&)", referenced from:
      apache::thrift::compiler::(anonymous namespace)::validate_mixin_field_attributes(apache::thrift::compiler::diagnostic_context&, apache::thrift::compiler::t_field const&) in standard_validator.cc.o
      apache::thrift::compiler::(anonymous namespace)::get_mixin_type(apache::thrift::compiler::t_field const&) in standard_validator.cc.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [lib/libcompiler_base.1.0.0.dylib] Error 1
make[1]: *** [thrift/compiler/CMakeFiles/compiler_base.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

How to run haskell-hie from cli

Just trying out the haskell-hie indexer. I feel like something like this should work but no luck:

# in the tests dir
$ pwd
/home/dons/Glean/glean/lang/haskell/tests/code

# make some .hie files
$ ghc --make -fwrite-ide-info B.hs
Loaded package environment from /home/dons/.ghc/x86_64-linux-8.10.7/environments/default
[1 of 2] Compiling A                ( A.hs, A.o )
[2 of 2] Compiling B                ( B.hs, B.o )

$ ls
A.hi  A.hie  A.hs  A.o  B.hi  B.hie  B.hs  B.o

$ cd
$ cd Glean/
$ glean-shell
Glean Shell, built on 2022-06-09 02:22:49.685202174 UTC, from rev b5af6c875a448322c83bfa50c2148428450e735a
type :help for help.

> :index haskell-hie Glean/glean/lang/haskell/tests/code
I0609 04:20:54.879848 2594231 DefaultMain.hs:44] BEGIN buildXrefMapFiles
I0609 04:20:54.880440 2594231 Builder.hs:93] Path of hiedb = /tmp/extra-file-46386356845652-2594231-0
I0609 04:20:54.880554 2594231 Builder.hs:94] Repo path = .
I0609 04:20:54.880653 2594231 Builder.hs:453] BEGIN getDefsVertices
I0609 04:20:54.880975 2594231 Builder.hs:453] END getDefsVertices
I0609 04:20:54.881105 2594231 Builder.hs:99] Number of vertices in the graph = 0
I0609 04:20:54.881202 2594231 Builder.hs:155] BEGIN mkFileLinesMap
I0609 04:20:54.881250 2594231 Builder.hs:155] END mkFileLinesMap
I0609 04:20:54.881429 2594231 DefaultMain.hs:44] FAILED buildXrefMapFiles: File line map is empty!!

CallStack (from HasCallStack):
  error, called at glean/lang/haskell/HieDBIndexer/Builder.hs:133:7 in main:HieDBIndexer.Builder
hiedb-indexer: File line map is empty!!
CallStack (from HasCallStack):
  error, called at glean/lang/haskell/HieDBIndexer/Builder.hs:133:7 in main:HieDBIndexer.Builder
E0609 04:20:54.888273 2594178 Work.hs:281] code/5: database failed in parcel 0 of task '': callCommand: /home/dons/Glean/dist-newstyle/build/x86_64-linux/ghc-8.10.7/glean-0.1.0.0/x/hiedb-indexer/build/hiedb-indexer/hiedb-indexer '--service' 'localhost:41303' '--repo-name' 'code' '--repo-hash' '5' '--repo-path' '.' '--dont-create-db' 'Glean/glean/lang/haskell/tests/code' '' (exit 1): failed
*** Exception: callCommand: /home/dons/Glean/dist-newstyle/build/x86_64-linux/ghc-8.10.7/glean-0.1.0.0/x/hiedb-indexer/build/hiedb-indexer/hiedb-indexer '--service' 'localhost:41303' '--repo-name' 'code' '--repo-hash' '5' '--repo-path' '.' '--dont-create-db' 'Glean/glean/lang/haskell/tests/code' '' (exit 1): failed

Add scip/codemarkup language support for Python and C#

We have indexers for Python and C# now via scip. To make their data more accessible, they should be exported to the codemarkup.angle layer of Glean schemas.

Regenerate the schema with make, confirm that it works by writing regression tests for codemarkup.* queries.

Check and confirm glass builds. cabal build glass. Add any missing cases until it compiles.

Non-clang CI is red

Failing in a specific test in glean-clang-codemarkup, it's reproducible:

I1010 17:36:02.003545 2335644 index.cpp:679] 0: 1/1 [1] /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4/test.cpp
I1010 17:36:02.003616 2335644 index.cpp:683] 0: fact buffer: 0B (0) cache: 0B (0) lifetime: 0B (0)
terminate called after throwing an instance of '(anonymous namespace)::FatalLLVMError'
  what():  Cannot chdir into "
*** Aborted at 1696919762 (Unix time, try 'date -d @1696919762') ***
*** Signal 6 (SIGABRT) (0x3e80023a39c) received by PID 2335644 (pthread TID 0xffff83c27010) (linux TID 2335644) (maybe from PID 2335644, UID 1000) (code: -6), stack trace: ***

terminate called after throwing an instance of '(anonymous namespace)::FatalLLVMError'
what(): Cannot chdir into "

Try with -v to see this message. eg.

$ /home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-0.1.0.0/t/clang-test-codemarkup/build/clang-test-codemarkup/clang-test-codemarkup  -v --only xrefs/macro4
Cases: 1  Tried: 0  Errors: 0  Failures: 0I1010 17:37:37.259212 2335667 Open.hs:448] test/xrefs_macro4: opening
I1010 17:37:37.360203 2335667 ownership.cpp:220] loadOwnershipSets loaded 0 sets, 0 bytes
I1010 17:37:37.360294 2335667 Open.hs:454] test/xrefs_macro4: opened
I1010 17:37:37.548393 2335667 Open.hs:456] test/xrefs_macro4: schema has 951 predicates
I1010 17:37:37.597450 2335667 Work.hs:234] scheduling ""
I1010 17:37:37.597479 2335667 Create.hs:161] test/xrefs_macro4: created
I1010 17:37:37.986143 2335692 index.cpp:679] 0: 1/1 [1] /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4/test.cpp
I1010 17:37:37.986188 2335692 index.cpp:683] 0: fact buffer: 0B (0) cache: 0B (0) lifetime: 0B (0)
terminate called after throwing an instance of '(anonymous namespace)::FatalLLVMError'
  what():  Cannot chdir into "
*** Aborted at 1696919857 (Unix time, try 'date -d @1696919857') ***
*** Signal 6 (SIGABRT) (0x3e80023a3cc) received by PID 2335692 (pthread TID 0xffff8777c010) (linux TID 2335692) (maybe from PID 2335692, UID 1000) (code: -6), stack trace: ***
/home/dons/.hsthrift/lib/libfolly.so.0.58.0-dev(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x37)[0xffff8bde750b]
/home/dons/.hsthrift/lib/libfolly.so.0.58.0-dev(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x37)[0xffff8bde750b]
/home/dons/.hsthrift/lib/libfolly.so.0.58.0-dev(+0x295887)[0xffff8bde5887]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffff9080f7c0]
/lib/aarch64-linux-gnu/libc.so.6(gsignal+0xdb)[0xffff898542ab]
/lib/aarch64-linux-gnu/libc.so.6(abort+0x107)[0xffff89840e9f]
/lib/aarch64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x187)[0xffff8b92b237]
/lib/aarch64-linux-gnu/libstdc++.so.6(+0x9ed4b)[0xffff8b928d4b]
/lib/aarch64-linux-gnu/libstdc++.so.6(_ZSt9terminatev+0xf)[0xffff8b928daf]
/lib/aarch64-linux-gnu/libstdc++.so.6(__cxa_throw+0x63)[0xffff8b9290a3]
/home/dons/.hsthrift/lib/libfolly.so.0.58.0-dev(__cxa_throw+0x1af)[0xffff8bdc54e3]
/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-clang-0.1.0.0/build/clang-index/clang-index[0x437b13]
/lib/aarch64-linux-gnu/libLLVM-11.so.1(_ZN4llvm18report_fatal_errorERKNS_5TwineEb+0x107)[0xffff8cb3214b]
/lib/aarch64-linux-gnu/libLLVM-11.so.1(_ZN4llvm18report_fatal_errorERKNS_5TwineEb+0x107)[0xffff8cb3214b]
(safe mode, symbolizer not available)
I1010 17:37:37.989368 2335667 Work.hs:504] workFinished Work {work_repo = Repo {repo_name = "test", repo_hash = "xrefs_macro4"}, work_task = "", work_parcelIndex = 0, work_parcelCount = 0, work_handle = ""}
I1010 17:37:37.989514 2335667 Work.hs:505] workFinished_outcome Outcome_failure (Failure {failure_message = "/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-clang-0.1.0.0/build/clang-index/clang-index -cdb_dir /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4 -cdb_target all -root /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4 --inventory /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/inventory.data -logtostderr -dump /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/indexer--6.data --work_file /tmp/2335667-3.glean-worklist --worker_index -6 --worker_count 1 returned exit code -6\nCallStack (from HasCallStack):\n  error, called at glean/lang/clang/Glean/Indexer/Cpp.hs:188:34 in glean-0.1.0.0-inplace-indexers:Glean.Indexer.Cpp"})
E1010 17:37:37.989550 2335667 Work.hs:279] test/xrefs_macro4: database failed in parcel 0 of task '': /home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-clang-0.1.0.0/build/clang-index/clang-index -cdb_dir /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4 -cdb_target all -root /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4 --inventory /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/inventory.data -logtostderr -dump /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/indexer--6.data --work_file /tmp/2335667-3.glean-worklist --worker_index -6 --worker_count 1 returned exit code -6
CallStack (from HasCallStack):
  error, called at glean/lang/clang/Glean/Indexer/Cpp.hs:188:34 in glean-0.1.0.0-inplace-indexers:Glean.Indexer.Cpp
I1010 17:37:37.989675 2335667 Stats.hs:223] mut_lat: - [-] mut_thp: - [-] ded_thp: - [-] dup_thp: - [-] rnm_thp: - [-] cmt_thp: - [-] ibk_mis: - [-] tbi_mis: - [-] fbi_mis: - [-] lch_mem: 0B lch_cnt: 0
### Error in:   0:0:xrefs/macro4
/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-clang-0.1.0.0/build/clang-index/clang-index -cdb_dir /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4 -cdb_target all -root /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4 --inventory /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/inventory.data -logtostderr -dump /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/indexer--6.data --work_file /tmp/2335667-3.glean-worklist --worker_index -6 --worker_count 1 returned exit code -6
CallStack (from HasCallStack):
  error, called at glean/lang/clang/Glean/Indexer/Cpp.hs:188:34 in glean-0.1.0.0-inplace-indexers:Glean.Indexer.Cpp
Cases: 1  Tried: 1  Errors: 1  Failures: 0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.