facebookincubator / glean Goto Github PK

System for collecting, deriving and working with facts about source code.

License: Other

Makefile 0.19% Haskell 38.85% C++ 8.96% Python 0.43% Thrift 1.60% C 0.13% Lex 0.08% Yacc 0.15% Shell 0.19% Objective-C 0.14% Perl 0.09% JavaScript 0.13% CSS 0.03% Dockerfile 0.04% Objective-C++ 0.02% Hack 45.93% Rust 1.37% Go 0.02% TypeScript 0.02% Java 1.63%

glean's People

Contributors

Stargazers

Watchers

glean's Issues

Support for C++.

README says:

Indexers for these languages exist but aren't in the open source release yet; we hope to make these available in the future:
...
C++

I'd like to know when C++ is available. Thanks!

cabal test test:dbjanitor failing

Current ci failure is in test:dbjanitor. trace below

dons@debian:~/Glean$ cabal test test:dbjanitor
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - glean-0.1.0.0 (test:dbjanitor) (ephemeral targets)
Preprocessing test suite 'dbjanitor' for glean-0.1.0.0..
Building test suite 'dbjanitor' for glean-0.1.0.0..
Running 1 test suites...
Test suite dbjanitor: RUNNING...
Cases: 14  Tried: 0  Errors: 0  Failures: 0WARNING: Logging before InitGoogleLogging() is written to STDERR
I0801 10:16:11.352203 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.452033 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.549765 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.614281 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.686482 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.746145 194470 rocksdb.cpp:608] loadOwnershipSets loaded 0 sets, 0 bytes
I0801 10:16:11.829859 194470 Janitor.hs:87] running database janitor
I0801 10:16:11.829896 194470 Janitor.hs:88] Assigned shards: [NoSharding]
I0801 10:16:11.829903 194470 Janitor.hs:337] fetching restorable databases list
I0801 10:16:11.830024 194474 Delete.hs:82] test/0005: deleting
I0801 10:16:11.830471 194576 Delete.hs:82] test2/0006: deleting
I0801 10:16:11.831920 194472 Delete.hs:82] test/0002: deleting
I0801 10:16:11.832860 194474 Delete.hs:98] test/0005: deleted
I0801 10:16:11.832888 194474 Delete.hs:82] test/0004: deleting
I0801 10:16:11.833371 194576 Delete.hs:98] test2/0006: deleted
I0801 10:16:11.833611 194472 Delete.hs:98] test/0002: deleted
I0801 10:16:11.833714 194472 Delete.hs:82] test/0003: deleting
I0801 10:16:11.834358 194474 Delete.hs:98] test/0004: deleted
E0801 10:16:11.834650 194470 Exception.hs:134] test2/0006: UnknownDatabase {unknownDatabase_repo = Repo {repo_name = "test2", repo_hash = "0006"}, unknownDatabase_callStack = "CallStack (from HasCallStack):\n  throwSTM, called at glean/db/Glean/Database/Open.hs:218:29 in glean-0.1.0.0-inplace-db:Glean.Database.Open\n  withActiveDatabase, called at glean/db/Glean/Database/Open.hs:70:3 in glean-0.1.0.0-inplace-db:Glean.Database.Open\n  withOpenDatabase, called at glean/db/Glean/Database/Janitor.hs:233:11 in glean-0.1.0.0-inplace-db:Glean.Database.Janitor"}

thrift annotations include path changes broke build : thrift-compiler: ./thrift/annotation/cpp.thrift: openFile: does not exist

mkdir -p .build/def/codegen/thrift-glean-hs
for f in glean/github/if/fb303.thrift glean/github/if/fb303_core.thrift glean/if/glean.thrift glean/config/recipes/recipes.thrift glean/config/recipes/recipes.thrift glean/config/server/server_config.thrift glean/config/service.thrift glean/config/client/client_config.thrift; do
/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/thrift-compiler-0.1.0.0/x/thrift-compiler/build/thrift-compiler/thrift-compiler --hs $f -o .build/def/codegen/thrift-glean-hs/$(dirname $f);
done
thrift-compiler: ./thrift/annotation/cpp.thrift: openFile: does not exist (No such file or directory)

internal goes in a subdir, so do it separately

/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/thrift-compiler-0.1.0.0/x/thrift-compiler/build/thrift-compiler/thrift-compiler --hs glean/if/internal.thrift
-o .build/def/codegen/thrift-glean-hs/glean/if/internal
thrift-compiler: ./thrift/annotation/cpp.thrift: openFile: does not exist (No such file or directory)
make: *** [Makefile:249: thrift-glean-hs] Error 1

free(): invalid pointer in glean test:angle on clang-11

The clang ci job started failing a couple of days ago in cabal test glean:angle with a core dump. clang-11 on ubuntu 20. (previously we were using ubuntu 22).

Stack trace below:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff39f1859 in __GI_abort () at abort.c:79
#2  0x00007ffff3a5c29e in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff3b86298 "%s\n")
    at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007ffff3a6432c in malloc_printerr (str=str@entry=0x7ffff3b844c1 "free(): invalid pointer") at malloc.c:5347
#4  0x00007ffff3a65b5c in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:4173
#5  0x000000000101a1ea in facebook::glean::thrift::internal::QueryCont::~QueryCont() ()
#6  0x0000000000fe0501 in glean_query_execute_compiled ()
#7  0x00000000025cb058 in MainCapability ()
#8  0x00000000025cb000 in last_free_capability ()

Fresh install seems broken

If I checkout Glean on a fresh machine and run ./install-deps.sh then I get the following

Using pinned rev 2459e172518a7b5d55292b0d8b741ff14def87ba for https://github.com/facebook/folly.git
Assessing folly...
Using pinned rev 2459e172518a7b5d55292b0d8b741ff14def87ba for https://github.com/facebook/folly.git
fatal: not a git repository (or any of the parent directories): .git
Command '['git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
!! Failed

Any ideas why this might be happening?

Binary distributions

Are there any plans of providing binary distributions? If so, what's the status there?

Provide make install

At the moment the Makefile does not provide a make install as it's customary for Makefiles. In particular, this would allow much easier packaging for most packaging systems (as they require install to a chroot before packaging). In addition, it helps to quickly move the right binaries to an easy accessible place.

Update scip-typescript support

Update CI steps, regenerate test output, check indexer is working for TypeScript.

https://github.com/sourcegraph/scip-typescript

Test failure on arm64 : SIGABRT raised in facebook::glean::rts::executeQuery() on runQuery for fact with wrong type

Around Feb 3, 'cabal test angle' started failing deterministically on arm64 builds. All other angle tests pass.

The query asks for facts of the wrong type, using a fact literal id. The id is a valid fact for a Cxx.Name, but we query for it as a Cxx.Type. This is exported to raise a catchable exception in Haskell with "fact has the wrong type" mentioned somewhere, by throwing a std::runtime_exception via https://github.com/facebookincubator/Glean/blob/main/glean/rts/error.cpp#L24

On ARM, now, we get a SIGABRT in the following runQuery:

  -- Literal fact ID with the wrong type
  r <- try $ runQuery_ env repo $ modify $ angle @Cxx.Type $
      "$cxx1.Type " <> factId (head names)
  assertBool "angle - fact id with wrong type" $
    case r of
      Left (SomeException x) -> "fact has the wrong type" `isInfixOf` show x
        -- it's actually a GleanFFIError when running locally,
        -- and probably a Thrift ApplicationError when running
        -- remotely. It should really be a BadQuery though.
      _ -> False

or as a standalone example:

  (r :: Either Glean.Types.Exception [Cxx.Type]) <- try $ runQuery_ env repo $ angle @Cxx.Type "$cxx1.Type 1026"

Reproduces in the shell on ARM too using the example.angle schema:

facts>  $1026 : example.Parent
fact has the wrong typeterminate called after throwing an instance of 'std::runtime_error'
terminate called recursively
*** Aborted at 1645421878 (Unix time, try 'date -d @1645421878') ***
*** Signal 6 (SIGABRT) (0x45f8) received by PID 17912 (pthread TID 0xffffa0dc04f0) (linux TID 17912) (maybe from PID 17912, UID 0) (code: -6), stack trace: ***
/root/.hsthrift/lib/libfolly.so.0.58.0-dev(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x47)[0xffffa4c4a4c7]
/root/.hsthrift/lib/libfolly.so.0.58.0-dev(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x47)[0xffffa4c4a4c7]
/lib/aarch64-linux-gnu/libc.so.6(+0xe7)[0xffffa14340e7]
(safe mode, symbolizer not available)
Aborted (core dumped)

Minimal test case:
https://gist.github.com/donsbot/08302e0ff4ba3c3d580129c04df67584

Correct execution (e.g. on x86) generates a raise / raiseError call which is then caught.

fact has the wrong type
Stack trace:
  facebook::glean::rts::raiseError(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > facebook::glean::rts::error<>(folly::Range<char const*>)
  facebook::glean::rts::Subroutine::execute(unsigned long const*) const

On ARM, though, our raiseError isn't caught, instead it gets rethrown.

terminate called after throwing an instance of 'std::runtime_error'
terminate called recursively

In gdb, we can see the trace, suggesting something is throwing when the first exception is raised. It's still the same execution path.

Thread 1 "barebones" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

And folly.git/folly/experimental/exception_tracer/ExceptionTracerLib.cpp gets entered twice.

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000ffffab3f0d68 in __GI_abort () at abort.c:79
#2  0x0000ffffac2af868 in __gnu_cxx::__verbose_terminate_handler() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#3  0x0000ffffac2ad21c in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x0000ffffac2ad280 in std::terminate() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#5  0x0000ffffac2ad5e0 in __cxa_rethrow () from /lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x0000ffffaeda3164 in __cxxabiv1::__cxa_rethrow ()
    at /tmp/fbcode_builder_getdeps-ZrootZgleanZGleanZhsthriftZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/experimental/exception_tracer/ExceptionTracerLib.cpp:119
#7  0x0000ffffac2af804 in __gnu_cxx::__verbose_terminate_handler() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#8  0x0000ffffac2ad21c in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#9  0x0000ffffac2ad280 in std::terminate() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#10 0x0000ffffac2ad574 in __cxa_throw () from /lib/aarch64-linux-gnu/libstdc++.so.6
#11 0x0000ffffaeda2e70 in __cxxabiv1::__cxa_throw (thrownException=0x41010cd0, type=0x297cac0 <typeinfo for std::runtime_error>, destructor=<optimized out>)
    at /tmp/fbcode_builder_getdeps-ZrootZgleanZGleanZhsthriftZbuildZfbcode_builder-root/repos/github.com-facebook-folly.git/folly/experimental/exception_tracer/ExceptionTracerLib.cpp:106
#12 0x0000000001135bec in facebook::glean::rts::raiseError(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()

#13 0x0000000001129f64 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > facebook::glean::rts::error<>(folly::Range<char const*>) ()
#14 0x00000000011690e8 in facebook::glean::rts::Subroutine::execute(unsigned long const*) const ()
#15 0x000000000115d6cc in facebook::glean::rts::executeQuery(facebook::glean::rts::Inventory&, facebook::glean::rts::Define&, facebook::glean::rts::DefineOwnership*, facebook::glean::rts::Subroutine&, facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag>, std::shared_ptr<facebook::glean::rts::Subroutine>, folly::Optional<unsigned long>, folly::Optional<unsigned long>, folly::Optional<unsigned long>, facebook::glean::rts::Depth, std::unordered_set<facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag>, folly::hasher<facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag>, void>, std::equal_to<facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag> >, std::allocator<facebook::glean::rts::WordId<facebook::glean::rts::Pid_tag> > >&, bool, folly::Optional<facebook::glean::thrift::internal::QueryCont>) ()
#16 0x0000000001142cb0 in glean_query_execute_compiled ()
#17 0x0000000000fb3dfc in c6RWQ_info$def ()

So something about how we catch this exception, or set it up. Or possibly clang/gcc differences? Thoughts?

How to run haskell-hie from cli

Just trying out the haskell-hie indexer. I feel like something like this should work but no luck:

# in the tests dir
$ pwd
/home/dons/Glean/glean/lang/haskell/tests/code

# make some .hie files
$ ghc --make -fwrite-ide-info B.hs
Loaded package environment from /home/dons/.ghc/x86_64-linux-8.10.7/environments/default
[1 of 2] Compiling A                ( A.hs, A.o )
[2 of 2] Compiling B                ( B.hs, B.o )

$ ls
A.hi  A.hie  A.hs  A.o  B.hi  B.hie  B.hs  B.o

$ cd
$ cd Glean/
$ glean-shell
Glean Shell, built on 2022-06-09 02:22:49.685202174 UTC, from rev b5af6c875a448322c83bfa50c2148428450e735a
type :help for help.

> :index haskell-hie Glean/glean/lang/haskell/tests/code
I0609 04:20:54.879848 2594231 DefaultMain.hs:44] BEGIN buildXrefMapFiles
I0609 04:20:54.880440 2594231 Builder.hs:93] Path of hiedb = /tmp/extra-file-46386356845652-2594231-0
I0609 04:20:54.880554 2594231 Builder.hs:94] Repo path = .
I0609 04:20:54.880653 2594231 Builder.hs:453] BEGIN getDefsVertices
I0609 04:20:54.880975 2594231 Builder.hs:453] END getDefsVertices
I0609 04:20:54.881105 2594231 Builder.hs:99] Number of vertices in the graph = 0
I0609 04:20:54.881202 2594231 Builder.hs:155] BEGIN mkFileLinesMap
I0609 04:20:54.881250 2594231 Builder.hs:155] END mkFileLinesMap
I0609 04:20:54.881429 2594231 DefaultMain.hs:44] FAILED buildXrefMapFiles: File line map is empty!!

CallStack (from HasCallStack):
  error, called at glean/lang/haskell/HieDBIndexer/Builder.hs:133:7 in main:HieDBIndexer.Builder
hiedb-indexer: File line map is empty!!
CallStack (from HasCallStack):
  error, called at glean/lang/haskell/HieDBIndexer/Builder.hs:133:7 in main:HieDBIndexer.Builder
E0609 04:20:54.888273 2594178 Work.hs:281] code/5: database failed in parcel 0 of task '': callCommand: /home/dons/Glean/dist-newstyle/build/x86_64-linux/ghc-8.10.7/glean-0.1.0.0/x/hiedb-indexer/build/hiedb-indexer/hiedb-indexer '--service' 'localhost:41303' '--repo-name' 'code' '--repo-hash' '5' '--repo-path' '.' '--dont-create-db' 'Glean/glean/lang/haskell/tests/code' '' (exit 1): failed
*** Exception: callCommand: /home/dons/Glean/dist-newstyle/build/x86_64-linux/ghc-8.10.7/glean-0.1.0.0/x/hiedb-indexer/build/hiedb-indexer/hiedb-indexer '--service' 'localhost:41303' '--repo-name' 'code' '--repo-hash' '5' '--repo-path' '.' '--dont-create-db' 'Glean/glean/lang/haskell/tests/code' '' (exit 1): failed

Fixing the demo

Hey folks, Glean seems really cool. I want to use this as a core component of my project but am unable to even try it out. The demo isn't working and attempting to build this locally also fails. Are there plans to keep the version of Glean available publicly fully functional?

Add scip-python support

We support SCIP generically via glean-encode-scip and the scip.angle protocol. Current indexers supported are:

rust-analyzer (full, production)
typescript
go

This task is to add Python indexing support via SCIP using the https://github.com/sourcegraph/scip-python indexer.

Non-clang CI is red

Failing in a specific test in glean-clang-codemarkup, it's reproducible:

I1010 17:36:02.003545 2335644 index.cpp:679] 0: 1/1 [1] /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4/test.cpp
I1010 17:36:02.003616 2335644 index.cpp:683] 0: fact buffer: 0B (0) cache: 0B (0) lifetime: 0B (0)
terminate called after throwing an instance of '(anonymous namespace)::FatalLLVMError'
  what():  Cannot chdir into "
*** Aborted at 1696919762 (Unix time, try 'date -d @1696919762') ***
*** Signal 6 (SIGABRT) (0x3e80023a39c) received by PID 2335644 (pthread TID 0xffff83c27010) (linux TID 2335644) (maybe from PID 2335644, UID 1000) (code: -6), stack trace: ***

terminate called after throwing an instance of '(anonymous namespace)::FatalLLVMError'
what(): Cannot chdir into "

Try with -v to see this message. eg.

$ /home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-0.1.0.0/t/clang-test-codemarkup/build/clang-test-codemarkup/clang-test-codemarkup  -v --only xrefs/macro4
Cases: 1  Tried: 0  Errors: 0  Failures: 0I1010 17:37:37.259212 2335667 Open.hs:448] test/xrefs_macro4: opening
I1010 17:37:37.360203 2335667 ownership.cpp:220] loadOwnershipSets loaded 0 sets, 0 bytes
I1010 17:37:37.360294 2335667 Open.hs:454] test/xrefs_macro4: opened
I1010 17:37:37.548393 2335667 Open.hs:456] test/xrefs_macro4: schema has 951 predicates
I1010 17:37:37.597450 2335667 Work.hs:234] scheduling ""
I1010 17:37:37.597479 2335667 Create.hs:161] test/xrefs_macro4: created
I1010 17:37:37.986143 2335692 index.cpp:679] 0: 1/1 [1] /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4/test.cpp
I1010 17:37:37.986188 2335692 index.cpp:683] 0: fact buffer: 0B (0) cache: 0B (0) lifetime: 0B (0)
terminate called after throwing an instance of '(anonymous namespace)::FatalLLVMError'
  what():  Cannot chdir into "
*** Aborted at 1696919857 (Unix time, try 'date -d @1696919857') ***
*** Signal 6 (SIGABRT) (0x3e80023a3cc) received by PID 2335692 (pthread TID 0xffff8777c010) (linux TID 2335692) (maybe from PID 2335692, UID 1000) (code: -6), stack trace: ***
/home/dons/.hsthrift/lib/libfolly.so.0.58.0-dev(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x37)[0xffff8bde750b]
/home/dons/.hsthrift/lib/libfolly.so.0.58.0-dev(_ZN5folly10symbolizer21SafeStackTracePrinter15printStackTraceEb+0x37)[0xffff8bde750b]
/home/dons/.hsthrift/lib/libfolly.so.0.58.0-dev(+0x295887)[0xffff8bde5887]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffff9080f7c0]
/lib/aarch64-linux-gnu/libc.so.6(gsignal+0xdb)[0xffff898542ab]
/lib/aarch64-linux-gnu/libc.so.6(abort+0x107)[0xffff89840e9f]
/lib/aarch64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x187)[0xffff8b92b237]
/lib/aarch64-linux-gnu/libstdc++.so.6(+0x9ed4b)[0xffff8b928d4b]
/lib/aarch64-linux-gnu/libstdc++.so.6(_ZSt9terminatev+0xf)[0xffff8b928daf]
/lib/aarch64-linux-gnu/libstdc++.so.6(__cxa_throw+0x63)[0xffff8b9290a3]
/home/dons/.hsthrift/lib/libfolly.so.0.58.0-dev(__cxa_throw+0x1af)[0xffff8bdc54e3]
/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-clang-0.1.0.0/build/clang-index/clang-index[0x437b13]
/lib/aarch64-linux-gnu/libLLVM-11.so.1(_ZN4llvm18report_fatal_errorERKNS_5TwineEb+0x107)[0xffff8cb3214b]
/lib/aarch64-linux-gnu/libLLVM-11.so.1(_ZN4llvm18report_fatal_errorERKNS_5TwineEb+0x107)[0xffff8cb3214b]
(safe mode, symbolizer not available)
I1010 17:37:37.989368 2335667 Work.hs:504] workFinished Work {work_repo = Repo {repo_name = "test", repo_hash = "xrefs_macro4"}, work_task = "", work_parcelIndex = 0, work_parcelCount = 0, work_handle = ""}
I1010 17:37:37.989514 2335667 Work.hs:505] workFinished_outcome Outcome_failure (Failure {failure_message = "/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-clang-0.1.0.0/build/clang-index/clang-index -cdb_dir /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4 -cdb_target all -root /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4 --inventory /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/inventory.data -logtostderr -dump /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/indexer--6.data --work_file /tmp/2335667-3.glean-worklist --worker_index -6 --worker_count 1 returned exit code -6\nCallStack (from HasCallStack):\n  error, called at glean/lang/clang/Glean/Indexer/Cpp.hs:188:34 in glean-0.1.0.0-inplace-indexers:Glean.Indexer.Cpp"})
E1010 17:37:37.989550 2335667 Work.hs:279] test/xrefs_macro4: database failed in parcel 0 of task '': /home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-clang-0.1.0.0/build/clang-index/clang-index -cdb_dir /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4 -cdb_target all -root /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4 --inventory /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/inventory.data -logtostderr -dump /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/indexer--6.data --work_file /tmp/2335667-3.glean-worklist --worker_index -6 --worker_count 1 returned exit code -6
CallStack (from HasCallStack):
  error, called at glean/lang/clang/Glean/Indexer/Cpp.hs:188:34 in glean-0.1.0.0-inplace-indexers:Glean.Indexer.Cpp
I1010 17:37:37.989675 2335667 Stats.hs:223] mut_lat: - [-] mut_thp: - [-] ded_thp: - [-] dup_thp: - [-] rnm_thp: - [-] cmt_thp: - [-] ibk_mis: - [-] tbi_mis: - [-] fbi_mis: - [-] lch_mem: 0B lch_cnt: 0
### Error in:   0:0:xrefs/macro4
/home/dons/Glean/dist-newstyle/build/aarch64-linux/ghc-8.10.7/glean-clang-0.1.0.0/build/clang-index/clang-index -cdb_dir /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4 -cdb_target all -root /home/dons/Glean/glean/lang/codemarkup/tests/clang/xrefs/macro4 --inventory /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/inventory.data -logtostderr -dump /tmp/glean-regression-d60ee8b365ffdb5a/xrefs/macro4/indexer--6.data --work_file /tmp/2335667-3.glean-worklist --worker_index -6 --worker_count 1 returned exit code -6
CallStack (from HasCallStack):
  error, called at glean/lang/clang/Glean/Indexer/Cpp.hs:188:34 in glean-0.1.0.0-inplace-indexers:Glean.Indexer.Cpp
Cases: 1  Tried: 1  Errors: 1  Failures: 0

Motivation: Document what kinds of facts you'd analyze.

The introductory documentation says that Glean is for analyzing "facts about source code", but does not give examples of those facts. Front-loading the introduction with examples of =questions it can usefully answer would help motivate a user. (I dug ten pages into the documentation, still don't know why I'd use it, and have lost interest.)

Accessing AST?

Do Angel or any toolkit in Glean provide ways to view the full AST (e.g of a function) after I get a queried fact?

What's the best way to build such pipeline? (e.g. accessing AST, walking it to mutate parts of existing code)

C++ indexer support

This ticket will help us track and discuss the progress on getting the C++ indexer building, running and fully integrated, including the additional layer that derives more information from the "raw" facts that the indexer provides and the testing side.

WIP branch at: https://github.com/alpmestan/Glean/tree/cpp-indexer - not cleaned up at all, lots of superfluous changes in there, but this is where I'm going to push and extract reviewable patches from.

[__1] unknown package: thrift-server (dependency of glean)

Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: glean-0.1.0.0 (user goal)
[__1] unknown package: thrift-server (dependency of glean)
[__1] fail (backjumping, conflict set: glean, thrift-server)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: glean, thrift-server

Are there any plans to release a Haskell indexer?

The readme doesn't mention a Haskell indexer: is this planned or likely? Thanks!

Add scip/codemarkup language support for Python and C#

We have indexers for Python and C# now via scip. To make their data more accessible, they should be exported to the codemarkup.angle layer of Glean schemas.

update the scip entity type https://github.com/facebookincubator/Glean/blob/main/glean/schema/source/code.scip.angle
extend the tag in the type Entity, in scip.angle https://github.com/facebookincubator/Glean/blob/main/glean/schema/source/scip.angle#L144
add a case to the untagger https://github.com/facebookincubator/Glean/blob/main/glean/schema/source/scip.angle#L167

Regenerate the schema with make, confirm that it works by writing regression tests for codemarkup.* queries.

Check and confirm glass builds. cabal build glass. Add any missing cases until it compiles.

New programming languages

Exist any map or checklist for adding support for new programming languages ? Example: I want to test it for Ruby and I would be interested in adding support.

Mac OS build

Hey folks,

Glean is looking like a very interesting project, is anybody have a instruction for compilation for mac os?

currently, I stuck on the compilation of fbthrift

[ 15%] Linking CXX shared library ../../lib/libcompiler_base.dylib
Undefined symbols for architecture x86_64:
  "apache::thrift::compiler::gen::cpp::find_ref_type(apache::thrift::compiler::t_field const&)", referenced from:
      apache::thrift::compiler::(anonymous namespace)::validate_boxed_field_attributes(apache::thrift::compiler::diagnostic_context&, apache::thrift::compiler::t_field const&) in standard_validator.cc.o
  "apache::thrift::compiler::cpp2::has_ref_annotation(apache::thrift::compiler::t_field const&)", referenced from:
      apache::thrift::compiler::(anonymous namespace)::validate_ref_field_attributes(apache::thrift::compiler::diagnostic_context&, apache::thrift::compiler::t_field const&) in standard_validator.cc.o
  "apache::thrift::compiler::cpp2::is_mixin(apache::thrift::compiler::t_field const&)", referenced from:
      apache::thrift::compiler::(anonymous namespace)::validate_mixin_field_attributes(apache::thrift::compiler::diagnostic_context&, apache::thrift::compiler::t_field const&) in standard_validator.cc.o
      apache::thrift::compiler::(anonymous namespace)::get_mixin_type(apache::thrift::compiler::t_field const&) in standard_validator.cc.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [lib/libcompiler_base.1.0.0.dylib] Error 1
make[1]: *** [thrift/compiler/CMakeFiles/compiler_base.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Issue building Glean: cannot stat ‘.build/def/cxx/defs.m4.new’: No such file or directory

When following the install instructions, the build fails immediately with this message;

$ ./install_deps.sh
mv: cannot stat ‘.build/def/cxx/defs.m4.new’: No such file or directory   
make[1]: *** [.build/def/cxx/defs.m4] Error 1                                           
make: *** [.build/def/cxx/defs.m4] Error 2

Clarifications on src.angle

I am trying to understand Glean's requirements for representing source locations. There are a couple of points that are not clear to me after reading src.angle:

What's a column? Is it
- a visual concept (based on the display column), which would take into account expansion of tabs and Unicode combining characters, or
- an offset into the string content of the line, in which case is the offset counted in UTF-8 bytes, UTF-16 code units, or code points?
How do byte spans work if a file uses \r\n as a line terminator? Do they assume \r\n has been normalized into \n or are they counted as two bytes?

Homebrew installation

It would be much easier to see what Glean is about if it was available from brew install instead of Docker.

https://docs.brew.sh/Homebrew-on-Linux

Recursive Queries in Angle?

Is it possible to do recursive queries and if so what is the recommended way? For instance, in the schema in the example, is there an easy query that could list all the ancestor classes of "Goldfish" (in this case, it should give back "Fish" and "Pet" for example)?

facebookincubator / glean Goto Github PK

glean's People

Contributors

Stargazers

Watchers

Forkers

glean's Issues

internal goes in a subdir, so do it separately

Recommend Projects

Recommend Topics

Recommend Org