daviddengcn / gcse Goto Github PK

Details:

• Use Github API
• Crawl by projects other than packages
• save/use branch/merge information

Hi. Firstly, thank you for making Go Search. It's a useful tool.

I have noticed that some of my own packages are showing as "not found" on go-search.org, even though they are shown as imports of other packages. They begin "xi2.org/x/..."

For example, if you view the page http://go-search.org/view?id=github.com%2fklauspost%2fpassword and select my package "xi2.org/x/xz" which appears as an import at the bottom of that page you get the "not found" response.

I decided I would try to debug the issue, so I downloaded the source and compiled gcse myself. I ran it and limited the crawl to just a handful of packages including my own, and to my surprise it worked fine when I ran it locally! So I can't help debug it it would seem since there must be something different about my local environment :(

As far as I know the packages download fine with "go get" and they show up ok on godoc.org. Any ideas? Is there something weird about my web server?

It might seem quite conceited of me to be concerned about my own packages, but you never know, it might also be happening to someone else's too.

Best wishes

I think having the landing (start) page look more like NPM makes sense (https://npmjs.org/). If you agree David, I will make the change and put in a pull request.

Hello, I've just updated visualization of go packages: https://anvaka.github.io/pm/#/galaxy/gosearch?l=1

On your home page http://go-search.org/ you mention that index was last updated 290 days ago. Any chance we can reindex?

Thank you!

Hi,

I'd like to propose you to use CloudFlare for having a CDN in front of the service and also enable some caching for queries. But more importantly, you'll get the SSL certificate for free which is a nice touch.
What do you think?

Thank you for your time and help.

many links open in new windows.
in my humble opinion this is not wanted always (and seldom in my case).

as current browsers make it easy to force the creation of a new window/tab (e.g. hold ctrl or use the middle mouse button) but not so easy to prohibit the creation of a new tab/window, i suggest that http://go-search.org/ should not create new windows by default and everyone who wants new tabs can do so by pressing ctrl.

Some times dead projects rank highly. It would be nice to have an indication of the last commit as a date or a human indication (such as "two years ago"). This is quite essential information to know how alive a project is.

http://go-search.org/search?q=github.com%2Fkisielk%2Fgotool

is this a temporary glitch?

What do you think about supporting https://github.com/Masterminds/glide ?

while searching https://github.com/typhoeus/ gives some results.

currently the "Project" link (e.g. the link to https://launchpad.net/goyaml/ on http://go-search.org/view?id=launchpad.net%2fgoyaml ) is in the footer of the site, below the list of imports. i think it would be a good idea to move it further to the top of the site, and probably make it more prominent.

I found a reflected XSS issue on gosearchresults.com
Steps to reproduce:
Call the following URL in Mozilla Firefox: https://lavasoft.gosearchresults.com/?sbtn=&q=ola%3Cscript%3Ealert%28%27I+can+send+any+message+here%21%27%29%3C%2Fscript%3E&tt=VM__GS__S4LAVA__vmn__webcompa__1_0__go__ch_WCYID10438__180722__yrff&pid=5ac784309091147a162b4431&sr=0

An alert box with "I can send any message here!" appears. This means that an attacker has full control of the scripts, that are executed in the victims browser.
An attack vector would be sending an evil link via e-mail, messenger, etc. As the victim trusts the domain gosearchresults.com, it will click the link and could be redirected to a site hosting a browser exploit kit.
This abuses the trust of gosearchresults.com
When i enter any script in URL bar then after clicking enter it encrypt the URL in encoded form but still script gets exicuted. This is a big problem by which a hacker can steal the cookies of victim's browser and also can redirect that user to any other site or search engine.

The number of stars of projects on google code is not crawled now.

On sites like github, you can see if a project is a fork of another. It would be nice if the output of the search could show a tree hierarchy, where the main project is first and all the forks are under it. Or maybe some other idea that's similar.

Hi @daviddengcn,

Thanks a lot for doing go-search.org, it's really wonderful.
I'd like to ask you if it's ok for you to allow the Go plugin for IntelliJ IDEA to use your service.
If you'd like to see how things will look like, you can view the PR here: go-lang-plugin-org/go-lang-idea-plugin#1555
Of course, the plugin will identify itself in the user agent and it tries to cancel the connections as soon as possible / delay doing them before sending the requests.
Please let me know your thoughts on this.
Thank you for your time and help.

Kind regards,
Florin

• Create a new analytics item.
• Add configuration to set analytics ID
• Use package: code.google.com/p/google-api-go-client/analytics/v3
• Call to report crawling stats

https://go-search.org has an expired TLS certificate that was valid until february 23.
On accepting the cert, I'm getting a 503 service unavailable.

I got there through a link on godoc.org.
Perhaps the link should be removed from there?

The ability to setup a mirror on a different domain that exchanges data in both directions would be great. That way a mirror can be used as a primary site if the main site goes down and sync when the main site comes back up.

Lately website has been really slow. Here's an example of a dummy search for testify:

Even a repeated search took 9-10 seconds in average. Are you having an intermittent issues or the project has outgrown the capacity of a Linode instance it's running on?

I'm posting this here on GH Issues as a way to track updates on website in general.

to github.com/awslabs/aws-sdk-go

godoc.org provides this at http://api.godoc.org/packages

There doesn't seem to be a Search API documented at http://go-search.org/infoapi. Is this available in any way? I'd like to enter the same query you'd enter at http://go-search.org/ and get the results as JSON.

Hi David,

OSS Index is an open vulnerability database. We are looking at supporting go at some point in the near future, and would like to use your package database as a starting point. Downloading via your API will take a fair while, and use up bandwidth you may not want used.

We are not creating a competing package search, but need a good list of packages in order to start performing vulnerability auditing.

Would it be possible to download the entire database from somewhere, or can I get permission to use your API to download information on all of the packages?

Regards,

Ken

Not sure of our own preferences on this: would you welcome a PR that pretties things up a bit?

Thinking bringing in a basic bootstrap 3 theme, including responsive support.

Inspired by this twitter thread: https://twitter.com/ggiesemann/status/441007698188840960

for example say i wan to find code that uses github.com/aws/aws-sdk-go

this is a highly used package.

but when i search by this i get lots of forks first, that no one is using.
useless when searching for well used packages

Example: "imap server"

Hi,

I tried adding

zikichombo.org/sound
zikichombo.org/dsp
zikichombo.org/sio
zikichombo.org/codec
zikichombo.org/plug

and it doesn't show up, but it says the packages are successfully added.

It would be nice to have some more details inside README, in particular:

• how it works (or it's supposed to work)
• dependencies (Go packages, database, etc...)
• setup instructions
• how to contribute

Try find the reason or find an alternative.

When I hit http://go-search.org/api?action=package_depends I get 113KB of data. This doesn't seem right, I remember it was returning several hundred megabytes.

Hi David,

I'm going to visualize the go packages universe similar to what I did for npm. Your data source is looking very promising.

After reading your documentation and examples, looks like I will have to:

1. Get all packages http://go-search.org/api?action=packages
2. For each found package make a request (e.g. http://go-search.org/api?action=package&id=github.com%2fdaviddengcn%2fgcse), to get Imported field.

This would probably take up to 20 hours of crawler work. Is there any chance you can expose dependencies information in one call? E.g. npm allows you to get full database via single call:
curl https://skimdb.npmjs.com/registry/_design/scratch/_view/byField

Thank you!