Why does v0.6.0 query a custom index domain-name, instead of crates.io API ? about crates HOT 33 CLOSED

Okay, here's the whole deal: At first, we were using the crates.io API, but they asked us not to and switch to the GitHub API instead. So, we did that, but then people started complaining about API limits.

To solve that problem, we added local index support, thinking it'd make things better. But then folks started saying cargo wasn't getting updated, and we couldn't see the latest versions or have a local registry.

Eventually, cargo went through a big protocol change, which was a major bummer for us. So, I took matters into my own hands and created a similar API using the crates.io GitHub index.

Why did I do it? Well, managing that local index was driving me nuts, and I simply didn't have enough time to deal with all the user issues. Plus, that protocol change was a real dealbreaker. And don't even get me started on how huge the extension code was getting!

The only info we send to the server is the crate name you're requesting. Don't worry; it's all open source, and you can check out the data we send—it's transparent like that. I got the cheapest domain because, honestly, nobody's paying or donating, and the privacy option that came with it was just the default, so I didn't bother changing it or thinking too much about it.

from crates.

Release notes are here https://github.com/serayuzgur/crates/blob/master/CHANGELOG.md, but yeah it wasn't done the usual way (not released on github), and I still see no reason not to use the official API

And because of this, people will think it fetches crates.io, but it obviously doesn't :

from crates.

As a long-time user of crates-io extension, I would appreciate, if the fork would stay up at least until this issue is resolved and the PR (with the additional feature of choosing the index url in the settings) is merged. When I read about this issue here, I immediately switched to the fork, as I'm in general pretty concerned about my privacy.

@simonsan
If so , please use older version of crates or use packaged version of the fork. I am not feeling good about it all. Regarding privacy I'm making an effort to provide reasonable justifications, but maintaining my stance is becoming more difficult. Please explain how the crate name being the only information transmitted to the server can cause you to be concerned about privacy.

from crates.

I think this is actually quite concerning.

All of the sudden we have data being send to a 3rd party URL. There are no release notes, except for all cis migration done, we don't know the owner and we don't know what happens with our data.

The WHOIS shows a registrant in Iceland (IS):

https://www.whois.com/whois/crates-vsc.space

...
Registrant Organization: Privacy service provided by Withheld for Privacy ehf
Registrant State/Province: Capital Region
Registrant Country: IS
Registrant Email: Please query the RDDS service of the Registrar of Record identified in this output for information on how to contact the Registrant, Admin, or Tech contact of the queried domain name.
...

from crates.

from crates.

Just thought I'd link to a discussion regarding this issue.

https://www.reddit.com/r/rust/comments/15ye9he/i_forked_crates_vscode_extension_to_make_it_use/

Seems as though a fork of this project is using the sparse HTTP protocol to get the info directly from crates.io's AWS S3 files directly (which is much easier on them because it doesn't even touch their API)

Would be nice if that could be merged. Not that I don't trust the new author, but it seems like post-sparse-protocol this is the optimal method for doing this.

from crates.

84b70c4...BarbossHack:crates-io:2ab0c2f649b9037ca2e3717a0592eac1dfecb94b

The diff seems pretty small. Perhaps preparing a PR for this repo (without all the funding removals and renamings etc.) would be appropriate.

from crates.

@simonsan I will think about it. I need to calculate the effect of this on new planned features

from crates.

@BarbossHack I'd kindly request that you consider removing it from the market altogether. Given that the fork no longer offers offline support due to the shift towards a sparse approach, it might inadvertently mislead users who are seeking that functionality. This step would ensure that users make informed decisions and that we avoid any potential confusion.

from crates.

As a long-time user of crates-io extension, I would appreciate, if the fork would stay up at least until this issue is resolved and the PR (with the additional feature of choosing the index url in the settings) is merged. When I read about this issue here, I immediately switched to the fork, as I'm in general pretty concerned about my privacy.

from crates.

@BarbossHack I'd kindly request that you consider removing it from the market altogether. Given that the fork no longer offers offline support due to the shift towards a sparse approach, it might inadvertently mislead users who are seeking that functionality. This step would ensure that users make informed decisions and that we avoid any potential confusion.

I am currently using my fork offline, so it still allow offline compatibility for users. I have a sparse index clone offline (git clone) with fake dns, so crates-io is able to fetch it (as it would with crates.io api, I would have used the same process). Maybe it wasn't clear, but when I mean "offline" I mean "on a network that does not have internet connection", not simply "local fetch"

I saw your comments on PR, I will be able to make the changes soon (can't do it at work), thank you for allowing us to implement this feature 👍

from crates.

Advantage of Json is that you don't have to add all unnecessary fields to make it compatible, just adding the versions under the same tree is sufficient.

Thank you for your answer.

from crates.

Or at least could you share your api server code on github ? So we will be able to deploy our own server (for offline uses for example).

Thank you.

What is the rationale behind requiring offline usage if you're unable to access the latest version information and install the desired version of the crate you seek?

I am planning some scenarios for the server , In progress.

from crates.

What is the rationale behind requiring offline usage if you're unable to access the latest version information and install the desired version of the crate you seek?

I have a full crates.io mirror offline which is often updated, so I'm able to install latest crates version.
If you publish your server code on github, I will be able to deploy my own server offline too, using this mirror.

from crates.

Since the PR is replacing the code but not implementing a different api connection code regarding to configuration I can not accept it. @BarbossHack Thanks for your effort, please take a look at my comments on PR

from crates.

@BarbossHack please check changelog

Thank you for the credits 🙏 🚀

from crates.

Thank you for your explanations @serayuzgur 👍

Quote from my first comment :

Or at least, have the same api URI than crates.io (/crates/NAME/versions and not /index/versions/NAME), and same json response, to allow us to set our own offline api domain name

Would that be possible ?

Thank for your work

from crates.

or should I open a new issue ?

from crates.

(Also if you publish your server api on github I would be able to make a PR to help you)

from crates.

Okay, here's the whole deal: At first, we were using the crates.io API, but they asked us not to and switch to the GitHub API instead. So, we did that, but then people started complaining about API limits.

To solve that problem, we added local index support, thinking it'd make things better. But then folks started saying cargo wasn't getting updated, and we couldn't see the latest versions or have a local registry.

Eventually, cargo went through a big protocol change, which was a major bummer for us. So, I took matters into my own hands and created a similar API using the crates.io GitHub index.

Why did I do it? Well, managing that local index was driving me nuts, and I simply didn't have enough time to deal with all the user issues. Plus, that protocol change was a real dealbreaker. And don't even get me started on how huge the extension code was getting!

The only info we send to the server is the crate name you're requesting. Don't worry; it's all open source, and you can check out the data we send—it's transparent like that. I got the cheapest domain because, honestly, nobody's paying or donating, and the privacy option that came with it was just the default, so I didn't bother changing it or thinking too much about it.

I appreciate the update.

With all of that said and crates.io now being sparse, is this still an issue?

from crates.

or should I open a new issue ?

No, there's no need for that. I don't plan to make them identical. You see, the crates.io API isn't quite designed for the specific task of version listing; it generates a response JSON that's quite bloated with unnecessary data. Given this situation, I've decided to pursue an alternative course of action.

from crates.

Or at least could you share your api server code on github ?
So we will be able to deploy our own server (for offline uses for example).

Thank you.

from crates.

Okay, here's the whole deal: At first, we were using the crates.io API, but they asked us not to and switch to the GitHub API instead.

May I ask you who declined to use crates.io? Namecheap is blocked in a lot of security appliances. For instance at PaloAlto the url https://api.crates-vsc.space/ is categorized as insufficient-content. If it would be possible to get it back to crates.io, it would be very nice.

Thank you

from crates.

@pingufreak

May I ask you who declined to use crates.io?

I can't recall the individual responsible for managing the crates.io API. 3-4 years ago.

Namecheap is blocked in a lot of security appliances.

I can transfer dns to Digital Ocean. Will it function properly?

the url https://api.crates-vsc.space/ is categorized as insufficient-content

Maybe you can help me to make url trustable. I don't understand why it is considered like that.

from crates.

Why is this closed? If you can change it back to crates.io it would be the best. :)

from crates.

Why is this closed? If you can change it back to crates.io it would be the best. :)

It was closed because he didn't wanted to uses crates.io, mainly for optimisation reasons (see #195 (comment)) (imho the only valid reason)

from crates.

Why is this closed? If you can change it back to crates.io it would be the best. :)

It was closed because he didn't wanted to uses crates.io, mainly for optimisation reasons (see #195 (comment)) (imho the only valid reason)

In addition several other minor issues that do not affect your life but do affect mine 😄 . Base for bigger features like private registry support, on premise crates index server, etc.

from crates.

But I have problem with this host. Is there any alternative？
Completed with errors xxx: Error: statusCode=504

from crates.

Would be nice if that could be merged. Not that I don't trust the new author, but it seems like post-sparse-protocol this is the optimal method for doing this.

(Don't worry you're totally right, and if this get merged I would even delete my fork 👍 )

from crates.

see #204

I know that @serayuzgur has other reasons not to use crates.io (whether with the api or sparse index, see #195 (comment) and #197 (comment)), so I'm quite sure this PR will be closed, but at least I tried. And again, I understand these reasons (and I'm thankful for your work @serayuzgur), but I needed an offline compatible version...

from crates.

see #204

I know that @serayuzgur has other reasons not to use crates.io (whether with the api or sparse index, see #195 (comment) and #197 (comment)), so I'm quite sure this PR will be closed, but at least I tried. And again, I understand these reasons (and I'm thankful for your work @serayuzgur), but I needed an offline compatible version...

Thanks for your work! It would be nice if serayuzgur would merge this and maybe there is a world, where both solutions can coexist, in the sense of making the crates index configurable in the settings?

from crates.

published v 0.6.2

from crates.

@BarbossHack please check changelog

from crates.