Expose dependency API endpoint about supermarket HOT 24 CLOSED

@gabeweaver @brettchalupa I just wanted to follow up on this. Are there any questions I can answer?

from supermarket.

@sethvargo I understand the need and can see how the implementation would go, but I am not sure at what stage this will happen. We are keeping issues like this open so we can figure out where it fits in. My first thought is that once the API is where it was with the previous community site, we can start implementing endpoints like this.

Once we get to that point, we should talk about how to potentially handle some of the drawbacks and consider the considerations. A Google Hangout closer to the implementation of it would be super helpful.

from supermarket.

Cool sounds like a plan

from supermarket.

The dependency api work is planned for milestone #3. We're making good
progress on Milestone #2 so I imagine we'll get to this sometime in mid to
late April

from supermarket.

It looks like we already have a model for cookbook dependencies, so exposing those via the v1 API would be backwards compatible.

Chef has an internal ticket for this (OC-4150) that specifies the following:

Seems that this is relatively straightforward from the API perspective.

/cookbooks/VERSION

Based on http://docs.opscode.com/api_cookbooks_site_cookbooks_version.html

Request

GET /cookbooks/COOKBOOK_NAME/versions/latest

Or:

GET /cookbooks/COOKBOOK_NAME/versions/VERSION

Response

The JSON would be updated to store the dependencies from the metadata, and display it:

{  
   "license": "GPLv2",
   "updated_at": "2009-09-26T00:51:36Z",
   "tarball_file_size": null,
   "version": "2.0",
   "average_rating": null,
   "cookbook": "http://www.example.com/api/v1/cookbooks/ptapache",
   "created_at": "2009-09-26T00:51:36Z",
   "file": "/tarballs/original/missing.png",
   "dependencies": {
     "mysql": ">= 0.0.0",
     "java": "= 1.2.3",
     "openssl": ">= 0.1.0",
     "build-essential": "~> 1.0.0"
   }
}

Having this exposed in the API would be helpful for Berks in the interim time between now and when we implement the full berks-compatible endpoint.

Apparently because the existing API does not expose the dependencies, https://github.com/berkshelf/berkshelf-api must download each entire cookbook, extract its contents, and read its metadata.rb to get this now. Gross.

Not sure if having those deps exposed should be a separate issue or not, but feel free to do whatever you think is best.

/cc @reset @jtimberman

from supermarket.

@smith it would definitely be nice to not need to download and extract each cookbook to get access to it's compiled metadata. It's worth noting that I definitely do not parse the raw metadata (metadata.rb), but instead just read the compiled metadata (metadata.json).

from supermarket.

@smith I have captured this in Trello: https://trello.com/c/kZFdp1BG

from supermarket.

yeah i beleve we just need a fixed /universe or /cookbooks/universe endpoint that spews out the dependency metadata.

@reset what are your exact requirements? I believe -

• you want every cookbook
• you want every cookbook version
• you only need the compiled dependency information (plus the cbv's name + version of course)

so the JSON response above is mostly what you need (maybe with a 'name' field added so you don't have to parse it out of the url), but you just need an array of every cookbook version?

from supermarket.

@reset Looks like you also use the information on what platform is supported by the cookbook, although that puzzles me a bit, since you might be running berks on your Mac workstation and pushing to a heterogenous mess of Unixes and Windows, so not sure what berks is doing with that?

from supermarket.

what the berks API /universe endpoint looks like right now:

{
  "7-zip": {
    "1.0.2": {
      "endpoint_priority": 0,
      "platforms": {
        "windows": ">= 0.0.0"
      },
      "dependencies": {
        "windows": ">= 1.2.2"
      },
      "location_type": "opscode",
      "location_path": "http://cookbooks.opscode.com/api/v1"
    },
    "1.0.0": {
      "endpoint_priority": 0,
      "platforms": {
        "windows": ">= 0.0.0"
      },
      "dependencies": {
        "windows": ">= 1.2.2"
      },
      "location_type": "opscode",
      "location_path": "http://cookbooks.opscode.com/api/v1"
    }
  },
[...snip...]

from supermarket.

@lamont-granquist the supported platforms is unused. I can't even remember why I wanted to expose it in the first place.

from supermarket.

Eventually the universe will get quite large. I don't know how soon that will be an issue, but the way rubygems.org works is you give it a list of your top level requirements, and it collects the set of all possibly valid dependencies for each requirement. Here's how I implemented it: https://github.com/danielsdeleo/gem-mirror-tools/blob/master/app/app.rb#L160-166

from supermarket.

Yeah, its already over 1.4MB of uncompressed JSON, and yes we should do at least some first-pass depsolving on the server to prune cookbooks that just can't be part of the solution. That is probably milestone 2 tho?

from supermarket.

I'd be happy for that to be a later milestone if it makes sense implementation-wise, but I think it's worth considering whether or not it's actually easier before coming to that conclusion. Definitely depends on how the data is stored and whether or not we already have some sort of background job implementation.

from supermarket.

As the universe grows, we can look into solutions like partial caching on the client end too. So, instead of downloading the whole thing, you download a "diff"

from supermarket.

So what should this endpoint be called? /universe? /cookbooks_universe? /cookbooks/universe? And ideally I'd sort of like to have parity with the server eventually where /organization/foobar/cookbook_universe or whatever pulled down basically the same information.

I'd say we should design it around a bulk-get of all the cookbook metadata off of the server and fashion it however the server team is going to solve bulk-get APIs, but that M2 goal of passing in the top-level deps changes the shape of the API slightly.

Obviously just doing /universe is appealing because it will require minimal changes in berkshelf. I don't think that /organization/foobar/universe makes a lot of sense on the server, though, taken out of context.

from supermarket.

I would like to use /universe

from supermarket.

Downloading a diff of the universe would require a lot of engineering, and still would not be completely real-time, whereas a rubygems-esque API where you supply a query with a list of cookbook names and receive a list of the available versions and those versions' dependencies could be done pretty easily and efficiently as long as the database schema is relatively sane. For example:

query

["mysql", "nginx"]

response

{ "mysql":
  {
     "1.0.0": [["apt", "~> 1.0"], ["yum", "~> 2.5"], ...],
     "2.0.0": [["apt", "~> 1.5"], ["yum", "= 3.0.1"], ...]
  },
{ "nginx":
  {
     "VERSION": [ dependency, ... ],
     ...
  }
}

You'd have to make a few HTTP calls depending on how deep the dependency graph gets, but the total data transfer is relatively small and it should be easy for the DB to cache the most popular cookbooks.

from supermarket.

The Supermarket API now returns dependencies and platforms when viewing details of a cookbook version. Here's a screenshot:

from supermarket.

@raskchanky We're still gonna need some form of dependency API to reduce the number of HTTP calls required to determine the complete set of cookbooks that berks (or ChefDK in the future), but this is awesome and great progress 😄 Thanks!

from supermarket.

@danielsdeleo we can at least change the berkshelf-api server to parse JSON instead of downloading/untarring shit 😄 /cc @reset

from supermarket.

I'm closing this, as #431 was merged earlier today, which should address the concerns here.

from supermarket.

Would be nice to open up another issue to track @danielsdeleo idea of passing in the cookbooks the user is concerned with and using that to filter the 'universe' that is passed back.

from supermarket.

@lamont-granquist Got it covered here #504

from supermarket.