prismarinejs / minecraft-data Goto Github PK

Just like node-minecraft-data but for other languages.

It might be nice to create python-minecraft-data for example.

It might be interesting to create a "doc/make-a-new-wrapper.md"

Currently blocks and items are indexed by id in items.json and blocks.json.
Is that really a good idea ? As seen in #7 users sometimes need to index by name or by metadata+id.

It should probably not be the responsibility of minecraft-data to choose that indexing, but the responsibility of the reader/user.

Not doing that indexing would simply mean using json arrays instead of json objects.

Example :

{
  "name": "chunkDataLength",
  "type": "count",
  "typeArgs": {
    "type": "varint",
    "countFor": "chunkData"
  }
},
{
  "name": "chunkData",
  "type": "buffer",
  "typeArgs": {
    "count": "chunkDataLength"
  }
}

should become

{
  "name": "chunkData",
  "type": "buffer",
  "typeArgs": {
    "countType": "varint"
  }
}

This greatly simplifies the protocol.json file, and makes it more obvious that count fields should not be supplied on write. I went over the 1.8 protocol, there are no instances where the count field is far away from the data it counts.

It'd be interesting to have the MCPE data available here as well. I believe the two games are close enough to actually be able to support both schema types. The protocol will probably be the one file to change the most, but I believe its schema won't have to change all that much either

"hasInventory": {true|false}

Extractable from drops ("plus content") http://minecraft.gamepedia.com/index.php?title=Chest&action=edit

Mojang provides Internationalization files in the client jar (for the display names)
It might be interesting in the long term to provide that kind of data somehow.

Example in english http://hastebin.com/tisalowasa

01:13 < roblabla> > you need to open 1.8.json
01:13 < roblabla> find the asset you want (probably minecraft/lang/en_GB.json
01:13 < roblabla> and see the associated hash
01:13 < roblabla> for me it's 03f31164d234f10a3230611656332f1756e570a9
01:13 < roblabla> then it's in assets/bojects/03/03f31164d234f10a3230611656332f1756e570a9
01:13 < roblabla> something like that
01:13 < roblabla> 1.8.json is inside .minecraft/assets/index

Option type documentation currently doesn't show the underlying type. Need to find a simple way to do that.

• https://github.com/andrewrk/mineflayer/blob/master/lib/windows.js#L22
• https://github.com/SpockBotMC/SpockBot/blob/master/spock/plugins/helpers/inventory.py#L204

22:09 <+Gjum> it's just the window name (for protocol), readable name (although that can get set by server), named slot ranges, names for the properties, and later buttons
22:10 <+Gjum> properties are for example the smelting progress of a furnace
22:11 <+Gjum> buttons would just be list[id -> "name"]
22:12 <+Gjum> and then there are some edge cases like the horse, and the player itself
22:12 <+Gjum> horse has an entity id, player has armor and crafting, but no nameID

See https://gist.github.com/rom1504/7b4c0cc29fb03122b6a5#relation-with-wikivgprotocol and https://github.com/PrismarineJS/minecraft-data/blob/master/bin/wiki_extractor/protocol_extractor.js

Would also be useful to work on having a .md from wiki.vg/Protocol for each packet description (also explained in the gist)

We need to be able to know things (like the metadata id) about variation of blocks, like http://minecraft.gamepedia.com/Polished_Granite is a variation of http://minecraft.gamepedia.com/Granite

blocks.json currently looks like that :

"1": {
    "id": 1,
    "displayName": "Stone",
    "name": "stone",
    "hardness": 1.5,
    "stackSize": 64,
    "diggable": true,
    "boundingBox": "block",
    "material": "rock",
    "harvestTools": {
      "270": true,
      "274": true,
      "257": true,
      "278": true,
      "285": true
    }
  }

I propose a schema like that instead :

"1": [{
    "id": 1,
    "displayName": "Stone",
    "name": "stone",
    "hardness": 1.5,
    "stackSize": 64,
    "diggable": true,
    "boundingBox": "block",
    "material": "rock",
    "harvestTools": {
      "270": true,
      "274": true,
      "257": true,
      "278": true,
      "285": true
    }
  },
{
    "id": 1,
    "displayName": "Polished Stone",
    "name": "polishedStone",
    "hardness": 1.5,
    "stackSize": 64,
    "diggable": true,
    "boundingBox": "block",
    "material": "rock",
    "harvestTools": {
      "270": true,
      "274": true,
      "257": true,
      "278": true,
      "285": true
    }
  },

]

or similar : maybe some properties never change between two blocks only different by metadata ?

something like http://prismarinejs.github.io/minecraft-data?p=1.8

The other schemas have them and it really improve the aspect of http://prismarinejs.github.io/minecraft-data/ (and I guess the understanding when reading the json schema file too)

With data like canFall , ...

Not sure what it would contain exactly and how it would be used.

The main point is we would have to build it manually though.

https://github.com/macbre/nodemw#botgetarticlerevisionstitle-callback

https://www.mediawiki.org/wiki/API:Revisions

Currently, buffer "hides" the countType field. Need to find a simple way to reveal it. A suggestion rom1504 made : make type array counted by <countType>.

Currently mc version are stored as different branches.

Problem with that is a lot of commits need to be cherry-picked from 1.8 to 1.9 because they change things common between versions : changes in the README, sometimes changes in the extractors, new data files.

I think storing version in enums/1.8 and enums/1.9 might be saner : only the data should really be different.

• the schemas should be the same : see #49
• the readme should be the same
• the extractor and tools are a bit more complicated, but basically it would be better to have everything that is common in common files and to have a few functions/files specific for different mc version / revision of the wiki / java decompiled code format

I think to be really sure it can work, the best way would be to try to use the extractors for 1.9.

So I'll do that at some point and that'll give us more info into that issue.

I think it would be useful to have a javascript provider for node.js users of this library.
For example it might provide indexes and function using these indexes to find blocks by name (something like this https://github.com/PrismarineJS/minecraft-data/blob/master/bin/wiki_extractor/lib/find_item_object_by_name.js)

That way users of that library wouldn't have to build the index themselves every time (see #14 (comment) )

The problem is minecraft-data should stay a language interdependent library imho. Does that mean that "data provider" should still be in this repo and other language users would just not use them or should there be a "node-minecraft-data" repo that would depend on this repo and provide these indexer and all ?

I also thought about simply putting that provider inside mineflayer, but the thing is minecraft-data will be used by projects not using mineflayer (see PrismarineJS/prismarine-server#23)

The advantage of a node-minecraft-data provider is would provide an example of how to read the data, so that other people using other languages could do that kind of providers too if they want (for example a python-minecraft-data )

What do you think ?

In order to be even more language independent, let's put js stuff in tools/js :

• package.json
• bin/
• test.js
• removing index.js (along with that issue PrismarineJS/node-minecraft-data#4 )

https://github.com/lbovet/docson
see an example here http://lbovet.github.io/docson/index.html#/docson/examples/example.json

But I'm not sure where this would go. And maybe the json file is readable enough ?

It might be fun to generate something looking like http://minecraft-ids.grahamedgecombe.com/ from all the data stored in minecraft-data.
It might also get some more people interested in minecraft-data and motivated to help figure out ways to update that data.

There's some more data that I think can go from mineflayer to minecraft-data :

• animationEvents / entityStatusEvents / spawnedObjectTypes / mobTypes https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/entities.js#L10 PrismarineJS/mineflayer#207 #11
• NAMED_ENTITY_HEIGHT / CROUCH_HEIGHT https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/entities.js#L5
• dimension_names https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/game.js#L3
• difficultyNames https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/game.js#L9
• gameModes https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/game.js#L46
• physics constants https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/physics.js#L16
• viewDistanceToBits https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/settings.js#L11
• armorSlots https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/simple_inventory.js#L9
• QUICK_BAR_COUNT and QUICK_BAR_START https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/simple_inventory.js#L6
• INVENTORY_SLOT_COUNT https://github.com/andrewrk/mineflayer/blob/master/lib/windows.js#L7
• windows https://github.com/andrewrk/mineflayer/blob/master/lib/windows.js#L22 : some change in mineflayer code needed for this, the data is at least : window names, inventorySlotStart

See what kind of .json this would go into.

Some cases are not handled by the recipes format, in particular a lot of thing related to metadata.
See https://github.com/PrismarineJS/minecraft-data/wiki/Recipe-format vs https://github.com/PrismarineJS/minecraft-data/wiki/Proposed-new-recipe-format
What should be done :

• finish defining the new format
  • check it handles all cases
  • check it isn't too ugly
• see how it would be implemented in mineflayer
• update the wiki extractor scripts
• update the resulting recipes.json file

Might also be related to PrismarineJS/mineflayer#222

It might also be a good idea to define a jsonformat for this ( see #2 ) if it's not too complicated.

The current plan is to support minecraft 1.8.
But at some point 1.9 will be out, and supporting both version might be nice.
How could we support multiple versions ?
Different branches ? Different folders ?

(this is a long-term issue, mainly becoming relevant when 1.9 comes out)

Main issue

Currently each version has its own schema.

What would be nice is having a common schema for every version.

Problem is : new things will be added and older versions didn't have as much data.

An other problem : some files, for examples blocks.json contain too many things which is annoying when creating extraction scripts (and possibly when reading it too)

One solution : keep things meaningful, split big files.

For example we might have a blockVariation.json instead of storing all the variations in blocks.json.

That would allow to have one common schema for all the versions. Some version just wouldn't have some of the data files.

Related/ideas

• database schema are usually split when putting it in https://en.wikipedia.org/wiki/Database_normalization#Normal_forms
• knowledge bases avoid that problem by putting data in its atomic form : triples

The knowledge base way is close to what we are doing (for example <1,name,stone>)
and it's even possible to convert all our data to a knowledge base forma (like n triples) by using json ld (http://json-ld.org/)

Also see http://wikiba.se/

Currently harvestTools is interpreted by mineflayer as requiredTools : one of these tool is needed to harvest a block (see https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/digging.js#L92 )

In the reality of data there are tools that improve the digging time but are not needed to dig that block. (see chest in http://minecraft.gamepedia.com/Breaking#Blocks_by_hardness for example)

I actually think that mineflayer algorithm to compute the digging time is correct (might be interesting to check that by computing a similar table than in http://minecraft.gamepedia.com/Breaking#Blocks_by_hardness using that algo) but it would be better to document that harvestTools means needed tools.

Also see http://minecraft.gamepedia.com/index.php?title=Template:Block&action=edit to understand better in which case a tool is required (=3.33 times penalty if not wearing it) or just better.

That would prevent stuff from breaking when we change stuff (like currently)

One way would be just submoduling node-minecraft-data.

Currently packets names are taken from https://github.com/aadnk/ProtocolLib/blob/master/ProtocolLib/src/main/java/com/comphenix/protocol/PacketType.java#L123
I don't know where the fields names are coming from (@roblabla ?)

These names are different from what is currently found in http://wiki.vg/Protocol

Can reasonable names be chosen to make protocol.json more standard ?

The packet name difference with wiki.vg can be seen here https://gist.github.com/rom1504/1b432d5e7efc278015dd/revisions?diff=split

I mapped the new names to the old and got that which show mainly the field names difference :
https://gist.github.com/rom1504/64535ed808327b78c72e/revisions?diff=split

Protocols define a data-type named "switch" which seem to handle everything condition does, in a cleaner, more readable way. It looks like this :

{
  "type": "switch",
  "typeArgs": {
      "compareTo": "this.someOtherArg",
      "fields": {
        "0": "int",
        "default": "void"
      }
  }
}

Void is a new datatype that does nothing. It doesn't read, and returns nothing. It may as well not have been called.

There are block state files available by doing this :

mkdir /tmp/dminecraft
cp ~/.minecraft/versions/1.8.4/1.8.4.jar  /tmp/dminecraft
cd /tmp/dminecraft && unzip 1.8.4.jar
cd /tmp/dminecraft/assets/minecraft/blockstates

Example : birch_fence_gate.json :

{
    "variants": {
        "facing=south,in_wall=false,open=false": { "model": "birch_fence_gate_closed" },
        "facing=west,in_wall=false,open=false":  { "model": "birch_fence_gate_closed", "y": 90, "uvlock": true },
        "facing=north,in_wall=false,open=false": { "model": "birch_fence_gate_closed", "y": 180, "uvlock": true },
        "facing=east,in_wall=false,open=false":  { "model": "birch_fence_gate_closed", "y": 270, "uvlock": true },
        "facing=south,in_wall=false,open=true": { "model": "birch_fence_gate_open" },
        "facing=west,in_wall=false,open=true":  { "model": "birch_fence_gate_open", "y": 90, "uvlock": true },
        "facing=north,in_wall=false,open=true": { "model": "birch_fence_gate_open", "y": 180, "uvlock": true },
        "facing=east,in_wall=false,open=true":  { "model": "birch_fence_gate_open", "y": 270, "uvlock": true },
        "facing=south,in_wall=true,open=false": { "model": "birch_wall_gate_closed" },
        "facing=west,in_wall=true,open=false":  { "model": "birch_wall_gate_closed", "y": 90, "uvlock": true },
        "facing=north,in_wall=true,open=false": { "model": "birch_wall_gate_closed", "y": 180, "uvlock": true },
        "facing=east,in_wall=true,open=false":  { "model": "birch_wall_gate_closed", "y": 270, "uvlock": true },
        "facing=south,in_wall=true,open=true": { "model": "birch_wall_gate_open" },
        "facing=west,in_wall=true,open=true":  { "model": "birch_wall_gate_open", "y": 90, "uvlock": true },
        "facing=north,in_wall=true,open=true": { "model": "birch_wall_gate_open", "y": 180, "uvlock": true },
        "facing=east,in_wall=true,open=true":  { "model": "birch_wall_gate_open", "y": 270, "uvlock": true }
    }
}

Might be interesting for handling metadata/blockstate (and placing that kind of blocks)

http://prismarinejs.github.io/minecraft-data/#protocolTableA is useful but it would be even more useful if it displayed stuff in tables like wiki.vg

inspiration from the transformation in https://gist.github.com/rom1504/0d41cc284de99dfc8b6e

Also see PrismarineJS/node-minecraft-data#4

The master branch wouldn't contain much (it might even be deleted if that's possible I guess).

Current name in blocks.json and items.json are legacy names and have been changed in minecraft itself,.

I think it would it be better to replace them by the current names.
For example swordDiamond -> diamond_sword (or even minecraft:diamond_sword ?)
see http://minecraft.gamepedia.com/Data_values/Block_IDs

That means changing bots using these names

It might be interesting to have variations of name/displayName of blocks/items, that might be useful to parse what an user is saying.
It could look like that :

{
   "variation":"unique_identifier",
}

That could be extracted from :

• redirect pages in the wiki
• links labels in the wiki

I propose something like one tab by data type (block, entity, protocol, ...) and then in each tab you can choose schema or value (like http://getbootstrap.com/javascript/#tabs )

That way the schema and the value display can take as much room as they need, and it will be less confusing to point somebody to the protocol values for example.

See http://json-schema.org/ and https://www.npmjs.com/package/jsonschema

The point of this is to check automatically the json format documentation (see https://github.com/PrismarineJS/minecraft-data/wiki/Recipe-format) is actually true and can be trusted, even if recipes.json is updated or if the format change.

It would be nice to add this test to a ci service (circle ci ?)

something like http://wiki.vg/Protocol
Or should it be like the table of packets at http://wiki.vg/Pre-release_protocol ?

Currently we have that https://github.com/andrewrk/mineflayer/blob/master/lib/plugins/entities.js#L28
A lot of missing data compared to http://minecraft.gamepedia.com/Data_values/Entity_IDs

I propose a json like that :

{
 "10": 
   {
      "id": 10,
      "name":"Arrow",
      "type":"Projectile"
   }
}

It would also be possible to have one json file for every type of entity but that would be a lot of files, I think it's probably better to put everything in one file.

Not having an index in json files makes it easier for users to index by what they need. That means the data doesn't have to change if one index or an other becomes more pratical (id vs id:metadata vs name ,... ). It also makes it possible to consider one element as a document of a database.

All json files are now unindexed except for recipes and materials.

I think materials could have a format like this :

{
  "material":"rock",
  "multipliers":  {
    "257": 6,
    "270": 2,
    "274": 4,
    "278": 8,
    "285": 12
  }
}

I'm not sure whether multipliers should be unindexed too since it only has one value, maybe.

The recipes file is currently indexed by output id. The simplest solution might be to just remove that indexing, and to say that the recipes should be ordered by a lexicographic order on (id,metadata) in order to have a decent diff.

There are many ways to extract the data that should go into minecraft-data, as discussed there PrismarineJS/mineflayer#229 .
In this issue I'll focus on progress on extracting information from the wiki http://minecraft.gamepedia.com/

There should be a oneOf for the field : the possible fieldArgs depend on the fieldType.

It might be cool for the protocol for example

http://jeremydorn.com/json-editor/

would be possible to put it there http://prismarinejs.github.io/minecraft-data/

see tabs in https://github.com/jdorn/json-editor/#arrays : useful for tabs

To make it display stuff, it's needed to set additionalProperties to true for the patternProperties, not sure why

It can probably still be improved even more but something like that seems nice :

Some tools use relative paths (e.g. https://github.com/PrismarineJS/minecraft-data/blob/1.8/tools/js/wiki_extractor/protocol_extractor.js#L53), which leads to them not working properly when used from another directory. They should instead use __dirname to behave properly at all times.

For recipes for which any metadata in the items produce the same result (for example bed) :

• remove the ingredients metadata (=any metadata)
• have juste one recipe

blocks.json and items.json already contain "simple variations" : only the metadata and the displayName changes (for example stones).

But there is an other kind of variation : functional variations, examples :

• http://minecraft.gamepedia.com/Stairs/DV
• http://minecraft.gamepedia.com/Sign/DV

These variations store information based on a state stored in bits.

How to store this in minecraft data ? related to #18

http://minecraft.gamepedia.com/Written_Book is an unordered recipe, ie it uses ingredients. But it doesn't consume the written book so there should be a "outIngredients" in the recipe format.

see https://github.com/PrismarineJS/minecraft-data/blob/master/enums/recipes.json#L3479

Currently, there's no way to know what kind of blocks drops we can have. The problem with block drops is that they depend on item used to mine, among other things. Anyway, a good start would be :

{
  "id": 1,
  "name": "stone",
  // ...
  "drops": [{ "id": 4, "minAmount": 1, "maxAmount": 1 }]
}

The problem with the above : it doesn't take into account tool used to mine. Not too sure how to make the schema not too fugly otherwise...

How does this sound ?

see #37

and #41

Some recipes are missing :

• wool
• banner

To do : fix the wiki recipes extractor to handle these.

Should be fairly easy, just a new dependency and some change in index.js in mineflayer.