Giter VIP home page Giter VIP logo

cardano-ledger-byron's Introduction

Cardano Ledger

Build Status Coverage Status

ATTENTION: this repository has now been merged into cardano-ledger-specs! Please see the code there.

A re-implementation of the Cardano ledger layer, designed to ease transition between the Byron and Shelley eras.

The formal specifications and associated executable specifications can be found in the cardano-ledger-specs repository.

Demo Mainnet Validation

The validate-mainnet package contains an executable that integrates the cardano-ledger validation logic with the cardano-shell application wrapper. It validates mainnet blocks held in the cardano-mainnet-mirror package and demonstrates:

  • We have implemented ledger validation compatible with the existing chain
  • We are ready to integrate with other parts of the system, including consensus, logging, and benchmarking

You can run the demo using stack by running

stack build && stack exec validate-mainnet

Building

cardano-ledger can be built using stack, cabal, nix, or a combination.

To use stack or cabal without nix, you first need to make sure you have git, openssl, and zlib installed on your machine. You should do this using a package manager appropriate to your OS. After that you can simply use stack build or cabal new-build.

Alternatively you can use nix to install the external dependencies. For stack simply add the --nix flag to your invocation of stack build. If you're on NixOS this will happen automatically. For cabal, you can run nix-shell to enter a shell with the dependencies, and use cabal new-build as normal from there.

You can build directly with nix, by running nix-build -A libs.cardano-ledger. To run the test executable you must first build it with nix-build -A tests.cardano-ledger.cardano-ledger-test -o cardano-ledger-test and then run it with ./cardano-ledger-test.

Running Specific Tests

The cardano-ledger-test test suite uses tasty, so you can choose to run only some of the tests using the -p flag followed by a pattern. Simple patterns such as foo just check for foo anywhere in the tests name or the name of the groups that hold it. So to run only test for the UTxO you could add -p UTxO. The patterns are actually awk expressions, so you can refer to the tasty documentation to help with more complex pattern matching.

Local Dependencies

The cardano-ledger library depends on other libraries of the input-output-hk organization, whose versions are pinned in the stack.yaml file, e.g.:

extra-deps:
  - git: https://github.com/input-output-hk/cardano-prelude
    commit: ff5fd5f33849be8a826506c34e5b0278f267f804
    subdirs:
      - .
      - test

  - git: https://github.com/input-output-hk/cardano-ledger
    commit: 4d9eec080374179bf15bf9c4fca09cc7d73e5f53
    subdirs:
      - crypto
      - crypto/test

Some of these extra dependencies, like cardano-crypto-wrapper above, have their source files in a sub-directory of the cardano-ledger repository. When developing, sometimes it is necessary to modify not only cardano-ledger but also one of these local dependencies. To avoid having to pin local libraries to a specific commit when developing, which allows us to test the changes in a more convenient manner, we recommend using a custom stack configuration file, stack-local.yaml, which specifies that the local dependencies are located in their corresponding sub-folders. For instance, the extra-deps field above, could be modified as follows (in the stack-local.yaml file):

extra-deps
  - ../cardano-prelude # Assuming `cardano-prelude` was checked out one directory above.
  - ../cardano-prelude/test
  - crypto
  - crypto/test

We do not include such stack-local.yaml file here, since it depends on the particular needs of the work being done. Such file would be a copy of the stack.yaml file, with the necessary adaptation. However, we do include a local-stack.sh file, which can be used with the same commands and flags that stack supports, e.g.:

./local-stack.sh build

Updating GHC and Package Dependencies

nix building is handled by haskell.nix, which generates nix infrastructure from stack and cabal files. To generate the infrastructure, run nix/regenerate.sh.

You should rerun this script whenever you update the dependencies in a cabal file or you update dependency/GHC versions in the cardano-prelude snapshot.

So an update of GHC should be as simple as:

  1. Updating snapshot.yaml in cardano-prelude
  2. Updating stack.yaml resolver in cardano-ledger
  3. Running nix/regenerate.sh

This may require updating the version of iohk-nix if the compiler version you're switching to isn't supported in the current version of iohk-nix. This will result in an error like missing attribute 'ghc864'. To update iohk-nix, simply change the git revision in iohk-nix.json, this can be done automatically by running the update-iohk-nix.sh script in the nix/ folder.

Formatting

This repo uses brittany to encourage a consistent formatting style.

If you have brittany installed, the scripts/brittany/brittany-all-hs script will run it over all .hs files with our brittany config file.

Otherwise, if your build fails the brittany CI tests, the buildkite page will include a git patch that you can apply and amend you your commit.

Scaling tests according to TestScenario

This repo uses custom Template Haskell helper functions allow the number of tests to scale for the scenarios of Development, ContinuousIntegration, and QualityAssurance (as defined here). This code block illustrates how to use said functionality:

import Test.Cardano.Prelude
import Test.Options (TestScenario, TSProperty, eachOfTS, withTestsTS)
import Hedgehog (property, (===))

ts_prop_trivial :: TSProperty
ts_prop_trivial = withTestsTS 1000 . property $ do
  True === True

ts_roundTripTrivial :: TSProperty
ts_roundTripTrivial = eachOfTS 1000 genTrivial roundTripsCBORBuildable

tests :: TestScenario -> IO Bool
tests ts = and <$> sequence
  [ H.checkParallel (($$discoverPropArg :: TestScenario -> Group) ts)
  , H.checkParallel (($$discoverRoundTripArg :: TestScenario -> Group) ts)
  ]

It is assumed that genTrivial is defined and in-scope, and that the type it generates has appropriate instances that allow it to roundtrip.

Note that we specify a concrete number of tests to run: 1000. This is the number which will execute in the ContinuousIntegration scenario, and the ratios by which that number will be multiplied for the other scenarios are given here.

Nix Tools

The nix directory contains files related to iohk-nix, the nix-based infrastructure were using to manage our dependencies and build on hydra. The important files are:

  • nix/sources.json, which contains JSON pointing to external nix dependencies (eg. iohk-nix), designed to be updated with niv (available in nix-shell).

  • nix/default.nix, which imports nixpkgs with overlays including iohkNix, commonLib and cardanoLedgerHaskellPackages.

  • nix/regenerate.sh, which update --sha256 in cabal.project.

  • nix/haskell.nix, which creates a package set from the cabal.project using the haskell.nix cabalProject function.

  • default.nix, the top-level nix expression for the project, based on the generated package set

  • release.nix, the specification for which packages to build on hydra

There are a couple of common issues that developers run into while working with haskell.nix:

  • While evaluating release.nix or default.nix, you might see
    "error": "attribute '1.0.0.0' missing, at /nix/store/.../default.nix:6:30"
    
    which indicates that a package version is missing from the hackage.nix package set. This is usually because the version is new to Hackage, and so it has only recently made it into hackage.nix, a set of nix expressions for all the packages on Hackage. haskell.nix is regularly updated with latest version of hackage.nix, so to solve this error use the following command:
    nix-shell -A devops --run 'niv update haskell.nix'
    

cardano-ledger-byron's People

Contributors

angerman avatar askalexsharov avatar avieth avatar cleverca22 avatar codiepp avatar coot avatar craigem avatar csoroz avatar dcoutts avatar deepakkapiswe avatar deepfire avatar disassembler avatar dnadales avatar edsko avatar erikd avatar intricate avatar iohk-bors[bot] avatar jaredcorduan avatar jbgi avatar jimbo4350 avatar ksaric avatar mdimjasevic avatar mhuesch avatar mrbliss avatar nc6 avatar nfrisby avatar redxaxder avatar ruhatch avatar rvl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cardano-ledger-byron's Issues

Model fees

Aspects to model:

Constraints:

  • fees should be modeled in a specific unit, probably Lovelace.

PR

Number Base
PR-210 develop

Consequences

Retrospective

Add a `Cardano.Prelude`

In the legacy codebase we were using the Universum prelude. For a large project like this it makes sense to have a Cardano.Prelude. While importing code from the old codebase this can simply re-export Universum, but we can then easily replace this when we want.

Clean up golden files in `test/golden`

Currently the golden files are bit disorganized. I plan to put all the binary golden files in test/golden/bi/* and all the json golden files in test/golden/json/*

Add cardano-crypto-wrapper package

The naming of this package is a bit unfortunate, but right now the crypto library here is called cardano-crypto, so we cannot reuse the name. It might be best to merge these two libraries now that we're going for a multi-repo structure.

The goal of this issue is to pull in the cardano-sl-crypto package and clean it up according to the conventions of the legacy-free codebase.

Safely remove SSC types

The legacy-free codebase will be using Ouroboros BFT, which doesn't require the complex slot leader schedule of Ouroboros classic. We still need to be able to decode blocks produced by the legacy codebase, so we imported the old SSC types into the new codebase. We can now remove them, but to do so safely we should make sure that old blocks decode at every step. This can be achieved by:

  1. Adding a new testing primitive to check for a successful decoding using a given decoder
  2. Write decoders for all deprecated types that mimic the operation of the Bi instances
  3. Add tests for all deprecated types that decode old golden test values
  4. Remove all types except SscPayload and SscProof
  5. Write an encoder for an empty SscPayload that can be read by the legacy codebase

Change `Coin` to `Lovelace` in `cardano-chain`

Duncan mentioned here that Coin and CoinPortion introduce rounding difficulties. We really do have a basic unit of currency, namely Lovelace, that we can treat as an Integer value. Ada or Coin then become a presentation of the underlying Lovelace.

There might be some problems when validating the old blockchain, so we need to investigate how much the validation relies of the types involved. We should look at all validation rules of existing blocks that involve Coin or CoinPortion.

Add explanations about the operational semantics framework in the ledger and blockchain specs

They should explain the operational rules notation.

Include examples. For instance: examples of States, Transitions, Signals, Rules, etc.
Check compatibility with @duncan Coutts's document. If we have good reasons for deviating from the conventions used there, we should submit a PR against that document.
Specify how definitions are split in Figures: abstract types and functions, constraints, derived functions, state and transition types, and transition rules.
Define the notation for quantifications: for instance, for separating the term of a quantification do we use a regular dot, or a center dot \cdot.

Add cardano-mainnet-mirror as a submodule for testing

We want to run validation over the real blockchain to make sure we're not breaking anything as we move away from the Byron types, serialisation, and validation. We can imagine that the whole blockchain is in a sub-directory by using a submodule. Then tests can just read those files directly as long as the submodule has been updated.

Address any technical-debt imported from `cardano-sl`

Starting a fresh implementation is our opportunity to avoid any 'technical debt' we might have accrued in cardano-sl. A large chunk of code was imported from cardano-sl, so this epic aims to track and deal with any debt that may have been brought over.

Model "composable" serialization

Damian Nadales commented 24 Oct 2018 12:44
@nicholas.clarke wrote some constraints on the serialization function. We might need to change the rules accordingly: For instance instead of [| txbody tx |] we'd write txbody_s [[ tx ]].

nicholas.clarke commented 24 Oct 2018 16:15
I think we should just make a note about this in that document, and chalk it down to another abuse of notation.

Damian Nadales commented 24 Oct 2018 20:14
That's what I'm not sure about. We do want to specify that we shouldn't de-serialize only to obtain the serialized version of a field, but I'm not sure where. I cannot see far enough to understand how txbody_s [[ tx ]] would translate to a Haskell model.

Review `cardano-binary` tests

This issue looks at a number of issues with cardano-binary testing:

  • Should we remove Test.Cardano.Cbor.Canonicity after we stop reserializing data?
  • Do we need Test.Cardano.Cbor.RefImpl?
  • We should move the binaryTests in Test.Cardano.CborSpec to use hedgehog
  • Should we add more/better testing to this package?

Implement UTxO inference rule as validation function

The UTxO inference rule here involves Coins and the calculation of fees. We would like to change Coins to Lovelace and calculate fees in Lovelace to simplify things. To make sure the changes don't break validation, I will implement the rule with the old types, check it works on mainnet blocks, and then update the types making sure it is still correct.

Review the `cardano-crypto` tests

There are a number of issues with crypto testing:

  • We should aim to replace hspec in all tests
  • The binaryTests in Test.Cardano.Crypto.CborSpec can be dropped as we already use hedgehog
  • If possible we could replace all QuickCheck things with hedgehog
  • Examine the use of ArbitraryUnsafe, maybe it can be removed

Activate currently inactive tests

Currently there are tests in Test.Cardano.Chain.Genesis.Json & Test.Cardano.Chain.Txp.Json that are not running. This issue brings the necessary types from the Byron release to make these tests run.

Use `cardano-prelude` from it's own repo

After moving cardano-prelude to an eponymous repo we need to make sure we can build referring to a specific git commit. We should also put the custom stack snapshot in this repo and other projects can refer to it via it's URL.

Add cardano-chain package

This package will contain all the types necessary to represent and deserailise the Block and Undo types from cardano-sl. This also involves clean up of that code to bring it in line with the conventions of the legacy-free codebase.

Model the `Lovelace` type in the ledger spec

At this moment Lovelace has no associated constraints, however we use the + operator, and we're assuming no negative balances. These assumptions should be written down explicitly in the model.

PR

Number Base
PR-210 develop

Consequences

Retrospective

ledger_test_validate_old

Test the blockchain layer validation and the ledger layer validation with

  1. existing mainnet chain; and
  2. executable spec and generator

Replace either (throwError . Error) pure

We have the pattern either (throwError . Error) pure in a bunch of places to lift between MonadError instances. We can instead use liftEither . first Error, which seems neater and less boilerplate-y.

Import first version of blockchain types from `cardano-sl`

The new implementation needs to be able to deserialise exisiting blockchains and serialise to the same format until a hard-fork is taken. The simplest way to ensure this is to import the types from the old blockchain along with their golden/round-trip tests. We can then modify and extend those types, ensuring that all tests pass and any new serialisation is done conditionally on the software version.

Write generators that generate blockchains according to the OBFT protocol

The blockchain operational rules should be weaker than the Ourobouros BFT rules, so that the current Cardano implementation and the OBFT one can be validate by these.

However, we should write generators that generate OBFT compatible chains (chains generated according to the OBFT rules), and write QuickCheck (or hedgehog) properties stating that the generated chains are compatible with the OBFT chains.

We do not need in principle rules for the OBFT generators, but they might be nice to have.

Build LaTeX specs with CI

There are makefiles inside cardano-chain/specs/ledger/latex/ and cardano-chain/specs/chain/latex/ which build pdfs. It would be nice to check these builds as a part of CI.

Safely remove Boundary blocks

Similarly to the removal of SSC types, we want to remove epoch boundary blocks, while maintaining compatibility with the old on-the-wire format. To do that we encode blocks, wrapped with the old tag for MainBlocks and we decode by either dropping boundary blocks or decoding MainBlocks as before.

To be able to check hashes of old style blocks, we only need to be able to hash the serialized representation of the header. We should be able to extract this using the new CBOR decoding-with-offsets work, without ever decoding to a Haskell type.

Add cardano-binary package

This is an import and clean up of the cardano-sl-binary package in the legacy codebase. The template haskell code should be removed, as it is better to have hand-written instances for that kind of thing, and TH makes cross-compilation more tricky.

Add conditional testing infrastructure

Not all tests should always be run. CI, developers, and QA all have different concerns, so we should design the tests to take that into account. This could be done by adding an argument to the test executable that tells us what situation we're in. Development should be the default, so stack test runs a reasonable set of development tests. CI and QA can then have special invocations to run more comprehensive or long-running tests, or e.g. adjust sizing parameters to generators.

Formalize the Byron update rules

Main Goal

Model the evolution of the BlockVersionData parameters in the ledger state.

Aspects to be modeled

  1. Authentication: the proposal needs to be properly signed.
  2. Authorization: the proposal needs to be signed by the genesis keys or one of the delegates.
  3. Voting: genesis key holders or delegates can only vote on a proposal after it is posted.
  4. Voting deadlines: voting ends with a majority of the voters (4/7 if we assume no state) agree on a proposal. Bear in mind that a rule that finishes the voting period after 4/7 positive votes will succeed to validate the existing chain only if the distribution of the stake is roughly equal (which might not be the case, and we'd have to revisit this rule).
  5. Block-version changes: when the voting ends, we need to update the block-chain parameters according to the proposed version.

Current subtasks

  • Determine which information do we need to keep track of in the ledger state:
    • A subset of the information in BlockVersionData.
    • Something else?
  • Determine whether software updates are related to protocol (BlockVersionData) updates.

References

ledger_spec_old

Deliverable: LaTeX and executable specifications of the blockchain and ledger layer for the (simplified) rules for the existing chain.

The aspects following aspects should be covered by the aforementioned specifications:

  • Delegation
    • Witnessing
    • Epochs
    • Activation
  • Updates
    • Witnessing
    • Proposal activation
    • Proposal deadline
    • Protocol parametes
  • Utxo
    • Balance and fees
    • Witnessing
  • Chain extension
    • Signers

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.