iabtechlab / adscert Goto Github PK

Both the ticker, and the time to renew are both hardcoded values in the dns refresh loops.
https://github.com/IABTechLab/adscert/blob/491ee311698f05459bb13d0c83ec1d5eb1affdc8/pkg/adscert/discovery/domain_indexer_impl.go

It would be good for these to be configurable by the users.

Support DNSSEC for domain data lookups

Does Prebid Support ads.cert?

Initial separation has been done with the following architecture:

• Dns Resolver - Discovers TXT records for a domain name (with the appropriate subdomains for policy and key records)
• Domain Store - Stores domain information for invoking and identity domains and the associated keys
• Domain Indexer - Uses the above components to maintain a list of domains used in signing/verification operations, runs background crawls to update domain records, stores private keys, calculates shared secrets, and is used by the Signatory for signing and verification operations.

Domain Indexer can be further separated with different components for private key management (see issue #31), background crawling, and data persistence.

Tried to set this up and run - seems there are lot of refactoring and updates to code parameters which are not reflected in README.

e.g This command doesn’t work

go run examples/signer-server.go --frequency 5s --logtostderr --body '{"sample": "request"}' --origin_callsign=ssai-serving.tk --url='http://ads.ad-exchange.tk:8090/request?param1=example&param2=another' --send_requests

The path and params have changed to

go run examples/signer-server/signer-server.go -frequency 5s --body '{"sample": "request"}' -origin=ssai-serving.tk -url='http://ads.ad-exchange.tk:8090/request?param1=example&param2=another' -send_requests

I will try to fix as much of this as possible and raise a PR. I would request code contributors to please update the readme as well.

Additionally I see there is no working domain with that I can test with
ssai-serving.tk doesn’t work

No record found for _delivery._adscert.ssai.tk in 126.8093ms: lookup _delivery._adscert.ssai-serving.tk on 8.8.8.8:53: no such host
2022/04/09 00:06:16 counterparty lookup error
2022/04/09 00:06:16 SIGNATURE_DECODE_STATUS_COUNTERPARTY_LOOKUP_ERROR

Neither does publica-ssai.com doesn’t work - seems there is private key mismatch

2022/04/09 00:25:09 Found records for _delivery._adscert.publica-ssai.com in 63.009852ms: [v=adcrtd k=x25519 h=sha256 p=StWZV1kkgId0iOAmVLutWhWSv6SGuM-pdavckWrQKXY]
2022/04/09 00:25:21 SIGNATURE_DECODE_STATUS_INVALID_SIGNATURE

Can someone provide steps to get a bare minimum domain and key setup for validating this. Or am I missing something?

This project is currently implemented to handle a single policy record or key record for a given domain, and should be updated to handle multiple.

Use key derivation to create multiple levels of shared secrets for different purposes and to secure different details.

Create a Verification endpoint that handles multiple requests in a single batch to help in offline processing or high traffic systems.

Should handle after #28 (multiple signatures per single request) to avoid cascading changes.

Current implementation uses an in-memory store for domain data for simplicity and performance.

There should multiple storage implementations for various scaling (single server, K8S, distributed data center, etc) and security (encrypted volumes, KMS, etc) concerns. There should also be an audit trail implementation to keep track of data mutations over time and archive historical records.

There are currently multiple layers of status and error codes throughout the adscert project:

• SignatureStatus - used in the AuthenticatedConnectionSignatureResponse to signal signing operations
• VerificationStatus - used in the AuthenticatedConnectionVerificationResponse to signal verifying operations
• AuthenticatedConnectionProtocolStatus used for both the signature message and in DomainStore records to denote the current state of domain data.

Operation status codes are very broad and should have more states so that responses are as helpful as possible.

AuthenticatedConnectionProtocolStatus is very overloaded. It doesn't line up with the usages (domain data and signature message) and may be conflicting with the operation status in a signature response.

ads.cert 2.0 Release pipeline proposal

Objective

Define release pipeline for ads.cert 2.0

Note: This doc does not cover details of each stage of the pipeline.

Guiding principles and Goals

• Automate release process
• Support adding multiple stages to the pipeline (build, test, vulnerability/security check, etc.)
• Build and publish binaries if applicable
• Build and publish versions and tag releases
• Create CHANGELOG entries on releases
• Modify README and publish go docs automatically on releases
• Publish deployment docs
• Configurable workflows based on different branches (main, staging, develop, etc.)

Nice to have:

• Push Slack deployment notifications
• Deploy Docker images to Docker Hub if applicable

Proposal

Since ads.cert is going to be hosted on a public github repository, we can leverage github actions and workflows to build a CI/CD pipeline since actions integrate seamlessly with github and source is also open source. Some advantages and downsides are:

Advantages

• Ease of setup (no installation is required).
• Tight integration with github.
• Asynchronous CI/CD
• Support for many languages and frameworks (including go)
• The individual actions in a workflow are isolated by default. We can use a completely different computing environment for different stages like compilation and testing. For example Travis CI (and most other "traditional" CIs) would run all stages in the same computing environment.

Disadvantages

• No native cache (need to implement if needed)
• GitHub actions are just consecutive docker runs. Very easy to reason about and debug. But it is a single platform, just whatever you can run inside docker. That seems unlikely to change, but might be an acceptable limitation (we can always add an action to call other cross-platform CI/CD services).

Alternative

The most reliable and popular options are Travis CI and Jenkis but they are not free. Task are also synchronous and they need installation.

Setting up Workflows

Workflows are custom automated processes that we can include in the repo to build, test, and deploy the code. We can have one or more workflows running in parallel. Workflows need be stored in .github/workflows.

Overall flow diagram

Each workflow has a name. We can then configure events to trigger the workflow

on:
 push:
   branches:
   — master
   — staging
   — develop
 pull_request:
   branches:
   — master
   — staging
   — develop

Each stage can be run as a job which runs on a runner

jobs:
 # The “build” workflow
 build:

runs-on: ubuntu-latest

In each job, we can add steps which represent a sequence of tasks to be executed as part of the job

steps:
 — uses: actions/checkout@v2

Setup environment to use

- name: Setup Go
 uses: actions/setup-go@v2
 with:
   go-version: ‘1.14.0’

Install dependencies

- name: Install dependencies
 run: |
   go version
   go get -u golang.org/x/lint/golint

Packaging and building the application

- name: Run build
 run: go build .

Run vet and lint

- name: Run vet & lint
 run: |
   go vet .
   golint .

Run tests

- name: Run testing
run: cd test && go test -v

Slack notifications

- name: Send slack notification
 uses: 8398a7/action-slack@v3
 with:
   status: ${{ job.status }}
   fields: repo,message,commit,author,action,eventName,ref,workflow,job,took
 env:
   SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
 if: always()

Deploy Docker image to Docker Hub

deploy:
  runs-on: ubuntu-latest
  needs: [build]
  if: ${{ github.ref == ‘refs/heads/master’ && github.event_name == ‘push’ }} 
  steps:
  - uses: actions/checkout@v2
  - name: Deploy to Docker registry
    uses: docker/build-push-action@v1
    with:
      username: ${{ secrets.DOCKER_USERNAME }}
      password: ${{ secrets.DOCKER_PASSWORD }}
      repository: test/gosimple
      tag_with_ref: true

Overview of workflows and running jobs

Putting it all together

name: CICD

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
  push:
    branches:
    - master
    - staging
    - develop
  pull_request:
    branches:
    - master
    - staging
    - develop

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # The "build" workflow
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2
    
    # Setup Go
    - name: Setup Go
      uses: actions/setup-go@v2
      with:
        go-version: '1.14.0' 
    
    # Install all the dependencies
    - name: Install dependencies
      run: |
        go version
        go get -u golang.org/x/lint/golint
        
    # Run build of the application
    - name: Run build
      run: go build . 
      
    # Run vet & lint on the code
    - name: Run vet & lint
      run: |
        go vet .
        golint .
    
    # Run testing on the code
    - name: Run testing
      run: cd test && go test -v
    
    # Send slack notification
    - name: Send slack notification
      uses: 8398a7/action-slack@v3
      with:
        status: ${{ job.status }}
        fields: repo,message,commit,author,action,eventName,ref,workflow,job,took # selectable (default: repo,message)
      env:
        SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} # required
      if: always() # Pick up events even if the job fails or is canceled.

  # The "deploy" workflow
  deploy:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    needs: [build] # Only run this workflow when "build" workflow succeeds
    if: ${{ github.ref == 'refs/heads/master' && github.event_name == 'push' }} # Only run this workflow if it is master branch on push event
    steps:
    - uses: actions/checkout@v2

    # Deploy to Docker registry
    - name: Deploy to Docker registry
      uses: docker/build-push-action@v1
      with:
        username: ${{ secrets.DOCKER_USERNAME }}
        password: ${{ secrets.DOCKER_PASSWORD }}
        repository: test/gosimple
        tag_with_ref: true

Automate changelog and releases notes

  changelog:
    name: Update Changelog
    runs-on: ubuntu-latest
    steps:
      - name: Get version from tag
        env:
          GITHUB_REF: ${{ github.ref }}
        run: |
          export CURRENT_VERSION=${GITHUB_TAG/refs\/tags\/v/}
          echo "::set-env name=CURRENT_VERSION::$CURRENT_VERSION"
      - name: Checkout code
        uses: actions/checkout@v2
        with:
          ref: main
      - name: Update Changelog
        uses: heinrichreimer/[email protected]
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          issues: true
          issuesWoLabels: true
          pullRequests: true
          prWoLabels: true
          addSections: '{"documentation":{"prefix":"**Documentation:**","labels":["documentation"]}}'
      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Update Changelog for tag ${{ env.CURRENT_VERSION }}
          file_pattern: CHANGELOG.md

  release_notes:
    name: Create Release Notes
    runs-on: ubuntu-latest
    needs: changelog
    steps:
      - name: Get version from tag
        env:
          GITHUB_REF: ${{ github.ref }}
        run: |
          export CURRENT_VERSION=${GITHUB_TAG/refs\/tags\/v/}
          echo "::set-env name=CURRENT_VERSION::$CURRENT_VERSION"
      - name: Checkout code
        uses: actions/checkout@v2
        with:
          ref: main

      - name: Get Changelog Entry
        id: changelog_reader
        uses: mindsers/changelog-reader-action@v1
        with:
          version: ${{ env.CURRENT_VERSION }}
          path: ./CHANGELOG.md

      - name: Create Release
        id: create_release
        uses: actions/create-release@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 
        with:
          tag_name: ${{ github.ref }}
          release_name: Release ${{ github.ref }}
          body: ${{ steps.changelog_reader.outputs.log_entry }}
          draft: false
          prerelease: false

Generate go doc

Go code that we upload to public repositories on github appears automatically on the godoc website.

We can also define a job for generating docs

on: [pull_request]

jobs:
  main:
    runs-on: ubuntu-latest
    name: Example
    steps:
    - name: Checkout
      uses: actions/checkout@v1
    - name: Generate GoDoc
      uses: ktr0731/[email protected]
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    - name: Push changes
      uses: ad-m/github-push-action@master
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        branch: gh-pages

References

https://docs.github.com/en/actions
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions
https://github.com/ktr0731/godoc-action
https://pkg.go.dev/github.com/posener/goreadme
https://github.com/renehernandez/camper/blob/main/.github/workflows/release.yml
https://medium.com/swlh/setting-up-github-actions-for-go-project-ea84f4ed3a40

Create allowlist/denylist filters to limit processing to accepted domains and counterparties.

Fill in remaining sections for these materials and ensure that they're up-to-date.

Private keys are currently read from environment variables or command line flags. This can be improved to interface with other dedicated key management systems (KMS) to improve security and access control.

Support additional options during verification operations:

• Return as quickly as possible or block until all domains have been looked up.

There can be multiple keys and counterparties involved in a single request. Verification should handle multiple signatures.

Ideally we should surface the details about the signature/verification operation outcome in the API and not rely on log statements that were added for the POC. Change the API and implementation to surface these.

Currently this isn't working correctly, so the demo doesn't always work. For the beta test, we'll only worry about the domain having one key present. Longer term, we'll upgrade to what's outlined in the design doc for key management and rotation.