Giter VIP home page Giter VIP logo

getting-started-with-analytics-engineering's Introduction

Getting Started with Analytics Engineering

Files/Folders and What They Do

Name Description
.devcontainer/devcontainer.json A file that tells VS Code how to access (or create) a development container with a well-defined tool and runtime stack
csv_files/ A folder that contains CSV files used for testing dbt seed
setup/ A folder that contains data and scripts used for initializing the Postgres database used in this project
yaml_files/ A folder that contains the YAML file examples such as profiles.yml or src.yml
.gitignore A file that specifies intentionally untracked files that Git should ignore
LICENSE A license of this repo
Makefile A Makefile file which defines set of tasks to be executed
README.md README file that provides the setup instruction on this project
docker-compose.yml A Docker Compose file that runs a Postgres database and SQLPad used in this project
greenery-dbdiagram.txt A file that provides code for drawing an ER diagram on dbdiagram.io
requirements.txt A file that contains Python package dependencies for this code repository

Getting Started

To start the Docker compose:

make up

To Set Up and Activate Your Python Virtual Environment

python -m venv ENV
source ENV/bin/activate
pip install -r requirements.txt

To Initialize A dbt Project

dbt init

Note: Let's specify the project name greenery.

To Set Up Your dbt Profile

  1. Change the directory to your dbt project.

    cd <dbt_project_name>
  2. Run the following command to copy the profiles example file to the real profiles file in the project folder.

    cp ../yaml_files/profiles.yml .
  3. Edit the content in the profiles.yml file by changing the output and target to your name (e.g., dbt_john), and save. See the example below.

    greenery:
    
      outputs:
        dbt_zkan:
          type: postgres
          threads: 1
          host: localhost
          port: 5432
          user: postgres
          pass: "{{ env_var('DBT_ENV_SECRET_PG_PASSWORD') }}"
          dbname: greenery
          schema: dbt_zkan
    
        prod:
          type: postgres
          threads: 1
          host: localhost
          port: 5432
          user: postgres
          pass: "{{ env_var('DBT_ENV_SECRET_PG_PASSWORD') }}"
          dbname: greenery
          schema: prod
    
      target: dbt_zkan
  4. Set the environment variable.

    export DBT_ENV_SECRET_PG_PASSWORD=postgres
  5. We then should be able to use dbt now. :-)

To Debug The dbt Project

export DBT_ENV_SECRET_PG_PASSWORD=postgres
cd <dbt_project_name>
dbt debug

To Create Your Data Models

export DBT_ENV_SECRET_PG_PASSWORD=postgres
cd <dbt_project_name>
dbt run

To Test Your Data Models

export DBT_ENV_SECRET_PG_PASSWORD=postgres
cd <dbt_project_name>
dbt test

To Generate The dbt Documentation and Serve It

export DBT_ENV_SECRET_PG_PASSWORD=postgres
cd <dbt_project_name>
dbt docs generate
dbt docs serve

getting-started-with-analytics-engineering's People

Contributors

atbee avatar zkan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.