The goal of drem is to:

• Download Dublin energy-related data (from SEAI, the CSO etc.)
• Transform this data into a bottom-up energy demand model for Dublin

drem uses open-source software and open-access data to enable:

• Reproducibility
• Usage of 'Live'/up-to-date data sources

• To install drem:

  Warning: you must accept all permissions-related requests made by VSCode and docker (to unblock your firewall and grant access to your C-Drive)

  • Install docker

    drem needs Docker Desktop to be running. You should see a small docker whale in your toolbar after installation!

    If you have trouble installing docker please see FAQ

    

  • Install Microsoft Visual Studio Code (VSCode)

  • Download drem as a ZIP (by selecting the green Code button at the top of the drem github page) and unzip it

    

    If you are familiar with git see Setup development environment

  • Open the drem folder in VSCode by selecting 'File > Open Folder'

  • Select 'Extensions' on the side-bar and install the “Remote - Containers” extension (or install directly from here)

    

  • Reopen the drem folder in a container by selecting 'Reopen in Container' in the popup menu or by opening the Command Palette (via View > Command Palette or by Ctrl + Shft + P) and searching 'Remote-Containers: Reopen in Container'

    If you want to view the installation progress click on 'Starting Dev Container (show log)' and once the installation is complete (files will appear in the left sidebar) click the + symbol in your Terminal window.

    

• To run drem (and consequently download & transform all drem related data):

  • Launch Jupyter Notebook:

    • Enter jnbook in the VSCode Terminal

      

    • Copy and paste the resulting URL to your browser (or Ctrl + Click)

      It should look like http://127.0.0.1:8888/?token=aa69433d1370ab87a15436c27cd3f6948f77539a6bbeb6ee

  • Open run-drem.ipynb and ...

    • Register your email address with SEAI at https://ndber.seai.ie/BERResearchTool/Register/Register.aspx

    • Fill in your email address in the notebook

    • Run all cells by selecting Cell > Run (or by manually running each cell via the Run button or by clicking each cell followed by Shft + Enter)

      

• To update your local drem code:

  • Re-download drem as a ZIP
  • (Optional) Copy & paste the files in drem/data/external across from the old version to the new version to skip redownloading of files

• prefect to orchestrate all drem tasks via a data pipeline
• pandas to transform columnar data
• geopandas to transform columnar geospatial data
• requests to download data
• pypostal to standardise and parse address string columns
• docker to create a reproducible build environment that runs on Windows, OSX and Linux
• git to track code changes

• Residential buildings:

  • SEAI's 2016 Census Small Area Statistics, Geometries & Glossary.

  • SEAI's BER Public Search: dwelling fabric information.

  • Dublin Postcodes Geometries (created by Shane McGuinness of Trinity College Dublin).

  • CRU Smart Meter Trials 2009-10: 15-minute resolution demands & participant surveys ... available upon request.

• Commercial:

  • Valuation Office data: commercial building floor areas etc.

  • CIBSE Energy Benchmarks

  • SEAI Dublin Measurement & Verification data: annual gas/electricity demands for Public sector buildings ... available upon request.

See energy-modelling-ireland/energy-data-sources for more Irish-specific energy sources.

Here's a brief overview of what each file and directory in drem does:

│
├── .github                 <- Scripts to run Github Actions CI
├── src                     <- Source code for use in this project.
│   ├── extract/            <- Scripts to download data
│   ├── transform/          <- ""         clean data
│   ├── load/               <- ""         load data to files or databases
│   └── etl.py              <- Orchestrates Extract, Transform, Load via prefect
│
├── tests                   <- Scripts to test src code via pytest
│
├── data
│   ├── external            <- Data from third party sources
│   ├── interim             <- Intermediate data that has been transformed
│   ├── processed           <- Final, canonical data sets for modeling
│   └── raw                 <- Original, immutable data dump (closed source)
│
├── externals               <- External libraries used by drem
├── .gitignore              <- Specifies files and folders to be ignored by source control
├── .pre-commit-config.yaml <- pre-commit hooks
├── LICENSE                 <- Terms & conditions for library usage etc.
├── README.md               <- Executive Summary of library
├── poetry.lock             <- Used by Poetry to store dependencies
├── pyproject.toml          <- ""             to setup library
├── Dockerfile              <- Used by docker to create the drem development environment
└── setup.cfg               <- Used by flake8 for linting style

Inspired by cookiecutter-data-science

For more information see:

• Extract, Transform, Load with Prefect
• Pytest
• Github Actions for Continuous Integration (CI)
• Pre-commit hooks
• Poetry for library setup
• flake8, we-make-python-style-guide for checking style issues
• externals
  • libpostal enables fuzzy address matching
  • nominatim-docker enables creation of local Nominatim server for geocoding at scale via OpenStreetMaps

If you've never contributed to an open-source project before checkout (or are new to git) first-contributions. They even have a VSCode-specific section

• Clone drem instead of downloading a zip file as git lets you update drem without having to manually redownload it:

  • via git clone https://github.com/codema-dev/drem

  • or via VSCode:

    • Select 'Source Control' on the side-bar
    • Select 'Clone Repository' and search codema-dev/drem

    

• Set your local VSCode Python Interpretor to your local poetry virtualenv Python

  • Run poetry shell (to entervirtualenv) followed by which python (to get the filepath to virtualenv Python)
  • Set your Python Interpretor by opening the Command Palette (via View > Command Palette or by Ctrl + Shft + P), searching “Python: Select Interpreter” and copying & pasting the resulting filepath

• Change your local VSCode settings to codema-dev by creating a local .vscode folder and copying & pasting this settings.json

• Install local dev dependencies via poetry install

• To update your local drem with the latest code:

  If you change any file git pull may result in merge conflicts, seefirst-contributions or the VSCode docs

  • via git pull https://github.com/codema-dev/drem

  • or via VSCode

    • Select 'Source Control' in the side-bar
    • Select the 3 dots in the top right hand corner of the popup
    • Select 'Pull' to merge in the latest changes

    

For more information see Developing inside a Container guide

• Install WSL2 for Docker Desktop on Windows:

  As of 10/11/2020 Docker Desktop raises the following pop-up on Windows:

  

  • Following the pop-up instructions:

    • Launch Powershell as an Administrator search your files for 'Powershell' and either select or right-click 'Run as Administrator':

      

    • Copy & paste the suggested command Enable-WindowsOptionalFeature -Online -FeatureName $("VirtualMachinePlatform", "Microsoft-Windows-Subsystem-Linux") and ENTER

      

  • Restart docker

    

  • 'Docker Desktop' should raise the following pop-up, follow link to complete installation

    

  • The 'Docker Desktop' whale should indicate that 'Docker Desktop' is running next time you launch it 😃