This provides a docker compose environment to quickly spin up a Spark, Iceberg REST Catalog, MinIO, dbt and soon more. Great for testing ideas and learning.

note: If you don't have docker installed, you can head over to the Get Docker page for installation instructions.

Start up the docker compose environment by running the following.

docker-compose build

docker-compose up

This opens up the following ports:

• MinIO UI http://localhost:9000 use admin for username and password for password
• Iceberg REST Catalog http://localhost:8181
• Spark UI http://localhost:4040
• Spark Cluster http://localhost:7077
• Spark Driver UI http://localhost:8080
• Spark Thrift Server http://localhost:10000
• Spark History http://localhost:18080
• Jupyter Notebook http://localhost:8888

I have found the best way to interact with these services is opening up a VS Code dev container by attaching to a running docker container or by using the following UIs specified above.

To stop everything

docker-compose down

• Spark
• Iceberg REST Catalog
• MinIO
• dbt
• Trino

• DuckDB

• Airflow
• Dagster

• Meltano
• Airbyte

• Kafka
• Redpanda

• Flink
• bytewax
• Materialize

• cube

• Metabase
• Rill Data
• Evidence
• Streamlit

• Iceberg's REST Catalog: A Spark Demo
• GitHub Repo Tabluar Docker-Spark-Iceberg
• Iceberg + Spark + Trino + Dagster: modern, open-source data stack demo
• GitHub Repo Jaffle Shop DuckDB
• Using dbt with Tabular