dbt-containerized-projects

Architectural Diagram

Requirements

docker, docker-compose

Setup

docker-compose build
docker-compose up -d
docker exec -it dbt-containerized-projects_dbt-services_1 bash

dbt-commands

cd dbt_postgres
source bin/activate
cd dbt_project/
dbt seed 
dbt compile 
dbt debug
dbt run 


cd dbt_oracle
source bin/activate
cd dbt_oracle_project
dbt seed 
dbt compile 
dbt debug
dbt run 


cd dbt_spark
source bin/activate
cd dbt_spark_project/
dbt seed 
dbt compile 
dbt debug
dbt run 

cd dbt_mssql
source bin/activate
cd dbt_microsoft_sql_server/
dbt seed 
dbt compile 
dbt debug
dbt run 

databases

Postgres

Image of postgres DB

DBT command for dbt run

Oracle

You need to create a Schema DBT_STAGING and have to run below query to give sufficient permission

ALTER USER DBT_STAGING quota 100M on USERS;

Image of oracle db

DBT command for dbt run

Spark Cluster

Note In order to run spark cluster these below scripts must be start(these already automated in docker file)

• /usr/local/share/spark/sbin/start-master.sh
• /usr/local/share/spark/sbin/start-thriftserver.sh

Only master node is created with below details Image of Master Spark Cluster

DBT command for dbt run

Image of Cluster Default Database

MSSQL DB

Image of MSSQL db