Why is spark not mentioned in docker compose file but its there on the Architecture diagram about sales-forecast-mlops-at-scale HOT 5 CLOSED

It indeed uses Spark, but I haven't set up a dedicated Spark cluster for this. The reason is that Spark's task in this system is very light and doesn't need scalability. The system is already resource-intensive (with Kafka, Airflow, and Ray) so I don't want to bloat it further if not necessary.

You can find the Spark code in /services/airflow/dags. It's used within a DAG and executed by Airflow workers.

from sales-forecast-mlops-at-scale.

Okay, I understand now. I'm trying to figure out why I'm encountering the following problem. I'm having a challenge with Docker here. When I try to set everything up, it works, but the issue is that sometimes it's very slow to open certain ports, while others don't open and just keep loading, especially pgAdmin. Sometimes I can log in, but I can't see the data in the database. So, I'm thinking maybe something is wrong with the DAGs in Airflow as they keep failing. Then for the machine I'm using 4 cores, 16GB RAM, and 32GB on GitHub Codespaces. What could be the problem? Is it the machine is slow, or am I missing something here?

from sales-forecast-mlops-at-scale.

I recommend you look at the logs from all services when you docker-compose up. If something went wrong, you should see the warnings or errors there. With only this information, It's hard to say what went wrong in your case. If you suspect the spec of your machine, you can try docker-compose up without Airflow first and see if other things work fine. Then you can start Airflow after that and try running a DAG.

For pgAdmin, you have to connect to the database server manually after you start the system. Otherwise, there won't be anything to show there. You can surely find a quick tutorial on this from Google or YouTube.

from sales-forecast-mlops-at-scale.

For pgAdmin it has already worked now i have to check on the logs and try running without airflow.Thank you this is a great project.I appreciate

from sales-forecast-mlops-at-scale.

Great to hear that! I'm closing the issue now but feel free to reopen or create a new one if you face any further issues.

from sales-forecast-mlops-at-scale.