Realtime Data Streaming

The project is designed with the following components:

Data Source: Used randomuser.me API to generate random user data for our pipeline.
Apache Airflow: Responsible for orchestrating the pipeline and storing fetched data in a PostgreSQL database.
Apache Kafka and Zookeeper: Used for streaming data from PostgreSQL to the processing engine.
Control Center and Schema Registry: Helps in monitoring and schema management of our Kafka streams.
Apache Spark: For data processing with its master and worker nodes.
Cassandra: Where the processed data will be stored.

Technologies

Name		Name	Last commit message	Last commit date
dags		dags
logs/scheduler/2025-01-09		logs/scheduler/2025-01-09
script		script
.gitattributes		.gitattributes
Data engineering architecture .png		Data engineering architecture .png
README.md		README.md
docker-compose.yml		docker-compose.yml
requirements.txt		requirements.txt
spark_stream.py		spark_stream.py