Data engineering
Streaming trip data with Kafka
A local streaming pipeline that sends generated trip events through Kafka and PostgreSQL to a live Streamlit dashboard.
The question
How do we get events from a running producer into a database and dashboard as they arrive? I used simulated ride-sharing trips to work through that flow.
The implementation
- Generated trip events in Python and published them to a Kafka topic.
- Consumed the events into PostgreSQL, using trip IDs to avoid inserting the same event twice.
- Built a dashboard with city filters, recent trips, and summaries of fares, distances, and vehicle types. Docker Compose starts the local services.
Results & discussion
The project connects a producer, message broker, database, and dashboard in one local example. The repository includes a screenshot and recording of the dashboard updating.
Limitations
The trips are synthetic and the system is a local demonstration. Kafka offsets and database writes are not committed as one transaction, so the example does not establish exactly-once processing or production reliability.
Project repository: Ammy Lin. The producer, consumer, and dashboard are separate Python scripts.
For the code and full project details, see the project repository.