# the project

To provide continuity across the concepts I'm covering I'll use a simple 'textbook' solution to illustrate some cross-cutting components of a complete production deliverable.

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2F6ba5b81b96adabc22560294d2bbf050cdbfc2b13.png?generation=1630870450928904\&alt=media)

## What are we building?

We'll' focus on a simple pipeline consisting of:

* [x] A blocking REST service that sends emails to a vendor (SendGrid) who then sends them to customers
* [x] A webhook listener that receives information back from the vendor and stores it in Kafka
* [x] A connector that streams records out of Kafka and into a Database
* [x] A pyspark "sweeper" utility that picks up Dead Letter data in AWS S3 and stores it for use in AWS Athena

## Components

| **Component**   | **Description**                                                                                                                                                              |   |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - |
| SendGridMailer  | Our rest-service that listens for request to send email, then formats and sends to SendGrid                                                                                  |   |
| SendGrid        | An external service owned by Twilio that sends emails on behalf of its customers and then provides asynchronous information about how the customer interacted with the email |   |
| EventActivity   | Our second rest-service that accepts callbacks from SendGrid, and sends them to Kafka.                                                                                       |   |
| Confluent Kafka | A high-volume message broker                                                                                                                                                 |   |
| Connector       | There are many ways to get information from Kafka to a database.  In this example we'll store the data to parquet data on AWS S3                                             |   |