CSV Email Vaildator

A web service that ingests csv files and parses them according to an email validation service.

Set Up and Usage

• Install Node Modules
• npm run start

Development Notes

I have designed the system with a layered architecture. The layers are:

• Route middlewares for presentation, processing the inputted request and generating output. The routes are composed in routes/ and there are middleware factories in handlers/.
• Data Layer, an abstracted repository layer for file storage and database interactions. In this implementation, I have used an in-memory data store, but a redis or sql repository could be implemented. The repository interface contains factory methods for creating UploadData classes injected with implementations of the repository. The filestore is abstracted as well, if temporary file storage should be moved away from a location in the service.
• Business logic layers, to convert between provided data and stored data. The UploadData fetches data via it's repository and computes business logic. The UploadProcessor handles the bulk of the work, processing uploads.

These layers allow different layers to be injected with environment-specific implementations. We can unit test the business logic without dealing with production implementations. Each layer provides its own error handling to separate their concerns and allow for custom error recovery.

Endpoints

There are three endpoints:

• POST /upload an endpoint that accepts requests of encoding type multipart/form-data with a single field, csv. The csv should be in the shape according to the spec. If the file is accepted, the server will respond with an OK and the id of the upload, before processing the upload (the Initial Response in the spec). The server does not send a second response on this endpoint.

• GET /status/:uploadId an endpoint that accepts a single upload id and returns the status of that upload. If the upload is being processed, it will return the id of the upload and the percentage progress. If the upload has finished processing, it will return the information about the processed upload and any failed records, including those which failed due to severe errors such as validation service timeout.

• GET /status/all an endpoint that returns the status, as described above, for all uploads that are in the datastore.

Concurrency, Rate Limiting and Logging

The EmailValidatorService handles concurrency using p-limit, sending requests to the injected email validation source.

The app uses a middleware from express-rate-limiter to provide rate limiting on all inbound requests.

The app uses a logger generated by winston and handles appropriate logging levels. Logger occurs inside of business logic or router middleware and not inside of injected abstractions, to ensure the logging is not dependent on implementation inside of logging.

Error Handing

The middlewares handle input validation of the request and file, responding with appropriate errors to the request.

The UploadProcessor handles errors during processing of the valid csv. If an error occurs, it is recorded in the repository for that Upload, so that the /status endpoint can retrieve it. We also log this error out.

The service also catches all uncaught errors to prevent any escaped errors from causing the service to crash, only crashing the execution thread that caused the error.

Testing

• To run the tests, run npm run test
• Unit Testing for business logic. Mock services can be injected to control the tests. The EmailValidatorService can be mocked to test if the UploadService can handle errors.

Improvements / Extensions

• Supply a path to a config file that details fileStore file paths, rate and concurrent limiting and server port numbers.
• Add happy path api integration tests.
• Add configuration for lint (and prettier), adding lint and typescript checking to a pipeline for continuous integration.