Bulk download of training certificates

Context

At Beedeez, learners earned a certificate at the end of a training path, based on their answers to the various quizzes. These PDF certificates were generated on the server, and each learner could view and download theirs. Administrators, for their part, could download the certificates of all their users at once.

The system had to handle bulk certificate downloads. Generating everything within a single HTTP request was not an option: the request could time out, and a dropped connection midway would have thrown all the work away. On the backend, we also had to be able to generate and store a large number of certificates before sending them.

Approach

I separated the download request from the generation itself: the server starts the work in the background and responds right away, then the browser regularly asks how the export is going.

Download request

An administrator, from the back office

↓

Browser

↓

Polling starts↻

Asks for the export status at a regular interval

↓

Response: “in progress”

Polling continues until the zip is ready

Server

↓

BullMQ queueRedis

One job per batch of certificates

↓

Workers×N

Generate each batch's certificates

↓

Certificates on S3

Stored temporarily until everything is generated

↓

Zip creation

Once every job has finished, also stored on S3

↓

Response: “done”

The next poll receives a signed URL to the zip and stops

↓

✓Zip download

Straight from S3, without going through our server

Backend: a BullMQ queue

When an administrator requested the download, the server added jobs to a BullMQ queue (backed by Redis), each job covering one batch of certificates. Splitting the work into batches avoided a single long, memory-hungry job for large accounts.

A worker then processed these jobs, generated the PDFs and stored them temporarily on AWS S3 while the rest were generated. Once every job had finished, the certificates were bundled into a zip, also uploaded to S3, and the export moved to a "completed" status.

Frontend: polling

On the frontend, I built a polling system in TypeScript. The client sent a request to the backend at a regular interval to check the export status. Once the zip was ready, the response contained a signed S3 URL. This temporary link made it possible to download the zip straight from S3, without routing it through our server. Polling stopped after a set delay (timeout), so the client would not query the server forever if an export got stuck.

Since generation runs on the server, independently of the client's connection, a network drop does not interrupt the work.

Outcome

Administrators could get the certificates of all their users in a single zip, without relying on one request that might time out. Splitting the work into batches improved performance when generating data in bulk.

Takeaways

• As soon as a task may outlast an HTTP request, it is better to hand it to a queue and expose a status. The work then no longer depends on the client's connection.
• To generate a large volume of files, splitting the work into batches improves performance.