I'm building a social app similar to Telegram using NestJS, PostgreSQL, and Prisma.
In one part of the app, I need to create a high volume of posts (e.g., thousands at once).
To prevent performance issues or crashes, I decided to use a queue-based approach.
This is my first time implementing queues.
I'm using Bull with NestJS. Here's how I'm currently adding jobs to the queue:
for (const phone of inputPhones) {
const params: CreatePostQueueDto = {
channelId,
title,
content,
phone,
postGroupId: postGroup.id,
postType,
media,
creatorId,
};
void this.QueueService.createPostJob(params);
}
Queue service:
async createPostJob(data: CreatePostQueueDto) {
await this.mainQueue.add('create_post', data);
}
Queue processor:
@Processor('main')
@Injectable()
export class QueueProcessor {
@Process('create_audience')
@Process('create_post')
async handleCreatePost(job: Job<CreatePostQueueDto>) {
try {
// create post
} catch (error) {
console.log(error);
throw new BadRequestException(`Failed to process create_post: ${error}`);
}
}
My Questions:
What is the best practice for creating a large number of posts using Bull in NestJS? Is there anything I should change to improve reliability, performance, or error handling?
Where should I handle charging the user (e.g., deducting credits or balance)? Should I do it before queueing the jobs?
I have to charge user once for all successful posts ( some post may not be successful for some reasons and we don't have exact number of created post until finish)
Or should I handle it inside the queue processor. Thank you!