1

I'm building a social app similar to Telegram using NestJS, PostgreSQL, and Prisma.

In one part of the app, I need to create a high volume of posts (e.g., thousands at once).

To prevent performance issues or crashes, I decided to use a queue-based approach.

This is my first time implementing queues.

I'm using Bull with NestJS. Here's how I'm currently adding jobs to the queue:

     for (const phone of inputPhones) {
        const params: CreatePostQueueDto = {
          channelId,
          title,
          content,
          phone,
          postGroupId: postGroup.id,
          postType,
          media,
          creatorId,
        };
        void this.QueueService.createPostJob(params);
      }

Queue service:

  async createPostJob(data: CreatePostQueueDto) {
    await this.mainQueue.add('create_post', data);
  }

Queue processor:

@Processor('main')
@Injectable()
export class QueueProcessor {

  @Process('create_audience')

  @Process('create_post')
  async handleCreatePost(job: Job<CreatePostQueueDto>) {
    try {
// create post
    } catch (error) {
      console.log(error);
      throw new BadRequestException(`Failed to process create_post: ${error}`);
    }
  }

My Questions:

  1. What is the best practice for creating a large number of posts using Bull in NestJS? Is there anything I should change to improve reliability, performance, or error handling?

  2. Where should I handle charging the user (e.g., deducting credits or balance)? Should I do it before queueing the jobs?

I have to charge user once for all successful posts ( some post may not be successful for some reasons and we don't have exact number of created post until finish)

Or should I handle it inside the queue processor. Thank you!

1 Answer 1

0

For large number of processes, currently you are using just 1 queue for creating 1000 jobs at once. Try to create from 2 3 or 4 more queues. I have never used BullMQ, but in RabbitMQ it has exchange which is how you process jobs for multiple queues not just all at once.

And you should charge user after create a post, it is non-sense if charging and then the create fails. From what you have said I have to charge user once for all successful posts ( some post may not be successful for some reasons and we don't have exact number of created post until finish). I assume that a user can create multiple posts at once and if one fails you cannot finish it. When some posts fail, you can set the **nAck (**Negative acknowledge) then make the fails one dont loop permanently. Charge for successful ones, for fail ones you can make the user know if it can restore or re-creating it.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.