@
Problem with multiple spawned workers
Updated on April 11, 2018 in Async Background Worker
Unfollow
sampo 1 on April 10, 2018

Hi, just gave your plugin a spin and it seems to work great. The only thing that is bothering me is scalability.

For example, I have 6 WordPress instances running a single site. All those instances have one background-worker listen-loop. That makes for 6 workers. Now I put 20 jobs in queue. The workers start to hog the jobs and run each parallel. That’s not what I want.

Here’s how to simulate the situation:
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &

Now imagine a situation where 16 people would send form data to an endpoint simultaneously and would get an email confirmation. My workers respond:
Working on job ID = 7
Working on job ID = 7
Working on job ID = 7
Working on job ID = 8
Working on job ID = 8
Working on job ID = 8
Working on job ID = 9
Working on job ID = 9
Working on job ID = 9
Working on job ID = 10
Working on job ID = 10
Working on job ID = 10
Working on job ID = 11
Working on job ID = 11
Working on job ID = 11
Working on job ID = 12
Working on job ID = 12
Working on job ID = 12
Working on job ID = 13
Working on job ID = 13
Working on job ID = 13
Working on job ID = 14
Working on job ID = 14
Working on job ID = 14
Working on job ID = 15
Working on job ID = 15
Working on job ID = 15
Working on job ID = 16
Working on job ID = 16
Working on job ID = 16
Working on job ID = 17
Working on job ID = 17
Working on job ID = 17
Working on job ID = 18
Working on job ID = 18
Working on job ID = 18
Working on job ID = 19
Working on job ID = 19
Working on job ID = 19
Working on job ID = 20
Working on job ID = 20
Working on job ID = 20
Working on job ID = 21
Working on job ID = 21
Working on job ID = 21
Working on job ID = 22
Working on job ID = 22
Working on job ID = 22
Working on job ID = 23
Working on job ID = 23
Working on job ID = 23

Job well done but too many emails were sent. And if the job took any longer, it would be 16*6 = 96 times.

What I would like to see is some kind of locking system for each task, that would probably solve the problem.

  • Liked by
Support Hours:
Weekdays: 09.00 - 17.00
Weekend: Off
Sorry, our support is not available at this time. We will reply at the next working hour.

By using the forum you are agree with tonjoo studio license agreement

Cancel
todiadiyatmo Moderator 0 on April 11, 2018

For now you must find mechanism to lock the row. Ideally one worker only work for 1 queue for now.<br />
<br />
The async worker actualy support `queue` parameter on the database (which mean you can have multiple separate queue) but it is not fully implemented yet, feel free to pull request : https://github.com/tonjoo/async-background-worker

  • Liked by
Cancel
Support Hours:
Weekdays: 09.00 - 17.00
Weekend: Off
Sorry, our support is not available at this time. We will reply at the next working hour.

By using the forum you are agree with tonjoo studio license agreement

Message :
Cancel
Loading more replies
You are not logged in

Please login here or simply sign up here .