Hi, just gave your plugin a spin and it seems to work great. The only thing that is bothering me is scalability.
For example, I have 6 WordPress instances running a single site. All those instances have one background-worker listen-loop. That makes for 6 workers. Now I put 20 jobs in queue. The workers start to hog the jobs and run each parallel. That’s not what I want.
Here’s how to simulate the situation:
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &
$ wp background-worker listen-loop –allow-root &
Now imagine a situation where 16 people would send form data to an endpoint simultaneously and would get an email confirmation. My workers respond:
Working on job ID = 7
Working on job ID = 7
Working on job ID = 7
Working on job ID = 8
Working on job ID = 8
Working on job ID = 8
Working on job ID = 9
Working on job ID = 9
Working on job ID = 9
Working on job ID = 10
Working on job ID = 10
Working on job ID = 10
Working on job ID = 11
Working on job ID = 11
Working on job ID = 11
Working on job ID = 12
Working on job ID = 12
Working on job ID = 12
Working on job ID = 13
Working on job ID = 13
Working on job ID = 13
Working on job ID = 14
Working on job ID = 14
Working on job ID = 14
Working on job ID = 15
Working on job ID = 15
Working on job ID = 15
Working on job ID = 16
Working on job ID = 16
Working on job ID = 16
Working on job ID = 17
Working on job ID = 17
Working on job ID = 17
Working on job ID = 18
Working on job ID = 18
Working on job ID = 18
Working on job ID = 19
Working on job ID = 19
Working on job ID = 19
Working on job ID = 20
Working on job ID = 20
Working on job ID = 20
Working on job ID = 21
Working on job ID = 21
Working on job ID = 21
Working on job ID = 22
Working on job ID = 22
Working on job ID = 22
Working on job ID = 23
Working on job ID = 23
Working on job ID = 23
Job well done but too many emails were sent. And if the job took any longer, it would be 16*6 = 96 times.
What I would like to see is some kind of locking system for each task, that would probably solve the problem.
