Parallel execution of PHP tasks
There is a file that runs every N minutes. When this file is run, something like this happens there:
- Displays a list of groups from MySQL.
- Starts updating information about groups by making a request to the VK API.
- Gets the necessary information and writes them to MySQL.
If you do it consistently, you get a critical waiting time. It is important that 100 groups run for no more than 1 minute. If you do not do this task sequentially and in parallel, in theory, this will take approximately 10 seconds or even less. The question will already consist in the server power, but this is not a problem.
You can of course create dozens of files and put them in the CROWN task. But I think there is a much more elegant solution and in the end the right one.
2 answers
In the context of script execution, PHP is initially single-threaded. There are various tricks for parallelization, but they are close in spirit to placing crutches.
The options are as follows:
- Use a different language. The Go example is very good in terms of multithreading. Sad but true.
- Optimize the script operation. 10-20 seconds to process 5 entities is, in general, a very long time. But here you need to know your specifics.
- Run multiple processes daemons that will spin in an infinite loop and read tasks from the queue (for example, RabbitMQ, Redis, etc.)
- The same as 3, but push tasks to workers from the master process of the orchestrator.
UPD in connection with the update of the question
Most likely, your bottleneck is a request to the VK API. In this case, you can use curl_multi_init to parallelize queries.
You can try the extension pthreads or multicurl (pay attention to the library http://docs.guzzlephp.org/en/stable/). Maybe this option will suit you http://php.net/manual/ru/function.pcntl-fork.php. As I understand it, the list of groups can be requested once and then, based on the contents of the list, you need to make requests to the external API.