Ubuntu

How do I run 512 processes in batches of 64

Asked by tomdean

I have a task that can be divided into chunks. The current one is 512 distinct processes.

I can run each one in a loop. That takes lots of time.

I have an AMD 3970X CPU so I can run 64 processes at a time.

I now have a script that cycles through all the processes, stopping to wait after 64 processes have been started. This is a significant improvement.

I think this can be improved:
- start 64 processes
- as each one completes, start another one.

I am looking into using sockets.
- start 63 processes - each one waiting on a socket
- send a job to each socket
- as each socket becomes available send another job

How do I do this?

Question information

Language:
English Edit question
Status:
Expired
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
actionparsnip (andrew-woodhead666) said : 		#1

You can run thousands of processes on your CPU. Each thread will run in time. Just because you have a 64bit CPU doesn't mean you can only run 64 processes at a time.

I have a Raspberry Pi running a 64bit OS and it is running 112 processes.

 $ ps -ef | grep -v TIME | wc -l
112

What is the purpose of running the processes in this way? What are you trying to achieve?

Cheers

Revision history for this message
tomdean (tomdean) said : 		#2

On 7/20/21 4:41 AM, actionparsnip wrote:
> Your question #698047 on Ubuntu changed:
> https://answers.launchpad.net/ubuntu/+question/698047
>
> Status: Open => Needs information
>
> actionparsnip requested more information:
> You can run thousands of processes on your CPU. Each thread will run in
> time. Just because you have a 64bit CPU doesn't mean you can only run 64
> processes at a time.
>
> I have a Raspberry Pi running a 64bit OS and it is running 112
> processes.
>
> $ ps -ef | grep -v TIME | wc -l
> 112
>
> What is the purpose of running the processes in this way? What are you
> trying to achieve?

Guess I was not clear. This has nothing to do with 'a 64bit CPU', but
with a 32-core cpu with 2 threads per cpu.

I want to do parallel processing.

I can break my task into 512 independent processes. Or 1024, etc.
I have an AMD Threadripper 3970X CPU, which will run 64 threads.

So, if I start 64 processes, each one can run on it's own 'cpu'.
I wait for these to complete and the start the next 64, etc.
Doing it this way took:
elapsed 132.50sec cpu 4862% ctx-sw (vol 3518 uvol 573833) pg-flt 135867

Starting all 512 processes, letting linux sort out which one runs where
and when, took:
elapsed 104.67sec cpu 6313% ctx-sw (vol 3634 uvol 940455) pg-flt 136578
A significant improvement, because it eliminated the 'wait'.

I believe this later try is doing a lot of unnecessary context switches.
There were almost 2x switches in less time. So, maybe linux was
stopping one process and starting another. It may be faster to let each
process run to completion before starting another.

I am looking for a way to
- start 64 processes maybe using taskset to select the 'cpu'
- as each one completes, start another one on the same 'cpu'
Maybe this would result is fewer context switches. Maybe devote one
'cpu' to control the other 63?

If I extend my task to several million independent processes this would
be worthwhile.

Revision history for this message
Launchpad Janitor (janitor) said : 		#3

This question was expired because it remained in the 'Open' state without activity for the last 15 days.