bash fork: retry: Resource temporarily unavailable - Ask Ubuntu

i trying run shell script create process using shell script. resource temporarily unavailable error. how identify limit (memory/process/filecount) creating problem. below ulimit results.

core file size          (blocks, -c) 0 data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 563959 max locked memory       (kbytes, -l) unlimited max memory size         (kbytes, -m) unlimited open files                      (-n) 65535 pipe size            (512 bytes, -p) 8 posix message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) unlimited cpu time               (seconds, -t) unlimited max user processes              (-u) 10000000 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited 

for case in comments, not using memory per thread, hitting cgroup limits. find default around 12288, value writable:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 12288 $ echo 15000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 15000 $ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 15000 

and if use "what thread limit" program (found here) check, before:

$ ./thread-limit creating threads ... 100 threads far ... 200 threads far ... ... 12100 threads far ... 12200 threads far ... failed return code 11 creating thread 12281 (resource temporarily unavailable). malloc worked, hmmm 

and after:

$ ./thread-limit creating threads ... 100 threads far ... 200 threads far ... 300 threads far ... ... 14700 threads far ... 14800 threads far ... 14900 threads far ... failed return code 11 creating thread 14993 (resource temporarily unavailable). malloc worked, hmmm 

of course, numbers above not exact because "doug" user has few other threads running, such ssh sessions sever. check with:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current 8 

program used:

/* compile with:   gcc -pthread -o thread-limit thread-limit.c */ /* from: http://www.volano.com/linuxnotes.html */  #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <pthread.h> #include <string.h>  #define max_threads 100000 #define pthread_stack_min 1*1024*1024*1024 int i;  void run(void) {   sleep(60 * 60); }  int main(int argc, char *argv[]) {   int rc = 0;   pthread_t thread[max_threads];   pthread_attr_t thread_attr;    pthread_attr_init(&thread_attr);   pthread_attr_setstacksize(&thread_attr, pthread_stack_min);    printf("creating threads ...\n");   (i = 0; < max_threads && rc == 0; i++) {     rc = pthread_create(&(thread[i]), &thread_attr, (void *) &run, null);     if (rc == 0) {       pthread_detach(thread[i]);       if ((i + 1) % 100 == 0)     printf("%i threads far ...\n", + 1);     }     else     {       printf("failed return code %i creating thread %i (%s).\n",          rc, + 1, strerror(rc));        // can allocate memory?       char *block = null;       block = malloc(65545);       if(block == null)         printf("malloc failed :( \n");       else         printf("malloc worked, hmmm\n");     }   } sleep(60*60); // ctrl+c exit; makes easier see mem use   exit(0); } 

see here

now, if have enough memory, next limit defined default maximum pid number, 32768, writable. obvioulsy in order have more 32768 simultaneous processes or tasks or threads pid have allowed higher:

$ cat /proc/sys/kernel/pid_max 32768 $ echo 80000 | sudo tee /proc/sys/kernel/pid_max 80000 $ cat /proc/sys/kernel/pid_max 80000 

note quite on purpose number bigger 2**16 chosen, see if allowed. , now, set cgroup max to, 70000:

$ echo 70000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 70000 $ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max 70000 

and @ point, realize above listed program seems have limit of 32768 threads, if resources still available, , use method. test server 16 gigabytes of memory seems exhaust other resource @ 62344 tasks, though there seem still memory available.

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current 62344 

top:

top - 13:48:26 21:08,  4 users,  load average: 281.52, 134.90, 70.93 tasks: 62535 total, 201 running, 62334 sleeping,   0 stopped,   0 zombie %cpu0  : 96.6 us,  2.4 sy,  0.0 ni,  1.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st %cpu1  : 95.7 us,  2.4 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st %cpu2  : 95.1 us,  3.1 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st %cpu3  : 93.5 us,  4.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st %cpu4  : 94.8 us,  3.4 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st %cpu5  : 95.5 us,  2.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st %cpu6  : 94.7 us,  3.5 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st %cpu7  : 93.8 us,  4.5 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st kib mem : 15999116 total,   758684 free, 10344908 used,  4895524 buff/cache kib swap: 16472060 total, 16470396 free,     1664 used.  4031160 avail mem    pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command 37884 doug      20   0  108052  68920   3104 r   5.7  0.4   1:23.08 top 24075 doug      20   0    4360    652    576 s   0.4  0.0   0:00.31 consume 26006 doug      20   0    4360    796    720 s   0.4  0.0   0:00.09 consume 30062 doug      20   0    4360    732    656 s   0.4  0.0   0:00.17 consume 21009 doug      20   0    4360    748    672 s   0.3  0.0   0:00.26 consume 

seems hit default ulimit settings both user processes , number of timers (signals):

$ ulimit -i 62340 doug@s15:~$ ulimit -a core file size          (blocks, -c) 0 data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 62340 max locked memory       (kbytes, -l) 64 max memory size         (kbytes, -m) unlimited open files                      (-n) 32768 pipe size            (512 bytes, -p) 8 posix message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) 8192 cpu time               (seconds, -t) unlimited max user processes              (-u) 62340 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited 

if raise limits, able go 69000 threads, asked , far i'm going take answer:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current 69011 

top:

top - 16:39:43 33 min,  3 users,  load average: 314.59, 181.48, 105.27 tasks: 69205 total, 234 running, 68971 sleeping,   0 stopped,   0 zombie %cpu(s): 93.7 us,  5.5 sy,  0.0 ni,  0.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st kib mem : 15999004 total,  2659452 free, 11393940 used,  1945612 buff/cache kib swap: 16472060 total, 16472060 free,        0 used.  2866316 avail mem    pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command  4166 doug      20   0  115408  75956   3252 r   5.1  0.5   1:30.52 top 62667 doug      20   0   28916   1516   1320 r   3.7  0.0   0:01.14 ps 73184 doug      20   0    7196   4396   1600 s   0.9  0.0   0:09.95 try_stuff5  2038 doug      20   0    4360    652    576 s   0.4  0.0   0:00.34 consume  4594 doug      20   0    4360    652    580 s   0.4  0.0   0:00.14 consume  5435 doug      20   0    4360    652    576 s   0.4  0.0   0:00.24 consume  8891 doug      20   0    4360    688    612 s   0.4  0.0   0:00.14 consume 

at point trouble, absolutely amazing how gracefully system bogs down. tried 118000 threads , system totally bogged down , had many of these errors:

feb 17 16:13:02 s15 kernel: [  967.907305] info: task waiter:119371 blocked more 120 seconds. feb 17 16:13:02 s15 kernel: [  967.907335]       not tainted 4.10.0-rc8-stock #194 feb 17 16:13:02 s15 kernel: [  967.907357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables message. 

and load average ballooned ~29000. left computer hour , sorted out. staggered spin out of threads 200 microseconds per spin out, , able run 118000 threads fine (admittedly @ little resource use per thread).