# [TIP] Calculate your world compile time

## rudregues

I was struggling very hard to find an utility that calculates the total time I'd compiled all my packages, so I developed a command. Here are the explanation:

1. Get a list of all installed packages without the version at the end.

2. Redirect this to show building times in seconds.

3. Print only building times.

4. Sum all lines of building times.

5. Redirect the sum to a file.

6. Evalute the sum from the file.

7. Print the evaluated sum in HH:MM:SS format.

8. Delete the file to keep the system clean  :Wink: 

```
qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime
```

Mine is 15h:51m:28s right now.

EDIT: steveL did a big enhancement over my command, eliminating steps 3, 5, 6 and 8:

```
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
```

EDIT2: Mitigating/troubleshooting the outlier problem. People with absurd build times can use two commands to adapt the situation.

1) Calculating total sum excluding outliers defined by the user. I want everything bigger than 5400 seconds (1h30min) to be removed from calculation:

```
qlist -I | xargs qlop -t | awk '{ if ($2 < 5400) secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
```

2) Discovering the problematic build times (maybe report a bug?). I want to point out which packages build times are bigger than 5400 seconds (1h30min):

```
qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'
```

----------

## albright

OK, that is fun

my desktop i7-6700: 16h:36m:10s

my thinkpad x1 carbon (4th gen / i7-6600U): 21h:42m:15s

very similar installations (but distcc probably skews result)

----------

## fedeliallalinea

You can skip to write in a file

```
qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' | eval && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60))
```

----------

## steveL

Luke.. use the awk, Luke ;) 

```
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
```

 (untested, as not in gentoo atm.)

----------

## Juippisi

Is this one of those cases where "I copied a bash line from the internet and now my / is empty"? 

Just kidding, 

```

# qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime

11h:50m:56s

```

Sure felt a lot longer when I did -e world a while back. i7-2700k with '-march=native -O3 -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -flto=8 -fuse-linker-plugin -pipe -fstack-protector-all -mindirect-branch=thunk'

----------

## Fitzcarraldo

It's a rather academic exercise, though, isn't it? It doesn't take into account one's time spent investigating between failed merges due to blockers and bugs in packages and/or ebuilds. In many cases I have had to spend hours trying to fix a problem and get a package to merge, and its build time may be a few minutes. Over time I have spent more time debugging than the actual time it took to build the packages.

```
clevow230ss /home/fitzcarraldo # qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime

40h:18m:44s
```

----------

## Hu

 *Juippisi wrote:*   

> Is this one of those cases where "I copied a bash line from the internet and now my / is empty"?

 Any time you see an eval in the program, that's a good question to ask.  Although sometimes useful, incautious use of eval is extremely dangerous.

----------

## Ant P.

```
 ~ $ qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime

5055h:12m:42s
```

My output's a bit skewed by this one entry...

```
openscenegraph: 18068512 seconds average for 1 merges
```

Probably related to this bug in qlop

----------

## Zucca

 *steveL wrote:*   

> Luke.. use the awk, Luke ;) 
> 
> ```
> qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
> ```
> ...

 Works. Returns the exact same time.

My time is 76h:19m:7s... FX-8350 shows its age.

----------

## szatox

 *Quote:*   

> 103h:16m:18s

 

My poor hamster powered pc  :Laughing: 

I bought this box nearly a decade (sic!) ago and it was dirt cheap already.... And I don't really feel like replacing it any time soon. That APU A8-3870 works well enough.

----------

## Goverp

What would be more useful is similar arithmetic in emerge --ask to estimate the time for the command in question.  Then this script becomes essentially emerge -ea world.

----------

## szatox

Goverp, this script completed in 20 seconds, which is significantly faster than 4 days emerge -aev @world would take.

Also, each of us has different @world, different set of USE flags and different hardware which means we simply _cant_ compare results anyway. There is no way to get any meaningful data from it besides "how badly screwed I am when my hard drive dies". And the answer is "you better verify your backups"

----------

## Goverp

szatox, I'm not sure what you think I was suggesting, but nothing that would take 4 days.  My point was that emerge could do the same arithmetic, and tell you how long the command would take if you say "Yes", thereby at least telling you whether to take a coffee, go out to the cinema, or take a small vacation while portage recompiled whatever.

----------

## Zucca

That would be useful... Also the estimations for each package in the listing, since by calcutaling total time the time for each packge is already known and wouldne add much overhead.

This can be done wia processing --pretend output, but wastes time.

----------

## tuggbuss

```
3h:35m:50s
```

i7 - 6900K 

64GB RAM

NVMe

Nvidia blob

Xfce4

----------

## rudregues

 *tuggbuss wrote:*   

> 
> 
> ```
> 3h:35m:50s
> ```
> ...

 This is humiliation...

----------

## krinn

 *Ant P. wrote:*   

> My output's a bit skewed by this one entry...
> 
> ```
> openscenegraph: 18068512 seconds average for 1 merges
> ```
> ...

 

Can't beat me  :Smile: 

```
qlop -vgH fuse

fuse-2.7.4: Sat Sep 19 00:01:48 2009: 1158050441 days, 5 hours, 1 minute, 10 seconds

fuse-2.7.4: Sat Sep 19 03:21:32 2009: 22 seconds

fuse-2.8.1: Wed Nov 11 00:44:36 2009: 18 seconds

fuse-2.8.1: Wed Jun  2 23:19:18 2010: 14 seconds

fuse-2.8.1: Wed Sep 15 21:20:45 2010: 34 seconds

fuse-2.8.5: Mon Oct 11 05:01:42 2010: 23 seconds

fuse-2.8.6: Wed Sep 28 02:18:14 2011: 18 seconds

fuse-2.8.6: Mon Dec  5 17:28:47 2011: 23 seconds

fuse-2.8.7: Mon Apr 16 02:45:26 2012: 38 seconds

fuse-2.9.2: Sun Dec  2 17:55:34 2012: 58 seconds

fuse-2.9.3: Sun Dec 22 10:38:25 2013: 15 seconds

fuse-2.9.5: Thu Apr 14 12:07:27 2016: 16 seconds

fuse-2.9.7: Sun Jan 29 21:19:25 2017: 25 seconds

fuse-2.9.7-r1: Sun Mar 25 03:44:15 2018: 9 seconds

fuse-2.9.7-r1: Sun Mar 25 08:28:48 2018: 14 seconds

fuse: 15 times
```

----------

## Irre

```
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
```

502h:0m:44s   :Exclamation: 

Linux localhost 4.12.10-gentoo #2 SMP Thu Aug 31 09:20:09 GMT 2017 x86_64 Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz GenuineIntel GNU/Linux

Something is wrong!

[Moderator edit: added [code] tags to preserve output layout. -Hu]

----------

## josephg

think i just topped everyone  :Shocked:  and i run a tight ship too!

```
$ qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

777h:50m:29s

$ uname -a

Linux localhost 4.9.76-gentoo-r1-josephg #7 SMP Sun Feb 18 20:38:56 GMT 2018 i686 Intel(R) Core(TM)2 Duo CPU T5870 @ 2.00GHz GenuineIntel GNU/Linux
```

and this is my fastest machine  :Rolling Eyes:  sigh i don't keep up with the rest of the world

atleast it's faster runtime than my neighbours' shiny new windoze laptops  :Laughing: 

----------

## krinn

i didn't first output my result because it is just bork, but here it is to make everyone a few more happy with their "tiny" 500-700hours to build  :Very Happy: 

```
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' 

341606371735829h:20m:0s

Linux beleg 4.14.11 #2 SMP PREEMPT Wed Jan 10 15:11:16 CET 2018 x86_64 Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz GenuineIntel GNU/Linux

```

----------

## khayyam

 *krinn wrote:*   

> 
> 
> ```
> qlop -vgH fuse
> 
> ...

 

krinn ... hehe, just over 3 million years, that makes you gentoo's oldest user, hows that beard of yours? :)

```
% printf "%'.f\n" $((1158050441 / 364))

3,181,457
```

... well, if you don't calculate leap years.

best ... khay

----------

## Tony0945

Athlon X4 (bulldozer) Not bad for a $50 cpu

```
 qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

2h:17m:38s

```

EDIT:Can't believe Kaveri is this fast.

```
 # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

0h:0m:0s

```

----------

## rudregues

I edited my first post in order to help people with huge build times. 

Don't know though what to do with Tony0945's Kaveri...

----------

## <3

can we make an ebuild of this to put in portage or add this to gentoolkit?

----------

## ali3nx

dual cpu 10 core hyperthreaded xeon  :Rolling Eyes: 

Total: 1153 packages (1153 reinstalls), Size of downloads: 0 KiB

```
fenrir ~ # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

9h:46m:11s

fenrir ~ # uname -a

Linux fenrir 4.16.0-gentoo #3 SMP Tue Apr 3 03:50:21 CDT 2018 x86_64 Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz GenuineIntel GNU/Linux

```

----------

## lordoftheworld

Great info, I added a sort to see which packages might be taking up more time: 

qlist -I | xargs qlop -t |sort -t" " -rnk2

----------

## krinn

```
 qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, $2 / 3600, ($2 % 3600) / 60, $2 % 60); }'

gzip:  1d:26h:52m:15s

perl:  0d:7h:48m:44s

libxslt:  0d:1h:41m:7s

gimp:  0d:6h:0m:47s

libpng:  0d:8h:18m:2s

wireshark:  0d:5h:19m:57s

diffutils:  4d:114h:42m:38s

findutils:  3d:81h:57m:18s

gawk:  0d:21h:10m:11s

grep:  0d:19h:58m:8s

kbd:  4d:110h:17m:48s

fuse:  14233598822306d:341606371735361h:55m:12s

e2fsprogs-libs:  0d:18h:23m:13s

perl-version:  0d:1h:50m:40s

lynx:  1d:25h:54m:59s

```

yep something is weird, thanks for the update

----------

## rudregues

 *krinn wrote:*   

> 
> 
> ```
>  qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, $2 / 3600, ($2 % 3600) / 60, $2 % 60); }'
> 
> ...

 Thanks for pointing out my mistake krinn, there was really an error. Since the format of the output from this command is d:h:m:s hours should be always less than 24 (remainder from 86400 seconds division), like minutes (remainder from 3600 seconds division) and seconds (remainder from 60 seconds division) are less than 60. $2 / 3600 should be ($2 % 86400) / 3600

I updated again the first post.

----------

## rudregues

 *<3 wrote:*   

> can we make an ebuild of this to put in portage or add this to gentoolkit?

 I think it would be suitable for qlop from q applets.

I can open a bug report asking for this feature if people want to.

----------

## josephg

 *rudregues wrote:*   

> EDIT2: Mitigating/troubleshooting the outlier problem. People with absurd build times can use two commands to adapt the situation.
> 
> 1) Calculating total sum excluding outliers defined by the user. I want everything bigger than 5400 seconds (1h30min) to be removed from calculation

 

Why would you want to remove bigger packages? Those are the ones which matter for this purpose.

Something still doesn't look right  :Question: 

```
$ qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'

sudo:  1d:18h:35m:44s

libreoffice:  0d:12h:38m:34s

libnl:  0d:1h:56m:27s

libpcre:  0d:1h:46m:3s

qtdbus:  0d:1h:51m:35s

qtprintsupport:  0d:2h:42m:36s

qtwebengine:  0d:19h:4m:1s

qtwebkit:  6d:18h:9m:2s

qtxmlpatterns:  0d:2h:0m:40s

terminus-font:  1d:13h:15m:10s

opus:  0d:2h:22m:54s

ffmpeg:  0d:10h:3m:37s

libpcap:  6d:2h:7m:37s

busybox:  0d:2h:4m:8s

man-db:  0d:8h:59m:9s

man-pages:  0d:1h:47m:35s

gcc:  0d:3h:46m:6s

llvm:  0d:2h:16m:34s

gentoo-sources:  0d:16h:49m:9s

kergen:  4d:3h:17m:40s

e2fsprogs-libs:  0d:2h:11m:54s

package-manager:  1d:22h:54m:13s

service-manager:  1d:22h:31m:27s

ssh:  1d:22h:50m:52s

firefox:  0d:8h:38m:9s

adwaita-icon-theme:  0d:3h:51m:9s
```

----------

## <3

 *rudregues wrote:*   

>  *<3 wrote:*   can we make an ebuild of this to put in portage or add this to gentoolkit? I think it would be suitable for qlop from q applets.
> 
> I can open a bug report asking for this feature if people want to.

 

that would be great

----------

## edgar_uriel84

It's funny

```
┌─ [4966][14:24:27][genomorro@genomorro-xps-15-9550][~]

└ᐅ qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

7h:31m:19s

┌─ [4966][14:29:10][genomorro@genomorro-xps-15-9550][~]

└ᐅ qlist -I | xargs qlop -t | awk '{ if ($2 > 1800) printf("%s  %dd:%dh:%dm:%ds\n", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'

libreoffice:  0d:1h:4m:54s

┌─ [4966][14:29:19][genomorro@genomorro-xps-15-9550][~]

└ᐅ uname -a

Linux genomorro-xps-15-9550 4.9.76-gentoo-r1 #1 SMP PREEMPT Mon Apr 2 01:56:45 CDT 2018 x86_64 Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz GenuineIntel GNU/Linux

```

I'm using DWM as WM.

----------

## Nreal

Two of those bastards Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz is together 72 threads, (36 actual cores)

```
qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

1024819115206107h:1m:20s
```

Waiting for more ram as 16gt swaps with anything above -j30.. Its fast though  :Smile: 

```
boxi /home/markus # uname -a

Linux boxi 4.17.1-gentoo #6 SMP PREEMPT Sat Jun 16 12:55:42 EEST 2018 x86_64 Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz GenuineIntel GNU/Linux

processor   : 71

vendor_id   : GenuineIntel

cpu family   : 6

model      : 63

model name   : Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz

stepping   : 2

microcode   : 0x3c

cpu MHz      : 2793.850

cache size   : 46080 KB

physical id   : 1

siblings   : 36

core id      : 27

cpu cores   : 18

apicid      : 119

initial apicid   : 119

fpu      : yes

fpu_exception   : yes

cpuid level   : 15

wp      : yes

flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts

bugs      : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass

bogomips   : 4590.71

clflush size   : 64

cache_alignment   : 64

address sizes   : 46 bits physical, 48 bits virtual

power management:
```

[Moderator edit: added [code] tags to preserve output layout. -Hu]

----------

## rudregues

I posted a bug report asking for add this functionality into qlop (app-portage/portage-utils) https://bugs.gentoo.org/658824

----------

## phelix

Hi there,

have you tried genlop?

```

$ emerge --pretend --emptytree @world | genlop --pretend

[...]

Estimated update time: 5 hours, 39 minutes.

```

cheers

-- edited for formatting

----------

## rudregues

 *phelix wrote:*   

> Hi there,
> 
> have you tried genlop?
> 
> ```
> ...

 Your tip worked, but it has some caveats:

1) It took a big while to complete (maybe 1 minute?);

2) The output is very verbose because of emerge (using --quiet hasn't effect to minimize this);

3) Emerge will look into new packages to satisfy dependencies, not only the already installed, maybe increasing the time in the answer.

Other than these problems, the final answer was Estimated update time: 1 day, 2 hours, 4 minutes., which is pretty close to my script answer of 25h:1m:32s

----------

## ppurka

qlop seems unreliable for some cases. My script (see my signature) provides an estimated time  

```
quietemerge -pe1 @world

...

Total ETA: 1d 16h 24m 10s
```

The command in the main post provides an estimated time of over 10 days

```
~» qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

290h:22m:38s
```

It seems that some of the estimates made by qlop are incorrect:

```
 » qlop -v -g -H qtcurve 

qtcurve-1.8.17-r1: Sat Mar 22 11:19:38 2014: 1 minute, 19 seconds

qtcurve-1.8.17-r1: Thu Apr 24 04:27:08 2014: 39 seconds

qtcurve-1.8.17-r1: Sun Oct 19 22:44:02 2014: 51 seconds

qtcurve-1.8.18-r1: Sat Feb 20 19:07:04 2016: 2 minutes, 16 seconds

qtcurve-1.8.18-r1: Sun May  8 14:10:10 2016: 2 minutes, 8 seconds

qtcurve-1.8.18-r1: Sun May  7 22:41:07 2017: 104 days, 13 hours, 34 minutes, 53 seconds

qtcurve-1.9.0_rc1-r1: Sun Oct  1 16:37:44 2017: 2 minutes, 20 seconds

qtcurve-1.9.0_rc1-r1: Sat Nov 11 17:14:05 2017: 1 minute, 32 seconds

qtcurve-1.9.0_rc1-r1: Mon Jul  2 19:38:20 2018: 1 minute, 7 seconds

qtcurve-1.9.0_rc1-r2: Sun Jul 15 19:13:07 2018: 1 minute, 50 seconds

qtcurve-1.9.0: Sun Oct 21 20:23:40 2018: 2 minutes, 20 seconds

qtcurve: 11 times
```

On the other hand, the estimates provided by the internal computations in the quietemerge script gives the following (in reverse order to above):

```
~ » mygenlop qtcurve

 *  Printing the times in reverse order (recent emerge appears first)

    Run no.  Time Taken

         1    2m 20s 

         2    1m 50s 

         3    1m 7s 

         4    1m 32s 

         5    2m 20s 

         6    1m 2s 

         7    2m 8s 

         8    2m 16s 

         9    51s 

        10    39s 

        11    1m 19s 

 *  Average time taken over 11 runs: 1m 34s
```

----------

## pcmaster

In my box:

```
Core2Duo ~ # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

42h:19m:8s

Core2Duo ~ # 
```

Intel Core2 Duo 6550

4 GB RAM

2 TB WD black HDD

----------

## Zucca

I don't know if some else has already mentioned this, but the estimation the command gives is almost always larger than the actual world compile time, unless one disallows parallel merging of packages (emerge --jobs <n>).

----------

## proxy

 *Zucca wrote:*   

> I don't know if some else has already mentioned this, but the estimation the command gives is almost always larger than the actual world compile time, unless one disallows parallel merging of packages (emerge --jobs <n>).

 

I was thinking the same thing. Though, you may be able to get a sensible estimate by dividing the result by the number of cores you have (assuming they are all the same spec). Won't be precise, but would probably be close enough to be informative as a real world estimate.

----------

## Zucca

Any bigger package that can run make -j<N> effectively at compile phase would normally limit emerge --jobs mto just one.

For example I have --jobs 8 --load-average 10.00 on my EMERGE_DEFAULT_OPTS.

So maximum of 8 concurrent package installs/compiles, but only is load average is under 10.00.

So sometimes for example Firefox and libreoffice both get to be compiled at the same time, effectively lenghtening the compile time of both on my 8-core CPU (would result around load average of 16). In this case compile times recorded for these packages are 2x at worst.

On the other hand some small packages that really can't fully utilize make -j8 (on my system), are almost always being run at 1x speed, but those can be "emerged" parallel (at most of 8 packages being compiled/installed at the same time).

In this case it's the opposite to the previous one. Since on average all those smaller packages are "emerged" in "time * 1/8". But the method counting the world compile time gives each 1x time instead of 1/8.

So you can't just divide the time by the cores on your system... unfortunately.

----------

## grobian

 *rudregues wrote:*   

> I posted a bug report asking for add this functionality into qlop (app-portage/portage-utils) https://bugs.gentoo.org/658824

 

For what is worth, with portage-utils-0.80_pre*:

```

% qlop --summary

...

sync: 1′56″ average for 1 sync

total: 9:38:35 for 352 merges, 1 sync

```

----------

## Zucca

I'm tempted to ACCEPT some KEYWORDS now.

Thanks for the work, btw.

----------

## brokendreams

 *rudregues wrote:*   

>  *phelix wrote:*   Hi there,
> 
> have you tried genlop?
> 
> ```
> ...

 

The other issue when genlop is you have to still have all the logs for the last build you did of every single package that's installed or the time is off. It must read the log files to get the time. Because it will warn you if there are log files missing

----------

## brokendreams

 *Zucca wrote:*   

> I'm tempted to ACCEPT some KEYWORDS now.
> 
> Thanks for the work, btw.

 

I run several systems that are fully ~amd64 keyworded. Have never had what I'd call serious problems. Sometimes something will fail or an issue will arise so I have to mask something. But it doesnt happen too often. My package.mask is maybe 15 lines. Of those half of them were by choice. And I could probably get rid of half of the rest as they are older packages.

----------

## Tony0945

```
tony@MSI ~ $  emerge --pretend --emptytree @world | genlop --pretend

!!! Error: no merge found for '-p'
```

```
ony@MSI ~ $  emerge --pretend --emptytree @world

These are the packages that would be merged, in order:

Calculating dependencies -...
```

  Something is wrong with genlop --pretend

----------

## brokendreams

 *Tony0945 wrote:*   

> 
> 
> ```
> tony@MSI ~ $  emerge --pretend --emptytree @world | genlop --pretend
> 
> ...

 

Do you have color forced on? Like in emerge_default_opts or something? I know having --color=y will totally confuse genlop -p

----------

## Anon-E-moose

 *Tony0945 wrote:*   

> 
> 
> ```
> tony@MSI ~ $  emerge --pretend --emptytree @world | genlop --pretend
> 
> ...

 

Worked for me. 

```
emerge --pretend --emptytree @world | genlop --pretend 

These are the pretended packages: (this may take a while; wait...)

...

Estimated update time: 7 hours, 58 minutes.
```

It did indeed take a while.   :Laughing: 

----------

## Zucca

 *brokendreams wrote:*   

> I run several systems that are fully ~amd64 keyworded. Have never had what I'd call serious problems. Sometimes something will fail or an issue will arise so I have to mask something. But it doesnt happen too often. My package.mask is maybe 15 lines. Of those half of them were by choice. And I could probably get rid of half of the rest as they are older packages.

 I've been on the ~ -side. The problem isn't so much the "unstabliness" of the packages, but rather the update frequency. And of course it's nice to have the feeling of stable system on a home server.

----------

## brokendreams

 *Zucca wrote:*   

>  *brokendreams wrote:*   I run several systems that are fully ~amd64 keyworded. Have never had what I'd call serious problems. Sometimes something will fail or an issue will arise so I have to mask something. But it doesnt happen too often. My package.mask is maybe 15 lines. Of those half of them were by choice. And I could probably get rid of half of the rest as they are older packages. I've been on the ~ -side. The problem isn't so much the "unstabliness" of the packages, but rather the update frequency. And of course it's nice to have the feeling of stable system on a home server.

 

I can certainly agree with that. I do run stable on my media server simply because that box is lower specs and would take forever to update if it was on ~arch. Still runs an old amd phenom 955. 

Although unstable is currently a little "hairy" too. The switch fro. Cblas blas and lapack reference packages to a single lapack package that includes all three blas cblas and lapack reference libraries has tons of stuff failing with undefined references to various cblas_* (cant remember the exact error) functions or whatever.  That revdep-rebuild wont catch. Something just wasnt rebuilt right I just don't know what yet. I'll prob just end up emerge -e @world to fix it if I cant figure out exactly what package it is. So it does get crazy sometimes too but I expected this since moving to that single package and allowing it to be selected and picked up by ld.so was a very radical change.

----------

## MrBrutico

To my market zero seconds

```
elover # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }' 

0h:0m:0s

```

----------

## radio_flyer

For those still playing with this, some notes:

Taking numpy, for example:

```

$ qlop -t numpy

2012-03-12T17:16:12 >>> dev-python/numpy: 2′56″

2012-06-02T12:34:59 >>> dev-python/numpy: 2′59″

2012-12-07T10:41:48 >>> dev-python/numpy: 3′00″

2013-02-27T08:25:16 >>> dev-python/numpy: 2′17″

2013-05-24T09:00:28 >>> dev-python/numpy: 2′45″

2013-05-24T10:51:23 >>> dev-python/numpy: 4′26″

2013-09-03T13:14:56 >>> dev-python/numpy: 3′29″

2013-09-05T08:35:27 >>> dev-python/numpy: 3′49″

2013-10-31T10:10:45 <<< dev-python/numpy: 4s

2013-10-31T10:38:07 >>> dev-python/numpy: 2′03″

2013-10-31T20:19:37 >>> dev-python/numpy: 2′16″

2013-12-02T00:49:14 >>> dev-python/numpy: 3′49″

2014-02-09T09:50:46 >>> dev-python/numpy: 2′42″

2014-07-06T12:07:37 >>> dev-python/numpy: 1′21″

2014-10-27T14:35:20 >>> dev-python/numpy: 1′25″

2014-10-28T20:37:48 >>> dev-python/numpy: 1′20″

2014-11-23T10:35:55 >>> dev-python/numpy: 1′16″

2014-11-24T09:04:41 >>> dev-python/numpy: 1′28″

2015-03-03T09:50:57 >>> dev-python/numpy: 3′32″

2015-03-31T00:30:17 <<< dev-python/numpy: 4s

2015-03-31T00:35:55 >>> dev-python/numpy: 3′29″

2015-07-26T09:11:41 >>> dev-python/numpy: 2′31″

2015-10-02T03:45:29 >>> dev-python/numpy: 2′46″

2016-01-26T09:59:43 >>> dev-python/numpy: 1′49″

2016-10-28T09:00:12 >>> dev-python/numpy: 1′46″

2016-11-22T09:07:14 >>> dev-python/numpy: 1′54″

2017-04-19T05:32:09 >>> dev-python/numpy: 1′47″

2017-04-20T03:32:36 >>> dev-python/numpy: 1′46″

2017-11-23T04:45:17 >>> dev-python/numpy: 1′46″

2017-11-29T12:14:24 >>> dev-python/numpy: 2′46″

2017-12-02T17:06:06 >>> dev-python/numpy: 2′44″

2018-06-22T11:00:27 >>> dev-python/numpy: 6′36″

2018-06-22T23:23:59 >>> dev-python/numpy: 6′41″

2018-06-23T00:47:55 >>> dev-python/numpy: 2′33″

2018-06-23T10:44:59 >>> dev-python/numpy: 5′03″

2018-07-24T10:36:58 >>> dev-python/numpy: 3′57″

2019-03-23T14:44:31 >>> dev-python/numpy: 4′31″

2019-06-26T10:48:30 >>> dev-python/numpy: 4′12″

2019-07-01T08:47:48 >>> dev-python/numpy: 5′40″

2019-07-01T12:00:21 <<< dev-python/numpy: 5s

2019-07-01T12:14:07 >>> dev-python/numpy: 5′08″

2019-07-01T13:47:51 >>> dev-python/numpy: 5′11″

2019-11-08T13:26:54 >>> dev-python/numpy: 4′23″

2019-11-10T06:39:16 >>> dev-python/numpy: 4′25″

```

Note that the times vary (on the same machine) quite a bit because different amounts of parallelization are needed to get a working build (numpy is very sensitive to this). Also note that some unmerge times are included, because occasionally I had to unmerge numpy to make portage happy, and then merge it back in "carefully". (numpy is also very touchy like this.)

The '-t' option to qlop reports ALL emerge and unmerge times for a package. This is why some folks are getting huge numbers. Also note that the times are listed in field 4, not field 2. On my system, the OP script returns:

```

$ qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

0h:0m:0s

```

If I change the awk field to 4:

```

$ qlist -I | xargs qlop -t | awk '{secs += $4} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

211h:59m:59s

```

That explains why some folks are getting no times, while others are getting outrageous numbers. This is with qlop from app-portage/portage-utils-0.80.

With that version, the '-a' option for qlop reports an average:

```

$ qlop -a numpy

dev-python/numpy: 3′10″ average for 41 merges

dev-python/numpy: 4s average for 3 unmerges

```

Note that the time is now output in field two. There still two entries for emerges and unmerges, but the unmerges tend to be fast and don't effect the overall results too much. The average build time is quicker than my current build times, but not bad as a wag. Numpy is a worst-case package; most package build times don't show this much variation.

Using '-a' instead of '-t' and reverting back to field 2, the script produces:

```

$ qlist -I | xargs qlop -a | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

15h:52m:43s

```

This estimate much more closely matches the average world build time that I see. Double that for the time spent resolving various build failures and filing the appropriate bug reports and patches.

----------

## acidbreez

I know this is an old post but I figured I would show you guys/gals what I got for numbers too  :Smile: 

```
$ qlist -I | xargs qlop -t | cut -f4 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime
```

43h:26m:45s

```
$ qlist -I | xargs qlop -a | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'
```

7h:1m:40s

```
$ uname -ar
```

Linux Gentoo 5.18.5-gentoo-custom #9 SMP PREEMPT_DYNAMIC Fri Jul 8 10:47:38 MDT 2022 x86_64 AMD Ryzen 9 3900X 12-Core Processor AuthenticAMD GNU/Linux

----------

## mv

 *acidbreez wrote:*   

> 
> 
> ```
> $ qlist -I | xargs qlop -t | cut -f4 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime
> ```
> ...

 

I realized that the number shown by this code is much too low (only ~20 hours).

Indeed:

```
$ qlop -t gcc | cut -f4 -d" "                    

5:21:17

8:14:55

34s

$ qlop -t gcc | cut -f4 -d" "| awk '{s+=$1} END {print "secs="s}

secs=47
```

WTF? Aha:

```
qlop -t gcc | cut -f4 -d" "| awk '{s+=$1;print s}'

5

13

47
```

The largest unit (typically hours or minutes) is just counted as a second and the rest discarded.

A fix is to pass the "-M" option to qlop -t:

```
$ qlist -I | xargs qlop -tM | cut -f4 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime

102h:57m:1s
```

This looks more realistic.

----------

## soundrolf

This show Compile Time with 'emerge -e @world

```
qlist -I | xargs qlop -a | awk '{secs += $2} END { printf("%dh:%dm:%ds\n", secs / 1800, (secs % 1800) / 60, secs % 60); }'
```

Result: 19h:11m:28s

This show the amount of packages

```
find /var/db/pkg/ -mindepth 2 -maxdepth 2 -printf "%P\n" | wc -l
```

Result: 1631

Regargs

soundrolf

----------

## flysideways

```
~ # qlist -I | xargs qlop -t | awk '{secs += $4} END { printf("%dh:%dm:%ds\n", secs / 3600, (secs % 3600) / 60, secs % 60); }'

6h:9m:23s

```

My Gentoo VM running on a usb ssd attached to a new MacBook Pro.

I found this thread while I was looking about for a way to use qlop to predict update times like I used to do with genlop, "emerge -p command | genlop -p". 

Any suggestions?

Thanks

----------

