How to launch the benchmarks ?

[LaurentPlagne]

Hi,
I have downloaded LoopVectorization source and all the tests pass.
I would like to launch the benchmarks (include("benchmark/driver.jl")) and I fail to adapt directcalljit.f90
which contains hard paths

! include "/home/chriselrod/intel/mkl/include/mkl_direct_call.fi"

include "/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/include/mkl_direct_call.fi"

got and error
Bad # preprocessor line #if defined MKL_DIRECT_CALL_SEQ_JIT

What is the correct way to launch the benchmarks ?

[chriselrod]

I would like to launch the benchmarks (include("benchmark/driver.jl"))

This works on the computer I run them on.
If you're willing to work with me a little here and then share the results (and your system specs afterward) I'd be happy to generalize them so that they work more generally.

The particular error you cited is from trying to compile code wrapping MKL.
That was was supposed to use MKL's JIT, but that aspect doesn't work locally for me either -- it's just regular MKL in the benchmarks (I don't mention the JIT anywhere, since I couldn't get it to work).

So I'll replace that part with MKL_jll, since I believe that should work for everyone.
Then for OpenBLAS, I could either be lazy and recommend ensuring BLAS.vendor() == :openblas64 (as is the case now), or use OpenBLAS_jll. The latter would have the advantage of ensuring the latest release of OpenBLAS. On my CPU architecture, the latest release (0.3.9) is like twice as fast as v0.3.7. Julia 1.4 uses v0.3.7.

Next comes the compilers.
I will definitely make the Intel compilers optional. Most folks won't have access to them. I will not for much longer, either. This is a shame, because they tend to perform extremely well.

I guess I may as well do the same for gcc, g++, gfortran, clang, and clang++ at that point.
Installing these won't be a problem for anyone on Linux, but not everyone is.

[LaurentPlagne]

Hi Chris,

first I use this occasion to express again how impressive I find your work on LoopVectorization.jl !

I am currently trying to write a package called BTL.jl that aims to performs benchmarks for a large number of micro-kernels (dot, axpy, gemm, copy, permutation,...) implemented with different language (Julia, C, C++, Fortran,...), different implementations for each language, different compilers, (different options), and different machines. I used to do this from C++ but I think Julia is a perfect candidate for this task. For example, one would have access to the generated assembly and source code for each performance curves.

All the measurements data will be stored on a database server and I have started to write a JS client to browse this database. The objective of this project is multiple

• Compare different implementations for a given kernel (for a given, language,machine, compiler)
• Compare different languages for a given kernel
• Compare different machine (Intel, AMD, GPU..)

I use this kind of performance analysis to teach HPC (and incidentally why Julia is so powerful). I use to do inside for my previous company but now is a good time to go open source :)

I have experienced some problem with BenchmarkTools which consumes more and more memory while the different benchmarks are performed... I haven't exactly understood in detail why (although François explained this to me twice ;). So, looking at your superb performance graphs I thought that I could learn how to do it properly on Julia ;). I guessed that you use Distributed in order to speed up the measurements but It may also be a proper way to release the memory allocated by @belapsed.

For Intel's compilers, I think that you can obtain free licenses being an OS developer. Let me know if you have problem to get licenses.

An easy installation would be key to gather data from many users and improve the machine dimension. I thought that a docker distribution for this purpose maybe appropriate (not having to install all the prerequisite manually).

[chriselrod]

first I use this occasion to express again how impressive I find your work on LoopVectorization.jl !

Thanks. It is my intention for LoopVectorization's macrokernels to be optimal, or very nearly so -- and by extension, the microkernels as well.
Code-gen isn't always perfect, and I'd also like to get benchmarks on more machines.

In particular, I'm curious about prefetching at the moment. In rare cases, LoopVectorization will insert prefetches. The prefetch distance has been tuned to my own computer, but I probably need a much larger sample to pick a good way of doing this.
Related is that it adds the cost of prefetches to the corresponding loads, but I believe this cost may not be high enough for <512-bit vector architectures, or that I should force those architectures to unroll by some multiple of the cacheline size.
Currently, with AVX2, it chooses a 12x4 gemm kernel. The cacheline-multiple restriction would force it to pick 8x6 instead. I ran a few benchmarks, but they proved inconclusive, so I would still need to do some more work figuring out how to really account for those costs.

BTL.jl sounds like an interesting project.

I have experienced some problem with BenchmarkTools which consumes more and more memory while the different benchmarks are performed... I haven't exactly understood in detail why (although François explained this to me twice ;).

Huh, I never noticed that. I did not deliberately add any workarounds. Is there a leak, or why doesn't the GC take care of it?
I've just started running the benchmarks again to make sure the latest changes (not pushed yet) work.
So far memory seems to bounce around between 0.2% and 0.3%.

So, looking at your superb performance graphs I thought that I could learn how to do it properly on Julia ;).

My benchmark script is a bit of a mess, so I wouldn't follow it too closely ;).

I guessed that you use Distributed in order to speed up the measurements but It may also be a proper way to release the memory allocated by @belapsed.

The computer has 18 cores, so I run them on 17. Maybe the extra noise of going with 18 would be fairly negligible.
I've disabled turboing, so the all-core and single-thread speeds are equal, meaning the times should be fairly independent of just how many cores were working at the time.

Cache conflicts might be an issue -- but I assume in "real world" use cases where you care about performance, you'll run into those sorts of problems anyway.

Also, you may want to set
BenchmarkTools.DEFAULT_PARAMETERS.seconds = 0.1
or something instead of the default 5. I leave it at 5, but as a result the benchmarks take several hours to run.

For Intel's compilers, I think that you can obtain free licenses being an OS developer. Let me know if you have problem to get licenses.

An easy installation would be key to gather data from many users and improve the machine dimension. I thought that a docker distribution for this purpose maybe appropriate (not having to install all the prerequisite manually).

That'd be a good idea.
I don't suppose I could distribute Intel's compilers within the docker container, but maybe precompiled binaries for different instruction sets?

[LaurentPlagne]

Huh, I never noticed that. I did not deliberately add any workarounds. Is there a leak, or why doesn't the GC take care of it?

well, you may try the following MWE and check the memory consumption (even after go() function ends)

using BenchmarkTools

function benchmark_all(fs,ss)
    MEAS=Vector{Tuple{Int,Float64}}
    perfs = Vector{Tuple{String,MEAS}}()
    for f in fs
        @show f
        fperfs=MEAS()
        for s in ss
            @show s
            x=rand(s)
            t=@belapsed $f.($x)
            push!(fperfs,(s,t))
        end
        push!(perfs,(string(f),fperfs))
    end
    perfs
end


function go()
    BenchmarkTools.DEFAULT_PARAMETERS.seconds = 0.01
    BenchmarkTools.DEFAULT_PARAMETERS.gctrial = false

    sizes = [2^i for i in 1:27]

    functions = [sin,cos,exp]
    perfs = benchmark_all(functions,sizes)
    @show perfs
end

go()

I don't suppose I could distribute Intel's compilers within the docker container, but maybe precompiled binaries for different instruction sets?

I guess you are right. But the docker image could include all other libs and OS compilers (e.g. Eigen).

BTW I think that there is room for improvement for interfacing Eigen with aligned Eigen's Map on Julia's Ptr
https://discourse.julialang.org/t/sum-performance-for-array-float64-2-elements/10873/14?u=laurentplagne

I should try to make a PR.

[chriselrod]

I can verify -- memory consumption was high. Repeating go() increased it further.

I also noticed that by the time the benchmarks finished, each process consumed about 0.9% of the 128GiB of RAM, which means they consumed about 20 GiB total.

Maybe the script should kill workers and add replacements between benchmarks?
I should confirm whether this actually frees the memory.

You should be able to go to activate and instantiate the /benchmark project to add all the Julia dependencies (including MKL_jll and OpenBLAS_jll, as well as miscellaneous ones like PrettyTables.jl and Gadfly.jl), and then include driver to run the benchmarks. This should work given that all the compilers are installed.

Regarding Eigen alignment, improvements are appreciated. It was only recently that I got it to start actually using AVX512, which dramatically boosted performance in some benchmarks when compiling with g++, but clang++ performance in many of them is still bad.

But on alignment, beware that small Julia arrays are not necessarily aligned:

julia> reinterpret(Int, pointer(rand(2,2))) % 64
0

julia> reinterpret(Int, pointer(rand(2,2))) % 64
48

julia> reinterpret(Int, pointer(rand(2,2))) % 64
48

julia> reinterpret(Int, pointer(rand(2,2))) % 64
16

Regarding performance, my understanding was largely based on this blog post.
That is, I've been operating under the assumption that on computers with AVX2, vmovupd is just as fast as vmovapd when it doesn't cross a 32-byte (Ryzen) or 64-byte (everything else) boundary.
But that when vmovupd does cross one of these boundaries, only 1 can be performed in a clock cycle rather than the usual 2 (while vmovapd cannot cross such a boundary).

Meaning that vmovapd works as a sort of free assert if you can guarantee that your loads are aligned, but that their performance is identical for aligned loads. Am I missing something here?

[LaurentPlagne]

Thanx, I think that you are right about alignment negligible impact for Eigen. I have also noticed that Eigen does not perform very well with Icc and clang and very well with gcc.

I "succeed" on a first launch of the benchmark, but it dies after the first table...

julia> using BenchmarkTools

julia> BenchmarkTools.DEFAULT_PARAMETERS.seconds = 0.1
0.1

julia> include("benchmark/driver.jl")
logdet(LowerTriangular(A)) benchmark results:
┌──────┬─────────────────────┬─────────────────────┬─────────────────────┬─────────────────────┬─────────────────────┬─────────────────────┬─────────────────────┐
│ Size │   LoopVectorization │               Julia │               Clang │            GFortran │                 icc │               ifort │       LinearAlgebra │
├──────┼─────────────────────┼─────────────────────┼─────────────────────┼─────────────────────┼─────────────────────┼─────────────────────┼─────────────────────┤
│  256 │ 0.40644787716906067 │ 0.09410227087077276 │ 0.19258256225080872 │  0.2236589201467762 │ 0.47329498767460976 │  0.4833116386291597 │ 0.11180124223602486 │
│  255 │ 0.29107708212185823 │ 0.06414489199132158 │ 0.13147040626933387 │  0.1499823550170568 │ 0.32348380979228863 │ 0.32303676310637375 │ 0.08511348464619492 │
│  254 │ 0.38258085295845096 │ 0.09505197505197507 │  0.1921040689759492 │ 0.21210855949895618 │   0.468703318590551 │  0.4787365956729196 │ 0.11166471277842908 │
│  253 │   0.382433190413264 │ 0.09347674370869083 │ 0.19213244228432563 │ 0.21408021661871723 │  0.4718612155481216 │  0.4832203355932881 │ 0.11450842343474982 │
│  252 │  0.4044377676796895 │ 0.09369964883288578 │ 0.19189765458422178 │ 0.22200687164126512 │ 0.47401247401247404 │ 0.48237839346779265 │ 0.11399276236429436 │
│  251 │ 0.38931922013950254 │  0.0944319036869827 │ 0.19186668705090965 │ 0.21944395873404443 │ 0.46614849461520325 │ 0.47519405334847903 │ 0.11314800901577761 │
│  250 │ 0.37700019548158287 │  0.0927299703264095 │ 0.19247055200554317 │  0.2146106961970985 │ 0.46498905908096294 │ 0.47463321537911024 │ 0.11113306332114987 │
│  249 │ 0.37882814774466816 │ 0.09432612172741814 │ 0.19217411437832832 │ 0.21371556089606047 │  0.4730779560533567 │  0.4826117240891976 │ 0.11346260948812718 │
│  248 │  0.4002894194912896 │ 0.09376969289585346 │ 0.19239720713731573 │ 0.21985815602836878 │ 0.47415415415415424 │ 0.48095998060645245 │ 0.11282414194004953 │
│  247 │  0.3860639243764332 │ 0.09608402489626558 │ 0.19211324570273006 │   0.220064148253742 │ 0.46263801261829657 │  0.4721193683325661 │ 0.11473548387096774 │
│  246 │  0.3796725014116319 │ 0.09432515337423314 │  0.1914694894146949 │ 0.22104411896846074 │  0.4682212257100149 │ 0.47687896288518233 │ 0.11061703722208345 │
│  245 │ 0.37822438038566264 │ 0.08504647664596753 │ 0.19200626959247652 │ 0.21427322022039533 │  0.4693533665660325 │  0.4828153079562164 │  0.1144384471662861 │
│  244 │ 0.40376956302238187 │  0.0923659305993691 │ 0.19203525893278767 │ 0.21930612978608666 │  0.4721390778533636 │ 0.48392798421571853 │ 0.11139856947192209 │
│  243 │  0.3834203179997573 │  0.0954604975993016 │ 0.19207967749585014 │ 0.21623064602242392 │  0.4656240955679085 │ 0.47442213342774775 │ 0.11461663434830459 │
│  242 │ 0.37962911087455886 │ 0.09492677824267784 │  0.1921397379912664 │ 0.21630318198069362 │ 0.46785415806591285 │ 0.47716638401340394 │  0.0926217308101212 │
│  241 │ 0.37842435345029024 │ 0.09522346123452455 │ 0.19227700654220517 │ 0.22120238641578707 │ 0.47158601251127946 │   0.480721380246355 │ 0.11496263316902527 │
│  240 │  0.3908176463472508 │ 0.08941877794336811 │ 0.19176987614862168 │ 0.20650490449148168 │  0.4521869512892853 │ 0.45750785967363644 │ 0.10914603335017688 │
│  239 │  0.3645781666744654 │ 0.08963993998999835 │  0.1895321173671689 │  0.2096123487107525 │  0.4463669871636511 │  0.4496930085165379 │ 0.10318031371420347 │
│  238 │  0.3604703668861713 │ 0.09005297233666865 │  0.1898683685680096 │ 0.20947016370357333 │  0.4454601873651651 │ 0.45067713600222076 │ 0.10844471445929528 │
│  237 │  0.3770970773510994 │ 0.09385725600633636 │  0.1918717616580311 │ 0.22226390321673078 │ 0.47401466024722416 │ 0.48168208437660426 │ 0.11110532347119491 │
│  236 │   0.401828717639976 │ 0.09365079365079367 │ 0.19166734345813372 │  0.2178126442085833 │  0.4755148884018986 │  0.4832645478630765 │ 0.11196036055031365 │
│  235 │  0.3881877648880037 │ 0.09545946921827045 │ 0.19210332706613262 │ 0.22226425801570038 │  0.4649696547199213 │ 0.47276837564294016 │  0.1149518995597587 │
│  234 │  0.3796753046932552 │ 0.09505325871095865 │ 0.19214977828871735 │ 0.21678710394663703 │ 0.46739287111587013 │   0.477406124776363 │ 0.11392405063291139 │
│  233 │ 0.38032965280448794 │  0.0955309553095531 │ 0.19226008746596254 │ 0.22192589770454327 │  0.4705159833036671 │ 0.48068174604853897 │ 0.11531481990651635 │
│  232 │  0.3993757389453772 │  0.0825329064389897 │ 0.19198940748096655 │ 0.21664020916985716 │ 0.46982288256526483 │  0.4795471650112891 │ 0.10975609756097561 │
│  231 │  0.3850396576426411 │ 0.09412350597609563 │ 0.19194017449106773 │ 0.21598877980364656 │ 0.46551294303403007 │  0.4739134551652306 │ 0.11592311938575803 │
│  230 │  0.3815119907784869 │ 0.09389884327511909 │  0.1920988891672931 │ 0.22149460708782742 │  0.4716435489468205 │  0.4798279680328658 │ 0.11135617838506646 │
│  229 │ 0.37674373326843513 │ 0.09451527102632305 │  0.1916799196450992 │  0.2168355269387369 │  0.4717788609597842 │  0.4827067022283404 │ 0.11253624256720232 │
│  228 │ 0.40512894395620236 │ 0.09565095790798492 │ 0.19113085757397938 │   0.218978102189781 │ 0.47530312699425653 │  0.4823039257795722 │ 0.11559521395254513 │
│  227 │  0.3853153909263224 │ 0.09392671601305687 │ 0.19172297297297297 │ 0.21724566944205184 │  0.4668909796641633 │   0.474925968842539 │ 0.11566289615815754 │
│  226 │  0.3805682285445752 │ 0.09521580376369254 │ 0.19160661297159812 │  0.2185052692642367 │  0.4691342254015904 │ 0.47777550073883923 │ 0.11237631147133409 │
│  225 │  0.3758784496355543 │ 0.09568586684307519 │  0.1917668115571465 │ 0.21707670043415342 │  0.4728715419257989 │   0.482538265583836 │ 0.11260697662779641 │
│  224 │  0.4032130068711895 │  0.0956039265898421 │ 0.19184652278177464 │ 0.21787763836202706 │  0.4773678659577476 │  0.4853284061413929 │ 0.11278384774180557 │
│  223 │ 0.35658164876816173 │ 0.08611516347721616 │  0.1794191004907877 │ 0.19889404209775244 │  0.4164352972561729 │ 0.41651348947524464 │ 0.10029985007496253 │
│  222 │ 0.37940118403535117 │ 0.09451724301054922 │ 0.19156096298213823 │ 0.21836716444603418 │  0.4709753536753007 │  0.4810374411544723 │ 0.11667017027538365 │
│  221 │  0.3789474715605679 │  0.0956986143187067 │ 0.19144144144144143 │ 0.21934758155230596 │  0.4747919779273041 │ 0.48470307942374924 │ 0.11271484673840974 │
│  220 │  0.4031346251315841 │ 0.09581881533101046 │ 0.19180470793374021 │ 0.21978021978021978 │ 0.47946720948756527 │  0.4871447902571042 │ 0.11581385554853654 │
│  219 │ 0.38403842671571664 │ 0.09593108147571304 │ 0.19180241723594327 │ 0.21918546462391256 │  0.4678722082338707 │  0.4760449309264245 │ 0.11542110256139981 │
│  218 │ 0.37918448816652417 │ 0.09265205893464301 │  0.1911274767666141 │  0.2226895805792573 │  0.4710644085644086 │ 0.47999910585552863 │ 0.11394522266360026 │
│  217 │   0.376395017488349 │ 0.09615479296932698 │ 0.19176387416048077 │ 0.21931006602883713 │ 0.47428273459515835 │  0.4821031843989139 │ 0.11306794497707379 │
│  216 │ 0.40296120817293457 │ 0.09627098499480019 │ 0.19182948490230906 │  0.2242721936024919 │  0.4773775199973075 │ 0.48666917010889976 │  0.1171493654409372 │
│  215 │    0.38236736106438 │ 0.09590127372751153 │  0.1917588298251873 │  0.2268445248323159 │  0.4707314091378908 │ 0.47767591647123514 │ 0.11376263294354198 │
│  214 │ 0.37838819436479726 │ 0.09728746779815124 │ 0.19173909147925813 │ 0.22115384615384617 │ 0.46897457004916904 │ 0.47953825260298777 │ 0.11267902274641955 │
│  213 │ 0.37886945461848154 │ 0.09737390155940469 │  0.1921342233447592 │  0.2262928818670962 │  0.4753231823458474 │  0.4841464814602227 │ 0.11694943172459232 │
│  212 │  0.4031742554737985 │ 0.09714372995265007 │ 0.19175108538350222 │ 0.22691446920307484 │ 0.47951746670017603 │  0.4901933444878231 │ 0.11280796041079126 │
│  211 │ 0.38279150537947365 │ 0.09584616161106346 │  0.1919403256617848 │ 0.22137928251482994 │  0.4698853745110896 │ 0.47880899443005487 │  0.1128885559895137 │
│  210 │ 0.37865201041365343 │  0.0957543824095653 │  0.1916932907348243 │ 0.22223006102508025 │  0.4680023077706257 │ 0.47606507825648897 │ 0.11182108626198083 │
│  209 │  0.3764206187155681 │  0.0972042788486383 │  0.1920602830362066 │ 0.22644879791501948 │  0.4765096034268343 │ 0.48591590433797155 │ 0.11629848088587169 │
│  208 │ 0.40358961920931363 │ 0.09597047062442325 │ 0.19213005726953633 │ 0.22283973350296293 │  0.4803862401931201 │  0.4854284675556181 │ 0.11545293072824157 │
│  207 │  0.3782107974148404 │  0.0903228934354698 │ 0.19101227276921656 │  0.2168198405189891 │  0.4561856530691747 │  0.4591497227356747 │  0.1046141405973619 │
│  206 │   0.378022020700235 │ 0.09451468189233281 │   0.191734921816828 │ 0.22695201706476645 │  0.4658823529411765 │  0.4728531057154392 │  0.1147632311977716 │
│  205 │ 0.37518374658061004 │ 0.09490252559024742 │ 0.19192959460724654 │ 0.22263641433561307 │  0.4765659680848579 │ 0.48766347096373164 │ 0.11405363302548126 │
│  204 │  0.3867278975700252 │ 0.08894486968317024 │ 0.19012115563839704 │ 0.21070422535211267 │  0.4475477749565682 │ 0.45291142930827627 │  0.1029886914378029 │
│  203 │  0.3834165636588381 │ 0.09686655002385876 │ 0.19161789692278647 │ 0.22713962623497203 │ 0.46733405420489027 │  0.4773648449039882 │ 0.11768115942028985 │
│  202 │  0.3804482509094875 │ 0.09711538461538462 │ 0.19157814871016693 │ 0.22834955273763885 │ 0.47509750620494035 │   0.483614051973051 │ 0.11885848779052663 │
│  201 │    0.37394796680743 │ 0.09485109060402686 │ 0.19201375620940012 │ 0.22601372283544072 │ 0.47698490305039476 │ 0.48652297662198657 │ 0.11822138571932714 │
│  200 │   0.404938580836386 │ 0.09659761725877429 │  0.1922522349322311 │ 0.22771969775385573 │ 0.48533625998414737 │  0.4930054846860171 │  0.1144230219120087 │
│  199 │ 0.38573678471913747 │  0.0962592712028378 │ 0.19164098613251157 │ 0.23230432849095764 │  0.4764087387517444 │ 0.48553164495193485 │ 0.11987229684958739 │
│  198 │ 0.37706765043122786 │ 0.09604915647065165 │ 0.19173041541590005 │ 0.22886966268520786 │  0.4772061747586855 │ 0.49041462327240304 │ 0.11529055549085827 │
│  197 │  0.3764356316779383 │ 0.09753923850076744 │  0.1912992814138668 │ 0.22806611334734445 │ 0.49602274862518636 │  0.5041185532229112 │ 0.11623104607941472 │
│  196 │  0.4019138755980861 │ 0.09578731306812628 │ 0.19084712755598832 │ 0.22149253731343285 │ 0.47332133930619785 │ 0.47942273588943923 │ 0.10790574763268003 │
│  195 │ 0.38962450695184947 │ 0.09707771195300444 │ 0.19134530468060051 │ 0.23102805509920385 │  0.4826492537313433 │  0.4928948692152918 │ 0.12084779375309868 │
│  194 │ 0.36957540818463813 │  0.0976346250629089 │  0.1906071919827078 │ 0.23108411904841797 │  0.4715777191681758 │ 0.47792079817700317 │ 0.11249637576109019 │
│  193 │  0.3851423633969718 │ 0.09691187547075071 │ 0.19192521877486082 │ 0.23436383896682766 │  0.5014327742329883 │  0.5153137272691924 │ 0.11696969696969697 │
│  192 │  0.4158860007916612 │ 0.09769003765136867 │ 0.19198254704117806 │ 0.23724857554747034 │  0.5084220316213703 │  0.5200378668349638 │ 0.11740964960557697 │
│  191 │ 0.34585031905725605 │ 0.08450911951231503 │  0.1759557807461999 │ 0.20183653390664288 │  0.4224382795747213 │  0.4274775337703503 │ 0.08480513073507646 │
│  190 │  0.3894342954445122 │   0.097425905035381 │  0.1916808939934813 │ 0.23760209465004498 │  0.5026974006866112 │  0.5138139504647988 │ 0.11788794440652728 │
│  189 │ 0.38057377729801545 │  0.0989943431803897 │ 0.19153094462540718 │ 0.23732960503320516 │  0.5033093748384094 │  0.5209686946249262 │ 0.12235385511749855 │
│  188 │ 0.42010224925795936 │ 0.09762176757711083 │ 0.19187344517445942 │ 0.23511020790995782 │  0.5320298430666324 │  0.5431627606966245 │ 0.10009583643914387 │
│  187 │  0.3930087926227751 │ 0.09878499735868992 │ 0.19182766284835015 │ 0.23585418933623506 │  0.5061985092193018 │  0.5142879593493805 │  0.1225104821802935 │
│  186 │ 0.38238115821323526 │ 0.09821522864082798 │ 0.19215920830841174 │ 0.24177239943404694 │  0.5132694288204073 │  0.5179677819083024 │ 0.11872087827918557 │
│  185 │  0.3838296785725685 │ 0.09935553168635876 │ 0.19207000543827557 │ 0.24279531871969803 │   0.544882187635106 │  0.5622638409725645 │ 0.12464627408705026 │
│  184 │ 0.42488309216862574 │   0.099723592217224 │ 0.19224924012158054 │ 0.24363127024424408 │  0.5315322646269021 │  0.5408813397001903 │ 0.11956592371174216 │
│  183 │  0.3948046230049533 │  0.0985407355554359 │ 0.19116177672813153 │ 0.24578657770460555 │  0.5130117423040305 │  0.5219346537017631 │ 0.12405097613882864 │
│  182 │ 0.38835393130639173 │ 0.09950248756218906 │ 0.19208037825059104 │ 0.24823620610140223 │  0.5425977653631285 │  0.5586128910667169 │ 0.12543073742246727 │
│  181 │  0.3949776241747531 │ 0.09986757890090489 │ 0.19205699560406247 │  0.2432306601627393 │  0.5333466666666667 │  0.5572600161333692 │ 0.12035374692466255 │
│  180 │ 0.44487284051308673 │ 0.09911348494025661 │ 0.19216610219408042 │ 0.25125527214300064 │  0.5474748567102845 │  0.5596279867597594 │ 0.12048999263672269 │
│  179 │  0.3954432865463147 │ 0.09934509934509934 │ 0.19219978525163658 │ 0.24341197074276294 │   0.543456260373234 │   0.561666735177649 │ 0.12149596144709157 │
│  178 │ 0.38975647167357647 │  0.0998541456299787 │ 0.19228754746628257 │  0.2475208331218725 │  0.5342778244687237 │  0.5494330146356776 │ 0.12080081438751274 │
│  177 │ 0.38647144883844653 │ 0.10019813189923579 │ 0.19208706706768783 │ 0.24624373956594325 │  0.5375209781826901 │  0.5468303938054828 │ 0.12552301255230128 │
│  176 │ 0.43963180836049814 │                 0.1 │ 0.19226454089164455 │ 0.25160579376346437 │  0.5567103479439675 │  0.5748439648788745 │ 0.12155535603287519 │
│  175 │ 0.40522457528508266 │ 0.09898749929294644 │   0.192193968066233 │ 0.24026290165530673 │  0.5276882659730041 │   0.540172698975007 │  0.1236836525549509 │
│  174 │ 0.38817405412713796 │ 0.09925841414717629 │  0.1921719688602753 │  0.2507932208636434 │  0.5359012319568551 │  0.5638396770166539 │  0.1259682907406067 │
│  173 │  0.3958535570655943 │ 0.09925415949512334 │ 0.19223678561507185 │ 0.24791099836365574 │  0.5517273588556331 │  0.5667029146025481 │  0.1215485140167217 │
│  172 │   0.449751397583533 │ 0.09949673164805924 │ 0.19227217496962334 │ 0.25633869666334425 │  0.5615817159441321 │  0.5813778177662137 │   0.126007326007326 │
│  171 │  0.4129736517980052 │ 0.10005265929436546 │  0.1922833649588868 │ 0.25332606950338343 │  0.5288519996827998 │  0.5429345290549997 │ 0.12589265994257529 │
│  170 │ 0.36619006275345917 │ 0.09859645052778102 │  0.1904939770286675 │ 0.22719204827831027 │  0.4957134026713968 │  0.5077301475755447 │ 0.11707988980716254 │
│  169 │ 0.39776463630987585 │  0.1004756242568371 │ 0.19233412067854844 │  0.2489960999566662 │  0.5415426871137561 │  0.5586706903244387 │ 0.12614764499514816 │
│  168 │  0.4465304429001142 │ 0.10038840752913059 │ 0.19219169249329204 │  0.2606006000427275 │  0.5715741991386122 │  0.5870779048125498 │ 0.12568265130545375 │
│  167 │  0.4140889423374948 │ 0.09970149253731343 │ 0.19224089183294288 │ 0.25399478002891634 │  0.5607642775312414 │  0.5667862994460482 │  0.1271896420411272 │
│  166 │ 0.39560729084333307 │ 0.10010251462340952 │  0.1921926557706038 │ 0.25619919744829717 │  0.5528469376385088 │  0.5656629126589009 │ 0.12654368043909134 │
│  165 │  0.4049228805968318 │ 0.09961963412425286 │ 0.19214062586643502 │  0.2570551744202416 │   0.564081753117391 │  0.5773310250658344 │ 0.12095000733030348 │
│  164 │  0.4414162559975035 │ 0.10018938236911235 │ 0.19213632732759386 │  0.2559801983855774 │  0.5741993173889498 │  0.5775946047208693 │ 0.12672900085001162 │
│  163 │ 0.40706458266033846 │ 0.10032003938946334 │ 0.19180120784232293 │ 0.24425649276028502 │  0.5296879597763648 │  0.5342352665754877 │ 0.12119860212655217 │
│  162 │  0.4120424583537945 │ 0.09996297667530546 │  0.1921740292722048 │  0.2533170487980903 │  0.5477493289283503 │  0.5509799603611539 │   0.126128931796948 │
│  161 │  0.3972170137175565 │ 0.09988212668279668 │ 0.19226553681558672 │  0.2592664991157032 │  0.5746037215713302 │  0.5966634303440763 │ 0.12214551248008498 │
│  160 │ 0.44064357153210615 │ 0.10016276449229998 │ 0.19222865092162256 │  0.2525651144435675 │  0.5469818323891384 │  0.5640945148086884 │  0.1212121212121212 │
│  159 │   0.369923071527112 │   0.096305269533616 │ 0.18848217861433722 │ 0.23041029455806628 │ 0.45841508829535904 │  0.4620103314679294 │ 0.11398666571080363 │
│  158 │ 0.40732043493797543 │ 0.10054088450524977 │  0.1920858722886922 │  0.2602384076379778 │  0.5620266970737331 │  0.5871821241585641 │ 0.12210200927357033 │
│  157 │  0.4042901359853867 │ 0.09940483727997974 │  0.1918031356351261 │  0.2507408398954977 │  0.5576120592648163 │  0.5719948673296367 │ 0.12148881838582375 │
│  156 │  0.4583925935864753 │ 0.10021198689535557 │ 0.19216446967873477 │  0.2578994069023491 │  0.5855266169268093 │   0.598003141372758 │ 0.12688084587230583 │
│  155 │  0.4217962975076862 │  0.1003106393994305 │ 0.19206939281288724 │ 0.25867534356411637 │  0.5631621941257514 │  0.5835749015441116 │ 0.12690355329949238 │
│  154 │  0.4162107555660513 │   0.099657024525982 │ 0.19206115059675474 │  0.2586986180589715 │  0.5605564736506702 │  0.5624863304317397 │ 0.12187401076289966 │
│  153 │ 0.41357320001574616 │ 0.10022928267278088 │ 0.19204518710284774 │  0.2564349853463047 │  0.5698069868313951 │  0.5870503597122303 │ 0.12647763908407042 │
│  152 │  0.4533043501312197 │ 0.10024401503660227 │ 0.19205528551720583 │  0.2626703306042664 │  0.5706929843381249 │  0.5877402479244854 │   0.126635007914688 │
│  151 │ 0.41797231971392623 │ 0.10025894694907378 │ 0.19211195928753183 │  0.2514430048178119 │  0.5291427783586291 │  0.5426465159707294 │ 0.12729725172820774 │
│  150 │ 0.41836558511415406 │ 0.10065087566261828 │ 0.19206380176576754 │ 0.26205600022787484 │  0.5710830421929589 │  0.5776929168881609 │ 0.12755102040816327 │
│  149 │ 0.41445087467208036 │ 0.10062128579146408 │  0.1920195596666185 │ 0.25769238078947865 │  0.5733253847821375 │  0.5987911130472664 │ 0.12307946472823394 │
│  148 │ 0.44874130616912905 │  0.1002642097418874 │ 0.19201283230722055 │ 0.25761288064403215 │  0.5839398151933927 │   0.588598551086685 │  0.1270713488451962 │
│  147 │  0.4223117196939186 │ 0.10043041606886657 │ 0.19208131453878977 │  0.2576269894941375 │  0.5563025210084035 │  0.5725367240219584 │ 0.12212345268754674 │
│  146 │  0.4101560419662353 │  0.1002816127481283 │ 0.19158355964307722 │ 0.26024456625905623 │  0.5599522311278506 │  0.5634655414908579 │ 0.12275096687405415 │
│  145 │  0.3925253295136471 │ 0.10021425115764739 │ 0.19186991869918701 │  0.2564956061931754 │  0.5656592523633786 │  0.5769642802485385 │ 0.12286053211320117 │
│  144 │  0.4606538615335825 │ 0.10076271779441608 │  0.1918486527295134 │ 0.26024683244580604 │  0.5905409582689336 │  0.5958419748090446 │ 0.12252190929975328 │
│  143 │  0.4066187857104715 │ 0.09985336219537742 │  0.1918102508178844 │ 0.24957540840550604 │   0.536509871969235 │  0.5467590301830777 │ 0.12183692596063729 │
│  142 │  0.4175062377773282 │ 0.10019050306921612 │ 0.19168220093848426 │   0.262584754468872 │  0.5627573178369388 │  0.5818082491122644 │ 0.12287988923502942 │
│  141 │  0.4095326052039833 │ 0.10041304657456203 │  0.1916810690047633 │  0.2600047080979284 │  0.5829182148544364 │  0.5971136676326806 │ 0.12249152984102164 │
│  140 │  0.4516188013425799 │ 0.10015739018457577 │ 0.19195545461205948 │ 0.26263304437116175 │   0.594406286494725 │  0.6052616273434016 │ 0.09900289937062444 │
│  139 │   0.417310162958956 │ 0.10065899051343326 │  0.1917734877734878 │  0.2574982357092449 │  0.5635451670780425 │  0.5703596427709389 │ 0.12294356978595439 │
│  138 │ 0.42026321791277055 │ 0.10000000000000002 │ 0.19182534572103152 │ 0.26299352444049773 │  0.5731281444214266 │  0.5862775168840076 │ 0.12263396427619302 │
│  137 │  0.4133428942526247 │  0.1002781437564046 │ 0.19182456458452954 │ 0.26019592163134747 │  0.5911113690890798 │  0.6101426455105331 │ 0.12280387235568305 │
│  136 │ 0.45441527446300717 │ 0.09937162063422476 │  0.1917676707068283 │ 0.25627801801086997 │  0.5914273289341829 │  0.5951777354231046 │ 0.12849584278155707 │
│  135 │  0.4267422474158053 │ 0.10067864866880452 │ 0.19175564936677428 │  0.2602488026747794 │   0.558171435242385 │  0.5774075737749741 │ 0.12314147587339233 │
│  134 │ 0.41274730482170363 │ 0.10108630054315029 │ 0.19173269915827076 │ 0.26184329091111586 │  0.5787703270237455 │  0.5887209786766656 │ 0.12343404568901993 │
│  133 │   0.407323330691633 │ 0.10068892421833599 │ 0.19160696610053218 │  0.2584188219186357 │  0.5788646765489446 │  0.5960942029652398 │  0.1286142539406247 │
│  132 │  0.4512911270211568 │ 0.10004547521600729 │  0.1916604137635331 │  0.2584882760308832 │  0.6001736548749406 │  0.6091762811913067 │  0.1224603395491233 │
│  131 │ 0.43116869547045605 │ 0.10019886798225484 │ 0.19168192873823853 │  0.2597684143472141 │  0.5751712824471678 │  0.5801494659883748 │ 0.12309716218755871 │
│  130 │  0.4201380150730473 │ 0.09924421711581036 │ 0.19166096533914068 │  0.2603650479022134 │   0.579237315784077 │  0.5959718297223343 │ 0.12285012285012284 │
│  129 │ 0.41405873856523834 │ 0.10046728971962618 │ 0.19152665759880458 │ 0.26005902402527226 │  0.5936990570201115 │  0.6140095724145226 │ 0.12310334955625535 │
│  128 │  0.4700744416873449 │ 0.10027418723070898 │ 0.19148895622510625 │  0.2622317010390511 │  0.6047693336304881 │  0.6145700864239184 │  0.1276073619631902 │
│  127 │ 0.34203163812263043 │ 0.09464897898345506 │ 0.18510101736520632 │ 0.22379393355801638 │  0.4367792838357274 │  0.4415800519067665 │  0.1145899124785708 │
│  126 │ 0.40795020549483957 │ 0.10037441249103803 │ 0.19137194230351973 │  0.2620017488110218 │  0.5867461315544864 │  0.5958542416519981 │ 0.12337217272104184 │
│  125 │  0.4127646647030076 │ 0.09970487357422032 │  0.1914116077635609 │  0.2607872297174404 │  0.5924737354762403 │  0.6134092325023685 │ 0.12371724748658645 │
│  124 │  0.4699820224204463 │ 0.09951047267474522 │  0.1915497177309878 │  0.2613361433993914 │  0.5947884706797459 │  0.6099845766701842 │ 0.12816979051819183 │
│  123 │ 0.42952578571790073 │ 0.10045736687357072 │ 0.19155103993695646 │  0.2612880150868142 │  0.5667676185875401 │  0.5821751170012298 │ 0.12401116798510937 │
│  122 │  0.4156090314152889 │ 0.10035370568396808 │ 0.19142438195216258 │ 0.26302887035699435 │  0.5858888273570539 │  0.5946557576812603 │ 0.12912680643191535 │
│  121 │  0.3902403706921518 │ 0.09981851179673322 │  0.1912587216371425 │ 0.25583603020496226 │  0.5823357024381441 │  0.5938011192423591 │ 0.12364061017443984 │
│  120 │ 0.47053298896454576 │ 0.10080645161290323 │  0.1911789367767902 │  0.2650788845157601 │  0.6092826586879653 │  0.6120660149760834 │  0.1242033011930054 │
│  119 │ 0.42767034202870213 │ 0.10052373711775639 │ 0.19118608604681575 │   0.262622528426595 │  0.5736075126746258 │  0.5859897810587631 │   0.128995171303748 │
│  118 │  0.4140969162995596 │ 0.10046828437633037 │ 0.19104333181357178 │ 0.26198993036477203 │  0.5780807585960659 │  0.5953740862238357 │ 0.12778569392989456 │
│  117 │  0.3917258204752203 │ 0.09987195902688861 │  0.1911657610217915 │ 0.25418233549112806 │  0.5677124037065981 │  0.5706741119483315 │  0.1291877625767532 │
│  116 │ 0.47067636318237155 │ 0.09966491966663803 │ 0.19125058548009372 │ 0.26368781787079665 │  0.6083733564608373 │  0.6157775118656591 │  0.1289155380957304 │
│  115 │ 0.42611130062461033 │ 0.10036655611799618 │ 0.19128517096053754 │ 0.26076201600989457 │   0.571782511966561 │  0.5841558848510336 │  0.1241509603706536 │
│  114 │  0.4071779586910202 │ 0.09974625951526817 │  0.1911604901262787 │ 0.25948704978904896 │  0.5752350183601589 │  0.5975571562793611 │ 0.12389380530973453 │
│  113 │  0.3860083558738746 │  0.0935198212364479 │     0.1911636168833 │ 0.25323991805412505 │  0.5547373588610703 │   0.561337037972552 │   0.113638121760656 │
│  112 │ 0.46781827743840776 │ 0.10052055286304075 │ 0.19106503298774744 │  0.2628115935193265 │  0.6042602775804689 │  0.6116488717579365 │ 0.12897685130967718 │
│  111 │ 0.40719491953525294 │ 0.09981116806042623 │ 0.19101378566971122 │     0.2593792933385 │  0.5469131643224122 │  0.5582159083426615 │ 0.12429348405673457 │
│  110 │ 0.41469834373527353 │ 0.09981851179673322 │ 0.19101752654602733 │ 0.26121718377088304 │  0.5694104907381582 │   0.595212947465917 │  0.1289350301406564 │
│  109 │  0.4058972317204553 │ 0.09886621315192744 │ 0.19097314797181492 │  0.2617220492772509 │  0.5929913412841039 │  0.6081075657234598 │ 0.12915869109470424 │
│  108 │  0.4697582379764261 │  0.0989010989010989 │ 0.19099329898385445 │ 0.26472534745201853 │   0.603806502775575 │  0.6181647388490411 │  0.1295398934894209 │
│  107 │  0.4243263798348544 │ 0.09948860994886101 │ 0.19101824568812448 │  0.2600464195010511 │  0.5680824484697066 │  0.5751573849878935 │ 0.12971858946188003 │
│  106 │  0.4106365112243184 │ 0.09998113563478589 │ 0.19097619093744556 │  0.2594795721034981 │  0.5795046762734315 │  0.5854864218331519 │ 0.12888272507931683 │
│  105 │   0.399003344938221 │ 0.10031527658354829 │ 0.19088310779281378 │ 0.26204796725648255 │  0.5920503256694316 │  0.6079717457114027 │ 0.12897572799633136 │
│  104 │ 0.47025953973972506 │ 0.10066789274997581 │ 0.19085549178087544 │  0.2644368825188802 │  0.6067639044718706 │  0.6212664277180406 │ 0.12898780403775925 │
│  103 │ 0.41548171484524676 │ 0.09897184587297012 │ 0.19064547404295326 │  0.2582612926162943 │  0.5583193007778046 │  0.5773818043032298 │  0.1287805006448896 │
│  102 │ 0.41019936681783176 │  0.0986555759744656 │ 0.19048465984882168 │    0.26369407187468 │  0.5738826329466432 │  0.5801827459356829 │ 0.12399447712810661 │
│  101 │  0.3997201237577759 │ 0.09927265578926676 │ 0.19057739289332037 │ 0.25741574876667045 │  0.5862919405873069 │  0.6060335354525548 │  0.1295751264055543 │
│  100 │  0.4655121964195462 │ 0.09928514694201748 │ 0.19065872088960764 │ 0.26379052693132354 │  0.5991085265125494 │  0.6160672247448349 │ 0.12903070207172662 │
│   99 │  0.4187317131426524 │ 0.09981515711645102 │ 0.19058610089200223 │  0.2576967956197828 │   0.563706160604576 │  0.5758113225636355 │ 0.12418010296077561 │
│   98 │  0.4071145028771887 │ 0.09147764398394474 │ 0.19063829787234043 │ 0.26289273609966113 │  0.5764065278130556 │  0.5896360652878738 │ 0.11778473605971472 │
│   97 │  0.3945762711864407 │ 0.09973153595704577 │  0.1905387112861694 │  0.2579619900110268 │  0.5892196694306508 │  0.5972372870541697 │  0.1294146322689895 │
│   96 │  0.4695235016926663 │ 0.10013749940733016 │ 0.18947497596792998 │   0.263946445648709 │  0.6053888782725015 │  0.6061031217116802 │  0.1290792797094711 │
│   95 │ 0.36771119542936886 │ 0.09787087912087912 │  0.1903139198678232 │ 0.25153804220855075 │ 0.46499237646406544 │  0.4697563944347709 │ 0.12715803041382132 │
│   94 │ 0.40139715649136615 │ 0.10005321979776478 │  0.1902595776646357 │ 0.26006724735908504 │  0.5724925955041014 │  0.5852093606816431 │ 0.12822033496228116 │
│   93 │  0.3970800909028304 │ 0.10001040691018838 │  0.1903037934833937 │ 0.25810458984762186 │  0.5877724751709016 │  0.6025075233769274 │ 0.12343021569895833 │
│   92 │  0.4286589099816289 │ 0.09828731679631424 │ 0.19041149686362335 │  0.2569254185692542 │  0.5242726141506369 │  0.5369818989555273 │  0.1280862188122878 │
│   91 │ 0.41691339580786824 │ 0.09930281903607154 │  0.1902520611980144 │   0.257835157638694 │  0.5589351494561976 │   0.569235556477544 │  0.1294209737167049 │
│   90 │ 0.40043719229690394 │ 0.09925758553905746 │ 0.19002682351420358 │ 0.25974548969072164 │  0.5669382442269683 │  0.5845763095738831 │ 0.12359915713551879 │
│   89 │ 0.39023181332926643 │ 0.09985200038193451 │ 0.19020411723656663 │ 0.25881846087565297 │  0.5863138142370479 │   0.593712349702032 │ 0.10739480198019802 │
│   88 │  0.4556648785159196 │ 0.09979587207983671 │  0.1898587230314314 │  0.2605898267669902 │  0.5960661716816466 │  0.6051221743107279 │ 0.12401103215026273 │
│   87 │ 0.40974283327705413 │ 0.09964412811387902 │  0.1898469172998959 │  0.2579247434435576 │  0.5473586222946242 │  0.5667683812845103 │ 0.12890612942060156 │
│   86 │  0.3996139686170784 │ 0.09967932619866322 │ 0.18970092599849958 │  0.2635058861473956 │  0.5709255370699403 │  0.5838167739619285 │ 0.12882044547497154 │
│   85 │  0.3856850117096019 │ 0.10068704098554845 │   0.189899126667118 │  0.2579875884122382 │   0.584539597711513 │   0.599025365487942 │ 0.12373114919550492 │
│   84 │  0.4663386267054351 │ 0.09821510130259241 │ 0.18991938246511525 │  0.2598101180710578 │  0.6028200333552333 │  0.6110222431763459 │  0.1233265720081136 │
│   83 │ 0.40088049239186185 │ 0.10063230915620491 │ 0.18999722530521643 │ 0.25678811316914313 │  0.5446676597286526 │  0.5640704032529122 │ 0.12891213428269352 │
│   82 │  0.3911104235955346 │ 0.10018252472915685 │ 0.18967581453197105 │ 0.25890695029767874 │  0.5602136093552987 │   0.576481049053907 │ 0.12361574635922333 │
│   81 │  0.3849776312691312 │ 0.10002554931016865 │ 0.18974020929219687 │  0.2563599214775049 │  0.5817289985602839 │  0.5973147535981285 │ 0.12358725135623871 │
│   80 │  0.4672578715040853 │ 0.10077946618376507 │ 0.18950801718904378 │  0.2620328593414977 │  0.6022143275963163 │  0.6151429194850536 │ 0.12849517653396014 │
│   79 │  0.4090879248937253 │  0.1010739724679223 │ 0.18926597883534188 │  0.2549836633728852 │   0.550151049941144 │  0.5597618176374706 │  0.1281085083846536 │
│   78 │  0.3915498849613052 │ 0.09974049418932643 │ 0.18914593336243274 │ 0.25957585644371944 │  0.5613683616865462 │  0.5722336455715482 │  0.1238106798382805 │
│   77 │ 0.37897237897237895 │ 0.09885185544036432 │  0.1890753723188744 │  0.2578426155463548 │  0.5804070214217256 │  0.5845640369384014 │ 0.12326343381389253 │
│   76 │ 0.46561825470243606 │ 0.10046265697290153 │  0.1891935624400991 │  0.2583336552203639 │  0.6007263405255288 │  0.6129252433811115 │ 0.12895298832753124 │
│   75 │ 0.40565376151669774 │ 0.10097365530017449 │  0.1890922318527903 │ 0.25522605736509485 │  0.5465278659388093 │   0.554825553119054 │ 0.12348219798312411 │
│   74 │ 0.38989935239710943 │ 0.10097723288792788 │ 0.18920082019087162 │ 0.26381797578273214 │  0.5628330276505308 │  0.5744809197010097 │ 0.12324217105324714 │
│   73 │  0.3749764090950119 │ 0.10026946829815565 │ 0.18895680364945733 │   0.256296609852965 │  0.5783964999021823 │  0.5816447620832107 │ 0.12851497842456253 │
│   72 │ 0.46281781686102097 │ 0.09938455471028408 │ 0.18878598829774867 │ 0.25030096061715357 │  0.5616877399358998 │  0.5757429718875503 │ 0.12333317560213886 │
│   71 │  0.3972808228963284 │ 0.10047203663085118 │ 0.18870737715545183 │  0.2608021562570812 │   0.538687544287171 │  0.5438886328658257 │ 0.12338679809692032 │
│   70 │  0.3827083470151705 │ 0.10085071350164655 │ 0.18874839655488362 │ 0.26642693690748614 │  0.5595603454428664 │  0.5710078681830911 │  0.1288813694267516 │
│   69 │ 0.36837599244009017 │ 0.10003210835109023 │  0.1886956178755966 │  0.2589373888998618 │  0.5672744120131906 │  0.5790486588513958 │ 0.12291293546709804 │
│   68 │  0.4644520335299597 │  0.1006618468679267 │ 0.18885625822226357 │  0.2638032036104534 │  0.5947465200697415 │  0.6062338816393158 │ 0.12319198612685404 │
│   67 │ 0.39465912859466545 │ 0.10050604308486903 │ 0.18854109707792643 │ 0.26132590402547196 │  0.5391073716656363 │  0.5474708171206226 │  0.1280827127872528 │
│   66 │ 0.37974502220312284 │ 0.10018405024381914 │ 0.18852085288990214 │  0.2619892854837669 │  0.5557649636527375 │  0.5587021912628864 │ 0.12807244501940493 │
│   65 │  0.3647296662713605 │ 0.10114360704911886 │ 0.18846902476676286 │ 0.25982361843135204 │  0.5704752526363088 │  0.5821931101407084 │ 0.12241175945841469 │
│   64 │ 0.46471132027901507 │ 0.10024761761836873 │ 0.18806782950961098 │  0.2656958975268335 │  0.5964346196627018 │  0.6081162192132009 │ 0.12867285178747198 │
│   63 │ 0.34987323433538575 │ 0.10053003863309756 │ 0.18786684837608456 │ 0.23978424321673755 │ 0.45193308051146097 │  0.4594968765828128 │  0.1278809151266546 │
│   62 │  0.3745408853663251 │ 0.10100367013706837 │   0.187598337838383 │ 0.26256003507443354 │  0.5482765920594291 │  0.5493534980374047 │  0.1282309571777616 │
│   61 │  0.3627181263924016 │ 0.10038898862956314 │  0.1879928942481092 │ 0.26210937500000003 │  0.5650795233584286 │  0.5776222643076331 │ 0.12251785899161403 │
│   60 │  0.4620479257385728 │ 0.10028639842246118 │ 0.18812574012272584 │ 0.26321760835035174 │  0.5803774208200851 │  0.6040483409488834 │ 0.12823744752677096 │
│   59 │   0.388251572731807 │ 0.10046543307980475 │ 0.18785553468419863 │  0.2618084445082007 │  0.5285253585643866 │  0.5310246854718652 │ 0.10094336055656383 │
│   58 │  0.3688541616732979 │ 0.10103567756920383 │  0.1874398174407408 │  0.2620145073914241 │  0.5471421196999909 │  0.5427403518614997 │ 0.12842585688786462 │
│   57 │  0.3532720973842291 │ 0.09380022962112515 │ 0.18767216727444447 │ 0.25966426236794105 │   0.557833409763622 │   0.573815293627833 │ 0.11682560584499713 │
│   56 │ 0.45473410686251514 │ 0.10093202124275924 │ 0.18703053931124108 │  0.2636085681750935 │  0.5811879999563477 │   0.592704357859235 │ 0.12697389622945537 │
│   55 │ 0.37808894861038284 │ 0.10124549584834718 │  0.1868317099218961 │ 0.25980160604629193 │   0.521904336164506 │  0.5226170997672741 │ 0.12301192842942348 │
│   54 │ 0.36077404048142747 │ 0.10121237829359482 │ 0.18662025124867562 │  0.2649992808129352 │  0.5396051669510115 │  0.5389415691052604 │  0.1275822766775894 │
│   53 │  0.3472294793366506 │  0.1011986284251882 │ 0.19169275992076173 │  0.2616306522093732 │  0.5583711310480964 │  0.5735579377233283 │ 0.12813441539508214 │
│   52 │  0.4557444901151092 │  0.1005425979915776 │ 0.18686745220750015 │ 0.26221759933430416 │   0.576966376899061 │   0.565307564327552 │ 0.12799010513296227 │
│   51 │  0.3761731566497953 │ 0.10100875347624863 │ 0.18674433810854188 │  0.2593350898685955 │  0.5143205102182529 │  0.5231930843227316 │ 0.12246597161888213 │
│   50 │ 0.35020325034978933 │ 0.10096699316130779 │ 0.18669342605773492 │ 0.26474885181328756 │  0.5366756582882812 │  0.5458140917196288 │ 0.12773491842671053 │
│   49 │  0.3391284191006027 │  0.1008869179600887 │ 0.18643600206908445 │   0.258143091825752 │  0.5376572643189521 │  0.5503287036492573 │ 0.12737197816480375 │
│   48 │ 0.44760618340087055 │ 0.10102984289258064 │ 0.18581338106959053 │ 0.26475090510014826 │   0.587263792492021 │  0.5888672991198994 │ 0.12732095490716183 │
│   47 │   0.364395285140492 │  0.1009158231453486 │ 0.18616992414294553 │ 0.26089630440165174 │  0.5086799831743614 │  0.5140818453574006 │   0.127606830880617 │
│   46 │ 0.34883580804820313 │ 0.10080873927892049 │ 0.18534915881527764 │  0.2632869991823385 │  0.5128856370584397 │  0.5263736263736264 │ 0.12242922816356157 │
│   45 │   0.331509316928515 │ 0.10038079358283952 │ 0.18543829000102766 │  0.2601484757542253 │  0.5365226210206051 │  0.5628416006615586 │ 0.12760623578728186 │
│   44 │ 0.45682902279987747 │  0.1005586592178771 │ 0.18538831064851882 │  0.2639387662062402 │  0.5711759504862954 │  0.5874079771453686 │ 0.12731481481481483 │
│   43 │  0.3598442304505533 │ 0.10091277890466532 │ 0.18521658626906398 │  0.2555123319362685 │ 0.49162211189612376 │  0.5021630817839348 │ 0.12185373808664501 │
│   42 │ 0.33686924295727366 │ 0.10006105663901162 │  0.1849143393569585 │   0.263380043896674 │  0.5098987456551308 │  0.5249253596308695 │ 0.12733318430758916 │
│   41 │ 0.32223611961217413 │ 0.10023714641957805 │ 0.18480650835532103 │ 0.25624387171862917 │  0.5459544706527703 │  0.5553666490147441 │ 0.12211261151990513 │
│   40 │ 0.45096998643787956 │ 0.10069005247404476 │ 0.18435839849915667 │  0.2590711000890858 │  0.5803909908570578 │  0.5692682713220161 │ 0.12203644965324445 │
│   39 │ 0.34653150811528916 │ 0.10065802082667862 │ 0.18461700299805392 │ 0.25630170787169193 │  0.4889093431167923 │  0.4896583959802127 │ 0.12191751415461403 │
│   38 │  0.3248045486851457 │ 0.10021330230754315 │ 0.18424598168696205 │  0.2595891418392195 │  0.5103528402882475 │  0.5148721662117752 │ 0.10464438913308267 │
│   37 │ 0.31098186794389326 │ 0.10029201094713686 │ 0.18381609969686763 │  0.2529893838025289 │  0.5348103609140589 │  0.5377971049026032 │ 0.12168010628921717 │
│   36 │   0.451173471373526 │ 0.09906203283546548 │  0.1837534829816789 │    0.24808009790474 │  0.5604427058486787 │  0.5746021922181144 │  0.1266443821436274 │
│   35 │   0.339426800929512 │ 0.10053556328102979 │ 0.18328766679096614 │  0.2517795608320547 │ 0.47892558493115966 │ 0.48214690225393586 │ 0.12637377118309043 │
│   34 │  0.3092607492591793 │ 0.09949161913238132 │  0.1828207881542722 │ 0.25862545328958725 │  0.5003313652247259 │  0.5094707136393614 │ 0.12007237786235729 │
│   33 │ 0.29455700382680317 │  0.0989157879102926 │ 0.18235233317166644 │ 0.25669735277276134 │  0.5141070320332267 │  0.5389033160207751 │ 0.12617999455387133 │
│   32 │ 0.43225196149351736 │  0.0989127659288527 │ 0.18412586629107383 │ 0.25815473149218815 │  0.5647767833121088 │  0.5472131375687148 │ 0.10758024493716481 │
│   31 │  0.3278670273760618 │ 0.09906790945406124 │ 0.19155206286836934 │ 0.24957191397039705 │  0.4645459146893262 │  0.4658860409367509 │ 0.12424561850594816 │
│   30 │  0.3010065505671833 │ 0.09935070014863491 │ 0.19129908017025624 │  0.2610455119740855 │  0.4762753325017416 │ 0.49542952376142624 │ 0.12579938483326328 │
│   29 │   0.278335015344057 │ 0.09958330118064666 │  0.1910091551053602 │ 0.24702932925749543 │  0.5176035261468139 │    0.50591868244982 │  0.1251279129875199 │
│   28 │  0.4358721577357291 │ 0.09865660678046291 │   0.191049843702815 │  0.2542462533059066 │  0.5593879038134564 │  0.5557544191284414 │ 0.12438966967950621 │
│   27 │ 0.31130173965206964 │ 0.09896809939150528 │  0.1907146410213895 │  0.2527123045027274 │ 0.44419661574066316 │  0.4434306569343066 │ 0.12486334304041176 │
│   26 │ 0.28136544997783364 │ 0.09990845492252574 │   0.190414245168266 │ 0.25580448065173117 │ 0.47375145825231935 │  0.4726312094733148 │ 0.12452662062820229 │
│   25 │  0.2639035501972332 │ 0.09860321195877188 │  0.1900459906983166 │ 0.24861062777478987 │  0.5050349993859757 │  0.5073402417962004 │ 0.11978141510411273 │
│   24 │  0.4339182361047313 │ 0.09901845792950184 │   0.190783753570985 │ 0.25769738847678086 │   0.552198568374082 │  0.5470368835474514 │ 0.12425927376197564 │
│   23 │  0.2957099216363395 │ 0.09982872046401184 │  0.1907605365911647 │ 0.24814529991857415 │  0.4198013564095912 │ 0.41795785702413707 │ 0.12389050748695711 │
│   22 │  0.2656113017558801 │ 0.10000272190315469 │  0.1899745082730531 │  0.2550270804936008 │ 0.44349584363064487 │  0.4535514564727486 │ 0.11913174388595306 │
│   21 │ 0.24069342629957738 │ 0.09720739551281478 │ 0.18596068606667954 │ 0.24296182028538377 │  0.4867598510922245 │  0.4626230676705128 │ 0.12334142228437463 │
│   20 │  0.4198984253809048 │  0.0984243544646143 │  0.1890397387814519 │  0.2493103359372986 │    0.53469882765126 │  0.5361582531618203 │  0.1221409041046548 │
│   19 │  0.2737081371492672 │  0.0997943314132136 │ 0.18849285669198562 │  0.2423411908871701 │  0.4012997562956946 │  0.3956164383561644 │ 0.11815064534771721 │
│   18 │ 0.23660460200016264 │  0.0999512627735477 │  0.1879100229667806 │ 0.25256061195384416 │  0.4295114367663718 │  0.4307781565015593 │ 0.12085964605745952 │
│   17 │ 0.21536166258784728 │ 0.09974365506480273 │ 0.18725659638207168 │ 0.24018606268747555 │  0.4601113172541745 │ 0.46715975511787033 │ 0.11724329991481552 │
│   16 │ 0.40793434356428004 │ 0.09956740881851014 │ 0.18868988513356136 │  0.2489312673818038 │  0.5182481751824818 │  0.5151093117408907 │ 0.12398816798625502 │
│   15 │  0.2400431097829815 │ 0.09326079715776618 │ 0.18850288206885812 │  0.2372489781411054 │  0.3636786221069629 │ 0.36166123303002295 │ 0.11050706229041767 │
│   14 │ 0.21246353423127062 │ 0.09923873705229004 │ 0.18779757386732177 │   0.247596110641816 │  0.3930338412824065 │  0.3912639666769863 │ 0.12310001154423368 │
│   13 │ 0.18753328993312424 │ 0.09932528532278767 │ 0.18690707677803312 │ 0.23493975903614459 │   0.423755492883846 │  0.4309200653616568 │ 0.12251833666419812 │
│   12 │ 0.38018741633199465 │ 0.09875869594871096 │  0.1859360904040644 │ 0.24216401889222844 │ 0.48853464132434093 │  0.4858339091906834 │ 0.12183662227406761 │
│   11 │  0.2070158008368521 │ 0.09882286668868127 │ 0.18453916460237466 │ 0.22633179224457498 │  0.3127469506957568 │ 0.31063018996701175 │ 0.11690096774914513 │
│   10 │  0.1718681702946062 │ 0.09853041158935202 │   0.182508801185844 │  0.2370973536103461 │ 0.34083513170965646 │   0.333994153422264 │  0.1202016285381931 │
│    9 │ 0.14459253430384106 │ 0.09768251476159596 │ 0.18078026741559317 │ 0.21980973974763407 │  0.3791076337492071 │  0.3731579385563234 │ 0.11908472074140902 │
│    8 │  0.3366997675892669 │ 0.09732114717932557 │ 0.18169304886441845 │  0.2317048271838997 │ 0.44985350462023893 │ 0.43331341337534635 │ 0.11389189031334873 │
│    7 │ 0.15429361028772653 │ 0.09703246762497686 │  0.1755447022800314 │ 0.21102003642987252 │ 0.23713867420244464 │ 0.23020678246484702 │ 0.11602340763792579 │
│    6 │ 0.11948670352286027 │ 0.09589353860204183 │ 0.17638699745396413 │ 0.22270302532920508 │  0.2608581780086429 │   0.250240777186885 │ 0.11025707995970302 │
│    5 │ 0.09207672749027819 │ 0.09465297110492465 │ 0.17249150544345052 │  0.1907445748025458 │  0.2803843344383885 │  0.2676797508457284 │  0.1107056505775947 │
│    4 │ 0.23947210557888426 │ 0.09373668512995313 │ 0.16759212518929834 │ 0.20086632416641484 │ 0.33456128600133966 │  0.3067701529239991 │ 0.10629520492901152 │
│    3 │ 0.07941294196130755 │ 0.09380013211286214 │ 0.15964771817453963 │ 0.15321176109005227 │   0.154135532079361 │ 0.14774748073503263 │ 0.10048524059846342 │
│    2 │ 0.04808976756612343 │ 0.09225075179273652 │ 0.14652767581852885 │ 0.13959994404811862 │  0.1486446231754543 │  0.1407914227269521 │ 0.09302542570562165 │
└──────┴─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┘

ERROR: LoadError: On worker 2:
could not load library "/home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/libietest.so"
/home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/libietest.so: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type
iedot3 at /home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/loadsharedlibs.jl:307 [inlined]
##core#3936 at /home/lolo/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:371
##sample#3937 at /home/lolo/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:377
#_run#641 at /home/lolo/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:405
inner at ./essentials.jl:715
#invokelatest#1 at ./essentials.jl:716 [inlined]
#run_result#37 at /home/lolo/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:32 [inlined]
#run#40 at /home/lolo/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:94
#warmup#45 at /home/lolo/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:141 [inlined]
warmup at /home/lolo/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:141
macro expansion at /home/lolo/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:287 [inlined]
dot3_bench! at /home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/benchmarkflops.jl:276
#71 at /home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/benchmarkflops.jl:284
#104 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:294
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:79
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:294 [inlined]
#103 at ./task.jl:358
Stacktrace:
 [1] (::Base.var"#726#728")(::Task) at ./asyncmap.jl:178
 [2] foreach(::Base.var"#726#728", ::Array{Any,1}) at ./abstractarray.jl:1919
 [3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::Base.Iterators.Enumerate{StepRange{Int64,Int64}}) at ./asyncmap.jl:178
 [4] wrap_n_exec_twice(::Channel{Any}, ::Array{Any,1}, ::Distributed.var"#204#207"{WorkerPool}, ::Function, ::Base.Iterators.Enumerate{StepRange{Int64,Int64}}) at ./asyncmap.jl:154
 [5] async_usemap(::Distributed.var"#188#190"{Distributed.var"#188#189#191"{WorkerPool,var"#71#72"{SharedArray{Float64,2}}}}, ::Base.Iterators.Enumerate{StepRange{Int64,Int64}}; ntasks::Function, batch_size::Nothing) at ./asyncmap.jl:103
 [6] #asyncmap#710 at ./asyncmap.jl:81 [inlined]
 [7] pmap(::Function, ::WorkerPool, ::Base.Iterators.Enumerate{StepRange{Int64,Int64}}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Array{Any,1}, retry_check::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:126
 [8] pmap(::Function, ::WorkerPool, ::Base.Iterators.Enumerate{StepRange{Int64,Int64}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:101
 [9] pmap(::Function, ::Base.Iterators.Enumerate{StepRange{Int64,Int64}}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:156
 [10] pmap at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:156 [inlined]
 [11] benchmark_dot3(::StepRange{Int64,Int64}) at /home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/benchmarkflops.jl:284
 [12] top-level scope at /home/lolo/Projects/LoopVectorization.jl/benchmark/driver.jl:27
 [13] include(::String) at ./client.jl:439
 [14] top-level scope at none:0
in expression starting at /home/lolo/Projects/LoopVectorization.jl/benchmark/driver.jl:27

[chriselrod]

could not load library "/home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/libietest.so"

Wow, my eigen benchmark results are probably all wrong.
When I switched from icpc to clang for eigen, I copy and pasted the command to compile with g++, and then it seems I did not change the name of the output file.
Meaning the "clang" results are actually icpc -- because instead of overwriting the old icpc-compiled-shared library, it overwrote the g++ one.

I pushed the fix. You should delete "libetest.so", because this file is supposed to be compiled by g++, but the bug replaced the g++ version with clang. Then you should be able to run the benchmarks.

[LaurentPlagne]

Hi again,

I probably do something wrong, I did make a pull this morning but I get the same error

│    3 │  0.07940234791889009 │ 0.09419420051803654 │ 0.15970739000427167 │  0.1524309448578127 │ 0.15325101193831023 │ 0.14780589049219217 │  0.0993006993006993 │
│    2 │ 0.047991476791205606 │ 0.08692995854243946 │  0.1466030113845024 │  0.1466137799324225 │  0.1486778398510242 │  0.1408410951171324 │ 0.09299071958214802 │
└──────┴──────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┘

ERROR: LoadError: On worker 3:
could not load library "/home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/libietest.so"
/home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/libietest.so: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type

[chriselrod]

I did make a pull

could not load library "/home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/libietest.so"

Do you have LoopVectorization checked out for development? Or did you add LoopVectorization#master?

(EDIT:
Actually, that's the exact same path as last time, meaning it was running the exact same version as last time. Both were packages/LoopVectorization/utRiX)

It looked for the shared library in /home/lolo/.julia/packages/LoopVectorization/utRiX, meaning it is trying a version that is not checked out for development but installed via add.
This add-ed version -- which wouldn't have been updated with a pull -- was probably not the master branch.

You can look inside /home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/loadsharedlibs.jl and probably see the bug where it is doing this:

1. Check if "/home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/libetest.so" exists, if not, create it with g++.
2. Check if "/home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/libietest.so" exists, if not, replace "/home/lolo/.julia/packages/LoopVectorization/utRiX/benchmark/libetest.so" (notice the difference in presence/absence of is) with a version compiled with clang++.

See here for the bad code. It checked for LIBIEIGENTEST, but then the file it compiled was LIBEIGENTEST.

The master branch on the other hand correctly creates the file LIBIEIGENTEST.

[LaurentPlagne]

Thanx,
I have removed and reinstalled LoopVectorization.jl and it works now. The tests are not completed yet but the x' * A * y benchmark results are all green for the LoopVectorization column !

Concerning BTL.jl jhe memory consumption of BenchmarkTools is related to the size of the arguments. I have problems because I use very large arrays. As a workaround, I can simply make a switch between @belapsed based timing for small cases and @elapsed for large ones. I think that I may not use several workers in the later case because they will consume the same RAM bandwidth...

I will send you the JLD2 of the results.
In the database, this kind of results should be connected to the machine description (CPUID.jl ?), the package/julia version and maybe some benchmark parameters.

Also, one should advise that the benchmark to be carried during winter in order to make a useful use of the generated heat ;)

[chriselrod]

I issued a new release yesterday that should improve AVX2 performance, mostly in the matrix multiplication benchmarks.

As a workaround, I can simply make a switch between @belapsed based timing for small cases and @Elapsed for large ones.

I don't see an issue with BenchmarkTools. Am I missing one? If not, you should open one with that minimal example.

In the database, this kind of results should be connected to the machine description (CPUID.jl ?), the package/julia version and maybe some benchmark parameters.

CpuId.jl will only work on Intel and AMD, but that covers most.
In particular, I'd want to know about support for the following instruction sets:
AVX, FMA, AVX2, and AVX512F.

Also, one should advise that the benchmark to be carried during winter in order to make a useful use of the generated heat ;)

Ha.
BenchmarkTools.DEFAULT_PARAMETERS.seconds = 0.1 hopefully helps.

[LaurentPlagne]

Hi Chris,

I think that this issue corresponds to the problem
JuliaCI/BenchmarkTools.jl#127

BTW, this memory issue is also a problem here because the LoopVectorization benchmarks also exhausted my memory machine 'nearly 16GB at the end).
I have tried to add and remove procs during the benchmark but I failed because I don't know how to use @Everywhere inside a function...

I should be able to send you a JLD2 for my machine tomorrow.

[LaurentPlagne]

Hi again,

I have made a simple experiment that shows how to release the memory allocated (and not deallocated) by BenchmarkTools measurements. Doing so I learn a little more about Distributed.jl

I need two files. The main one that should be included by the user

using Distributed

const MEAS=Vector{Tuple{Int,Float64}}

nprocs_toadd() = (Sys.CPU_THREADS >> 1)-1

function start_workers()
    #Here we should have only one worker
    addprocs(nprocs_toadd())
    #Now we have nprocs_toadd() workers (the initial/main one is not counted...)
    @show nworkers()
    futures=[] #An array to store the workers's futures
    for wid in workers()
        @show wid
        #We put all environment operations (using,...) needed by each workers
        #in a file and command each worker to include this file
        # remotecall_wait(include,wid,"./BenchmarkSingle_memory.jl")
        push!(futures,remotecall(include,wid,"./BenchmarkSingle_memory.jl"))
    end
    # #We wait for all workers environment set up to finish
    for f in futures
        wait(f)
    end

    @show nworkers()
end

function stop_workers()
    rmprocs(workers())
    #Now we should have only one worker
    @show nworkers()
end

function benchmark_all(fs,ss)
    MEAS=Vector{Tuple{Int,Float64}}
    perfs = Vector{Tuple{String,MEAS}}()
    for f in fs
        #Add workers performing the benchmark in parallel for all sizes ss
        start_workers()
        @show f
        ts=pmap(s->(s,benchmark_single(f,s)),ss)
        push!(perfs,(string(f),ts))
        #Remove workers in order to release the BenchmarkTools memory
        stop_workers()
    end
    perfs
end

function go()
    sizes = [2^i for i in 1:26]
    functions = [sin,cos,exp,sin,cos,exp,sin,cos,exp,sin,cos,exp]
    perfs = benchmark_all(functions,sizes)
end

go()

And a second one (here) named BenchmarkSingle_memory.jl
that is included by the workers and used to set up the workers environment

using BenchmarkTools
BenchmarkTools.DEFAULT_PARAMETERS.seconds = 0.11
BenchmarkTools.DEFAULT_PARAMETERS.gctrial = false
function benchmark_single(f,s)
    x=rand(s)
    t=@belapsed $f.($x)
    return t
end

You can check that the memory usage does not increase during the loop on the functions.
You may adapt your benchmark with this strategy to allow machine with less than 16GB of RAM to run the tests.
Best

[chriselrod]

Thank you, I'll update the benchmarks to follow that approach soon.

[chriselrod]

The benchmarks now start and remove workers for each benchmark. That should stop the memory from growing over time.

[LaurentPlagne]

Great ! I think that you forgot to commit setup_worker.jl

[chriselrod]

Oops, just added it!

[LaurentPlagne]

Sorry if I missed something obvious but I get

[ Info: Precompiling Revise [295af30f-e4ad-537b-8983-00126c2a3abe]
julia> @time include("driver.jl")
[ Info: Precompiling BenchmarkTools [6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf]
[ Info: Precompiling Gadfly [c91e804a-d5a3-530f-b6f0-dfbca275c004]
logdet(LowerTriangular(A)) benchmark results:
┌──────┬────────────────────┬────────────────────┬────────────────────┬─────────────────────┬─────────────────────┬────────────────────┬─────────────────────┐
│ Size │  LoopVectorization │              Julia │              Clang │            GFortran │                 icc │              ifort │       LinearAlgebra │
├──────┼────────────────────┼────────────────────┼────────────────────┼─────────────────────┼─────────────────────┼────────────────────┼─────────────────────┤
│   23 │ 0.3071601474390714 │ 0.1050343186852252 │ 0.2002994494300037 │ 0.25986861503261033 │ 0.43386253518968915 │ 0.4300248657062051 │ 0.12488052496726988 │
└──────┴────────────────────┴────────────────────┴────────────────────┴─────────────────────┴─────────────────────┴────────────────────┴─────────────────────┘

x' * A * y benchmark results:
┌──────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┬───────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┐
│ Size │  LoopVectorization │              Julia │              Clang │           GFortran │               icc │              ifort │      g++ & Eigen-3 │  clang++ & Eigen-3 │      LinearAlgebra │
├──────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┼───────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┤
│   23 │ 30.042853028703085 │ 11.762042618218864 │ 11.754259001595006 │ 13.929561994995321 │ 13.28957162597881 │ 13.074034627778687 │ 13.996280087527353 │ 10.390732526662536 │ 10.997771922839062 │
└──────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┴───────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┘

ERROR: LoadError: UndefVarError: tests not defined
Stacktrace:

[chriselrod]

That's what I get for not rerunning all of them locally. I fixed that in the last commit.

[LaurentPlagne]

Thanks ! It seems to work now. I happen to have an issue with pango but it is not important and probably related with my local environment.

I think that it may be safer to save the results (JLD2) before plotting them.

I will sent you the results when it is done.

[LaurentPlagne]

BTL, the foreach(wait,remote_call(...),nworkers()) construct is super cool !