I am having big trouble setting up a code for parallelization. When I try to “declare” the packages that I am using in the workers by running the following code:
#--------------------------------#
# Initialization #
#--------------------------------#
# Packages
#using Distributed
using QuantEcon, Optim, Distributions, DelimitedFiles, ExcelReaders
using ProgressMeter, BenchmarkTools, DataFrames, Combinatorics
using LinearAlgebra, Statistics, Random, StatsBase
using BlackBoxOptim, Distributed
# Number of cores/workers
addprocs(2)
#everywhere begin
using Pkg; Pkg.activate(".") # required
using QuantEcon, Optim, Distributions, DelimitedFiles, ExcelReaders
using ProgressMeter, BenchmarkTools, DataFrames, Combinatorics
using LinearAlgebra, Statistics, Random, StatsBase
using BlackBoxOptim, Distributed
end
# Set Directory
cd("XXX\\julia_codes")
I get the following error:
<strong>On worker 2:</strong>
<strong>ArgumentError: Package QuantEcon not found in current path:
- Run `import Pkg; Pkg.add("QuantEcon")` to install the QuantEcon package.
require at .\loading.jl:823
top-level scope at XXX\julia_codes\4-Simulations.jl:26
eval at .\boot.jl:328
#116 at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:276
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:56
run_work_thunk at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\process_messages.jl:65
#102 at .\task.jl:259
#remotecall_wait#154(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:421
remotecall_wait(::Function, ::Distributed.Worker, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:412
#remotecall_wait#157(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433
remotecall_wait(::Function, ::Int64, ::Module, ::Vararg{Any,N} where N) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Distributed\src\remotecall.jl:433
(::getfield(Distributed, Symbol("##161#163")){Module,Expr})() at .\task.jl:259
...and 7 more exception(s).</strong>
in top-level scope at [stdlib\v1.1\Distributed\src\macros.jl:183](#)
in remotecall_eval at [stdlib\v1.1\Distributed\src\macros.jl:199](#)
in macro expansion at [base\task.jl:245](#)
in sync_end at [base\task.jl:226](#)
The usual solutions:
https://discourse.julialang.org/t/packages-and-workers/14072/9
and
https://discourse.julialang.org/t/understanding-everywhere-and-environments/27438
Do not seem to help either.
I had the same problem, but the suggestions in the comments - removing Pkg.activate(".") solved it!
Generally the following works for me:
using Distributed
addprocs(12)
#everywhere begin
...all using statements...
end
I'm running Julia version 1.0.5
Related
I'm new to Julia and, after a suggestion, I started using nightly build 1.6.0-DEV.1371 due to the high loading time of packages in Julia 1.5.2.
So I tried to change the default directory of DEPOT_PATH and copied all files from ~/.julia to /opt/julia (owned by my user). Now, if I start Julia using the default directory and run the code below, it takes about 4 seconds. However, when I start julia in the new directory (JULIA_DEPOT_PATH=/opt/julia julia), the same code takes incredible 83 seconds. The same happens with Julia 1.5.2 (17s in the default directory, 200s in the new directory).
This is the code I'm using to measure the time.
t = time(); using Plots; time() - t
Is there some explanation about this strange (and annoying) behavior?
My Platform Info:
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i3-6100U CPU # 2.30GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
And I'm using a SSD.
There is a good chance that now you are having files from your previous Julia installation with the new Julia's JULIA_DEPOT_PATH. The normal approach is to set JULIA_DEPOT_PATH to an empty folder and then install them there.
Regarding startup times there are the following things you can do:
Precompile package (this is also happening when you use a new version of the package for the first time)
using Pkg
pkg"precompile"
Once Plots is precompiled it still takes around 10s to load each time.
If the above time is unacceptable build-in your Plots into Julia system image.
using PackageCompiler
create_sysimage(:Plots, sysimage_path="sys_plots.so", precompile_execution_file="precompile_plots.jl")
For the precompile_plots.jl you could use commands that you use regularly or perhaps test sets from Plots.jl.
Once done you will be starting Julia with the following comand:
julia --sysimage sys_plots.so
Building the system image takes long, however once done, loading Plots.jl will be a matter of milliseconds.
More information can be found here:
package compiler tutorial https://julialang.github.io/PackageCompiler.jl/dev/examples/plots/
excellent tutorial video https://live.juliacon.org/talk/Z8TE39
My Julia version is 0.6.4. I am facing problem of installing Julia. I had it installing for so long, but all of a sudden julia stopped starting in REPL. Therefore, I deleted the julia and Atom, an trying to reinstall it eversince. Please give me suggestions. Below is the output when I start Julia in REPL:
Hold on tight while we're installing some packages for you.
This should only take a few seconds...
ERROR:LoadError: unknown package Atom
macro expansion at .\pkg\entry.jl:53 [inlined]
(::Base.Pkg.Entry.##1#3{String,Base.Pkg.Types.VersionSet})() at .\task.jl:335
Stacktrace:
[1]
sync_end()
at .\task.jl:287
[2] macro expansion at .\task.jl:303 [inlined]
[3] add(::String, ::Base.Pkg.Types.VersionSet) at .\pkg\entry.jl:51
[4] (::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}})() at .\pkg\dir.jl:36
[5] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}}, ::String) at .\file.jl:59
[6] #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{String,N} where N) at .\pkg\dir.jl:36
[7] add(::String) at .\pkg\pkg.jl:117
[8] include_from_node1(::String) at .\loading.jl:576
[9] include(::String) at .\sysimg.jl:14
[10] process_options(::Base.JLOptions) at .\client.jl:305
[11] _start() at .\client.jl:371
while loading C:\Users\Nitin\.atom\packages\julia-client\script\boot_sync.jl, in expression starting on line 22
Julia has stopped: 1
Have you tried installing Atom directly on your computer? The error lines seems to suggest you tried installing Atom through the Julia package manager Pkg, which would be odd as Atom is a general text editor and not a package ment for usage in Julia.
Also, on related note, any particular reason for using Julia 0.6 instead of the now standard 1.0 branches?
I have a problem with installing the "Pkg" package in Julia.
The following is the error I get after using Pkg.add("Pkg"):
unknown package Pkg
macro expansion at ./pkg/entry.jl:53 [inlined]
(::Base.Pkg.Entry.##1#3{String,Base.Pkg.Types.VersionSet})() at ./task.jl:335
Stacktrace:
[1] sync_end() at ./task.jl:287
[2] macro expansion at ./task.jl:303 [inlined]
[3] add(::String, ::Base.Pkg.Types.VersionSet) at ./pkg/entry.jl:51
[4] (::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}})() at ./pkg/dir.jl:36
[5] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}}, ::String) at ./file.jl:70
[6] #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{String,N} where N) at ./pkg/dir.jl:36
[7] add(::String) at ./pkg/pkg.jl:117
[8] include_string(::String, ::String) at ./loading.jl:522
Also, it would be great if you tell me how I can find the Package directory of Julia in my mac. I am using Jupyter.
From Julia v0. 7 on, Pkg is a standard library (a package that ships with julia) and doesn't need to be installed. Just do using Pkg and you are good to go.
To your second question, by default you can find everything Julia related in your home folder under ~/.julia.
I'm trying to run those Flink Benchmarks:
https://github.com/dataArtisans/flink-benchmarks
I've generated the jar file using maven with that command:
mvn clean package -Pbuild-jar
Then I'm trying to run the benchmark on a Flink Cluster with that command:
./bin/flink run -c org.apache.flink.benchmark.WindowBenchmarks ~/flinkBenchmarks/target/flink-hackathon-benchmarks-0.1.jar
I've used the -c option to add to the classpath the Main of the benchmark (WindowBenchmarks) I want to run.
Finally, I get that error:
# JMH version: 1.19
# VM version: JDK 1.8.0_151, VM 25.151-b12
# VM invoker: /usr/lib/jvm/java-8-oracle/jre/bin/java
# VM options: -Dlog.file=/home/user/flink-1.3.2/flink-dist/target/flink-1.3.2-bin/flink-1.3.2/log/flink-user-client-mypc.log -Dlog4j.configuration=file:/home/user/flink-1.3.2/flink-dist/target/flink-1.3.2-bin/flink-1.3.2/conf/log4j-cli.properties -Dlogback.configurationFile=file:/home/user/flink-1.3.2/flink-dist/target/flink-1.3.2-bin/flink-1.3.2/conf/logback.xml -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
# Warmup: 10 iterations, 1 s each
# Measurement: 10 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.flink.benchmark.WindowBenchmarks.sessionWindow
# Run progress: 0.00% complete, ETA 00:04:00
# Fork: 1 of 3
Error: Could not find or load main class org.openjdk.jmh.runner.ForkedMain
<forked VM failed with exit code 1>
<stdout last='20 lines'>
</stdout>
<stderr last='20 lines'>
Error: Could not find or load main class org.openjdk.jmh.runner.ForkedMain
</stderr>
# Run complete. Total time: 00:00:00
Benchmark Mode Cnt Score Error Units
The program didn't contain a Flink job. Perhaps you forgot to call execute() on the execution environment.
I don't have any previous experience with Flink and Maven so I find out what is missing. My first thought was that it's a missing dependencies error, but they look fine. Any suggestions?
Thank you in advance!
flink-benchmarks is a repository that contains sets of micro benchmarks designed to run on single machine, not on the cluster. The main functions defined in the various classes (test cases) are 'JMH' runners, not Flink programs. As such you can either execute whole benchmark suite (which takes ~1hour):
mvn -Dflink.version=1.5.0 clean install exec:exec
or if you want to execute just one benchmark, the best approach is to execute selected main function manually. For example from your IDE (don't forget about selecting flink.version, default value for the property is defined in pom.xml).
There is also a possibility to execute single benchmark from console, but I haven't tried it for very long time.
I have installed Julia a while ago and I realized that it used by default a pre-compiled binary. After reading about it, I realized that I could compile it natively using the PackageCompiler module. Indeed, upon loading the module, I get this message:
julia> using PackageCompiler
WARNING: Your Julia system image is not compiled natively for this CPU architecture.
Please run `PackageCompiler.force_native_image!()` for optimal Julia performance
So I proceed with using PackageCompiler.force_native_image!(). However, this gives me this error after some compiling:
julia> PackageCompiler.force_native_image!()
INFO: Copying system image: /home/user/.julia/v0.6/PackageCompiler/sysimg/backup/native/sys.o to /usr/lib/julia/sys.o
ERROR: open: permission denied (EACCES)
Stacktrace:
[1] uv_error at ./libuv.jl:68 [inlined]
[2] open(::String, ::UInt16, ::UInt16) at ./filesystem.jl:81
[3] sendfile(::String, ::String) at ./file.jl:523
[4] #cp#10(::Bool, ::Bool, ::Function, ::String, ::String) at ./file.jl:227
[5] (::Base.Filesystem.#kw##cp)(::Array{Any,1}, ::Base.Filesystem.#cp, ::String, ::String) at ./<missing>:0
[6] copy_system_image(::String, ::String, ::Bool) at /home/user/.julia/v0.6/PackageCompiler/src/PackageCompiler.jl:49
[7] force_native_image!() at /home/user/.julia/v0.6/PackageCompiler/src/api.jl:116
[8] macro expansion at ./REPL.jl:97 [inlined]
[9] (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:73
So basically it's trying to copy something to a root-access-only directory: /usr/lib/.
Coming from Python, I never had issues like that, and when I did it meant I was doing something wrong. So I'm weary of running Julia with sudo and doing the same steps (which I think will solve this issue but possibly create other permission issues down the road).
My question is: what is the right way to do this? Is the right way really running Julia as root?! (The docs weren't helpful at all with this.)