Julia has strftime as a built-in but not gmtime.
julia> strftime
strftime (generic function with 3 methods)
julia> gmtime
ERROR: gmtime not defined
What is the preferred Julia way to do the equivalent of gmtime? The idea is to turn seconds since the epoch into a time structure in the Z (+00:00) time zone. Here in Los Angeles, I see:
julia> strftime("%H:%M:%S", 0)
"16:00:00"
I would like to see "00:00:00". I can do it in Python:
>>> from time import strftime, gmtime
>>> strftime("%H:%M:%S", gmtime(0))
'00:00:00'
I tried to use ccall in Julia but it did not work
julia> ccall( (:gmtime, "libc"), TmStruct, (Int64,), 0)
TmStruct(1590498096,32767,16041550,1,-1924564896,32744,1,0,0,0,0,0,1590498144,32767)
julia> strftime("%H:%M:%S", ans)
"16041550:32767:1590498096"
What went wrong with my ccall? And better, is there just a nicer all-Julia way to get the effect of gmtime?
Looks like gmtime is on Julia's TODO list. Until it gets included, will something like this work for you?
julia> function gmtime(t::Real)
t = floor(t)
tm = TmStruct()
ccall(:gmtime_r, Ptr{TmStruct}, (Ptr{Int}, Ptr{TmStruct}), &t, &tm)
return tm
end
gmtime (generic function with 1 method)
julia> strftime("%H:%M:%S", gmtime(0))
"00:00:00"
Related
I'm trying to implement a convert for struct containing NTuple:
import Base: convert
abstract type AbstractMyType{N, T} end
struct MyType1{N, T} <: AbstractMyType{N, T}
data::NTuple{T, N}
end
struct MyType2{N, T} <: AbstractMyType{N, T}
data::NTuple{T, N}
end
foo(::Type{MyType2}, x::AbstractMyType{N, T}) where {N, T} = x
convert(::Type{MyType2}, x::AbstractMyType{N, T}) where {N, T} = MyType2{T}(x.data)
println(foo(MyType2, MyType1((1,2,3)))) # MyType1{Int64,3}((1, 2, 3))
println(convert(MyType2, MyType1((1,2,3)))) # MethodError
Defined functions foo and convert have the same signature. For some reason function foo returns normally while convert throws MethodError. Why Julia cannot find my convert method?
julia version 1.4.1
Julia is finding your convert method:
julia> println(convert(MyType2, MyType1((1,2,3)))) # MethodError
ERROR: MethodError: no method matching MyType2{3,T} where T(::Tuple{Int64,Int64,Int64})
Stacktrace:
[1] convert(::Type{MyType2}, ::MyType1{Int64,3}) at ./REPL[16]:1
[2] top-level scope at REPL[18]:1
That stack trace is saying that it's inside your convert function (in my case, I defined it on the first line of the 16th REPL prompt). The problem is that it cannot find a MyType2{T}(::Tuple) constructor.
Julia automatically creates a number of constructors for you when you don't use an inner constructor; in this case you can either call MyType(()) or MyType{T, N}(()), but Julia doesn't know what to do with only one type parameter passed (by default):
julia> MyType2((1,2,3))
MyType2{Int64,3}((1, 2, 3))
julia> MyType2{Int, 3}((1,2,3))
MyType2{Int64,3}((1, 2, 3))
julia> MyType2{Int}((1,2,3))
ERROR: MethodError: no method matching MyType2{Int64,T} where T(::Tuple{Int64,Int64,Int64})
Stacktrace:
[1] top-level scope at REPL[7]:1
[2] eval(::Module, ::Any) at ./boot.jl:331
[3] eval_user_input(::Any, ::REPL.REPLBackend) at /Users/mbauman/Julia/release-1.4/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
[4] run_backend(::REPL.REPLBackend) at /Users/mbauman/.julia/packages/Revise/AMRie/src/Revise.jl:1023
[5] top-level scope at none:0
So the fix is either to define that method yourself, or change the body of your convert method to call MyType{T,N} explicitly.
Just define the method
convert(::Type{MyType2}, x::AbstractMyType{N, T}) where {N, T} = MyType2(x.data)
Testing:
julia> convert(MyType2, MyType1((1,2,3)))
MyType2{Int64,3}((1, 2, 3))
I have looked up several tutorials on how to generate random numbers with lua, each said to use math.random(), so I did. however, every time I use it I get the same number every time, I have tried rewriting the code, and I always get the lowest possible number. I even included a random seed based on the OS time. code below.
require "math"
math.randomseed(os.time())
num = math.random(0,10)
print(num)
I'm using the random function like this:
math.randomseed(os.time())
num = math.random() and math.random() and math.random() and math.random(0, 10)
This is working fine. An other option would be to improve the built-in random function, described here.
This might help! I had to use these functions to write a class that generates Nano IDs. I basically used the milliseconds from the os.clock() function and used that for math.randomseed().
NanoId = {
validCharacters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-",
generate = function (size, validChars)
local response = ""
local ms = string.match(tostring(os.clock()), "%d%.(%d+)")
local temp = math.randomseed(ms)
if (size > 0 and string.len(validChars) > 0) then
for i = 1, size do
local num = math.random(string.len(validChars))
response = response..string.sub(validChars, num, num)
end
end
return response
end
}
function NanoId:Generate()
return self.generate(21, self.validCharacters)
end
-- Runtime Testing
for i = 1, 10 do
print(NanoId:Generate())
end
--[[
Output:
>>> p2r2-WqwvzvoIljKa6qDH
>>> pMoxTET2BrIjYUVXNMDNH
>>> w-nN7J0RVDdN6-R9iv4i-
>>> cfRMzXB4jZmc3quWEkAxj
>>> aFeYCA2kgOx-s4UN02s0s
>>> xegA--_EjEmcDk3Q1zh7K
>>> 6dkVRaNpW4cMwzCPDL3zt
>>> R2Fct5Up5OwnHeExDnqZI
>>> JwnlLZcp8kml-MHUEFAgm
>>> xPr5dULuv48UMaSTzdW5J
]]
I got some time intervals using the tic() and toc() functions which are in seconds.
Let's suppose I have a time interval dt=3600.125 seconds. How can I print it in "H:M:S.s" format using Julia?
If you convert it to Dates formats then you can use this method.
julia> t1 = now()
2017-11-10T10:00:51.974
# Wait
julia> t2 = now()
2017-11-10T10:10:07.895
julia> x = Dates.canonicalize(Dates.CompoundPeriod(t2-t1))
9 minutes, 15 seconds, 921 milliseconds
julia> x.periods
3-element Array{Base.Dates.Period,1}:
9 minutes
15 seconds
921 milliseconds
julia> x.periods[2]
15 seconds
julia> x.periods[2].value
18
You can make your own function. The main function to know is divrem which gives you the divisor as well as the remainder in one convenient function call.
dt=3600.125
function hmss(dt)
(h,r) = divrem(dt,60*60)
(m,r) = divrem(r, 60)
#(s,r) = divrem(r, 60)
string(Int(h),":",Int(m),":",r)
end
hmss(dt)
hmss(3452.98)
Have a look at the dates section in the manual.
Not sure if this is the most efficient way, but this works for instance:
julia> Dates.format(DateTime("2017-10-01T01:02:03"), "H:M:S.s")
"1:2:3.0"
""" working properly only if 0<= sec <=86400 otherwise truncating """
function fmtsec(sec, fmt::Dates.DateFormat)
# nanos = Dates.Nanosecond(sec * 1e9) # we could get InexactError here!
nanos = Dates.Nanosecond(trunc(Int, sec * 1e9))
mytime = Dates.Time(nanos)
Dates.format(mytime, fmt)
end
fmt = dateformat"H:M:S.s" # Creating a DateFormat object is expensive. (see doc)
fmtsec(3600.125, fmt) # "1:0:0.125"
EDIT: Without truncating nanoseconds we could get error ->
julia> tic();sleep(1);old_fmtsec(toc(), dateformat"S.s")
elapsed time: 1.002896514 seconds
ERROR: InexactError()
Stacktrace:
[1] convert(::Type{Int64}, ::Float64) at ./float.jl:679
[2] fmtsec(::Float64, ::DateFormat{Symbol("S.s"),Tuple{Base.Dates.DatePart{'S'},Base.Dates.Delim{Char,1},Base.Dates.DatePart{'s'}}}) at ./REPL[47]:2
julia> tic();sleep(1);old_fmtsec(toc(), dateformat"S.s")
elapsed time: 1.002857122 seconds
"1.002"
tic & toc are deprecated. toc could bring rounding error because it internally use time_ns but nanoseconds convert to seconds dividing by 1e9.
This date from toc() and time() in Julia uses UNIX timestamp, i.e., seconds elapsed since January 1, 1970
The same occurs with the function ctime(f) that returns timestamp from a file path f.
The conversion is very straightforward.
tic()
....
dif = toc()
# My local is Brazil, one should replace for its local format
Dt = DateTime(LibC.strftime(dif),"d/m/y H:M:S")
LibC does not have to be imported explicitly
I had a go now, but doing the carry over is a bit messy sorry:
using Dates
using Printf
function formatDuration(duration::Period, types::Vector{DataType}=[Hour, Minute, Second]; addUnit::Bool=false)
periods = canonicalize(Dates.CompoundPeriod(duration)).periods
missingTypes = Vector{DataType}(types)
newPeriods = vcat(periods, map(t->t(0), missingTypes))
sort!(newPeriods; rev=true)
newPeriods1 = Vector{Period}()
carryover = nothing
for (i, p) in enumerate(newPeriods)
typeofp = typeof(p)
if !isnothing(carryover)
newPeriod = convert(typeofp, floor(Millisecond(p) + Millisecond(carryover), typeofp(1)))
carryover = nothing
else
newPeriod = p
end
if (typeof(newPeriod) in types)
push!(newPeriods1, newPeriod)
filter!(e->e≠typeofp,missingTypes)
else
carryover = newPeriod
end
end
m = map(types) do t
f = findfirst(x -> typeof(x) == t, newPeriods1);
r = isnothing(f) ? 0 : newPeriods1[f].value;
#sprintf("%02d",r) * (addUnit ? lowercase("$t"[1]) : "")
end
join(m, ":")
end
Usage examples:
#show formatDuration(Millisecond(3600.125*1000))
# returns "01:00:00"
#show formatDuration(Day(2), [Hour, Minute, Second])
# returns "48:00:00"
#show formatDuration(Second(100000000), [Hour, Minute, Second])
# returns "27777:46:40"
#show formatDuration(Second(100000001), [Day, Hour, Minute, Second, Millisecond])
# returns "1157:09:46:41:00"
#show formatDuration(Millisecond(10000000001),
[Week, Day, Hour, Minute, Second, Millisecond]; addUnit=true)
#returns "16w:03d:17h:46m:40s:01m"
How do you move data from one processor to another in julia?
Say I have an array
a = [1:10]
Or some other data structure. What is the proper way to put it on all other available processors so that it will be available on those processors as the same variable name?
I didn't know how to do this at first, so I spent some time figuring it out.
Here are some functions I wrote to pass objects:
sendto
Send an arbitrary number of variables to specified processes.
New variables are created in the Main module on specified processes. The
name will be the key of the keyword argument and the value will be the
associated value.
function sendto(p::Int; args...)
for (nm, val) in args
#spawnat(p, eval(Main, Expr(:(=), nm, val)))
end
end
function sendto(ps::Vector{Int}; args...)
for p in ps
sendto(p; args...)
end
end
Examples
# creates an integer x and Matrix y on processes 1 and 2
sendto([1, 2], x=100, y=rand(2, 3))
# create a variable here, then send it everywhere else
z = randn(10, 10); sendto(workers(), z=z)
getfrom
Retrieve an object defined in an arbitrary module on an arbitrary
process. Defaults to the Main module.
The name of the object to be retrieved should be a symbol.
getfrom(p::Int, nm::Symbol; mod=Main) = fetch(#spawnat(p, getfield(mod, nm)))
Examples
# get an object from named x from Main module on process 2. Name it x
x = getfrom(2, :x)
passobj
Pass an arbitrary number of objects from one process to arbitrary
processes. The variable must be defined in the from_mod module of the
src process and will be copied under the same name to the to_mod
module on each target process.
function passobj(src::Int, target::Vector{Int}, nm::Symbol;
from_mod=Main, to_mod=Main)
r = RemoteRef(src)
#spawnat(src, put!(r, getfield(from_mod, nm)))
for to in target
#spawnat(to, eval(to_mod, Expr(:(=), nm, fetch(r))))
end
nothing
end
function passobj(src::Int, target::Int, nm::Symbol; from_mod=Main, to_mod=Main)
passobj(src, [target], nm; from_mod=from_mod, to_mod=to_mod)
end
function passobj(src::Int, target, nms::Vector{Symbol};
from_mod=Main, to_mod=Main)
for nm in nms
passobj(src, target, nm; from_mod=from_mod, to_mod=to_mod)
end
end
Examples
# pass variable named x from process 2 to all other processes
passobj(2, filter(x->x!=2, procs()), :x)
# pass variables t, u, v from process 3 to process 1
passobj(3, 1, [:t, :u, :v])
# Pass a variable from the `Foo` module on process 1 to Main on workers
passobj(1, workers(), [:foo]; from_mod=Foo)
use #eval #everywhere... and escape the local variable. like this:
julia> a=collect(1:3)
3-element Array{Int64,1}:
1
2
3
julia> addprocs(1)
1-element Array{Int64,1}:
2
julia> #eval #everywhere a=$a
julia> #fetchfrom 2 a
3-element Array{Int64,1}:
1
2
3
Just so everyone here knows, I put these ideas together into a package ParallelDataTransfer.jl for this. So you just need to do
using ParallelDataTransfer
(after installing) in order to use the functions mentioned in the answers here. Why? These functions are pretty useful! I added some testing, some new macros, and updated them a bit (they pass on v0.5, fail on v0.4.x). Feel free to put in pull requests to edit these and add more.
To supplement #spencerlyon2 's answer here are some macros:
function sendtosimple(p::Int, nm, val)
ref = #spawnat(p, eval(Main, Expr(:(=), nm, val)))
end
macro sendto(p, nm, val)
return :( sendtosimple($p, $nm, $val) )
end
macro broadcast(nm, val)
quote
#sync for p in workers()
#async sendtosimple(p, $nm, $val)
end
end
end
The #spawnat macro binds a value to a symbol on a particular process
julia> #sendto 2 :bip pi/3
RemoteRef{Channel{Any}}(9,1,5340)
julia> #fetchfrom 2 bip
1.0471975511965976
The #broadcast macro binds a value to a symbol in all processes except 1 (as I found doing so made future expressions using the name copy the version from process 1)
julia> #broadcast :bozo 5
julia> #fetchfrom 2 bozo
5
julia> bozo
ERROR: UndefVarError: bozo not defined
julia> bozo = 3 #these three lines are why I exclude pid 1
3
julia> #fetchfrom 7 bozo
3
julia> #fetchfrom 7 Main.bozo
5
Coming from a R background, I was exploring the parallel possibilities by Julia. My objective is to replicate the performance of mcapply (parallel apply)
** The problem: **
I iterate a function on the rows of a data-frame that looks like that:
for i in 1:_nrow # of my DataFrame
lat1 = Raw_Data[i,"lat1"]
lat2 = Raw_Data[i,"lat2"]
lon1 = Raw_Data[i,"long1"]
lon2 = Raw_Data[i,"long2"]
iata1 = Raw_Data[i,"iata1"]
iata2 = Raw_Data[i,"iata2"]
a[i] = [(iata1::String,iata2::String, trunc(i,2), get_intermediary_points(lat1,lon1,lat2,lon2,j) ) for j in 0:.1:1]
end
Now, as a step toward parallelization, I can also create an anonymous function that does quite similar work, running calculation on each chunk of my dataframe:
Raw_Data["selector"] = rand(1:nproc,_nrow) # Define how I split my dataframe. 1 chunck per proc
B = by(Raw_Data,:selector,intermediary_points)
Is there a way to speed up calculations with a parallelized "by"? Otherwise, please suggest good alternative.
Thanks!
Note: This is how my dataframe Raw_Data looks like
6x7 DataFrame:
iata1 lat1 long1 iata2 lat2 long2
[1,] 1 "ELH" 0.444616 -1.3384 "FLL" 0.455079 -1.39891
[2,] 2 "BCN" 0.720765 0.0362729 "UFA" 0.955274 0.976218
[3,] 3 "ACE" 0.505053 -0.237426 "VCE" 0.794214 0.215582
[4,] 4 "PVG" 0.543669 2.12552 "LZH" 0.425277 1.91171
[5,] 5 "CDG" 0.855379 0.0444809 "VLC" 0.689233 -0.00835298
[6,] 6 "HLD" 0.858699 2.08915 "CGQ" 0.765906 2.18718
I figure out what happened. I didn't made all the inputs available to all processors.
Basically, if you are running into the same problem:
All functions should have #everywhere in front of them
All packages should also be declared as #everywhere using DataFrames
All parameters should also be declared with #everywhere in front
of it
Now, that's a lot of work. You can follow http://julia.readthedocs.org/en/latest/manual/parallel-computing/ to use stand-alone packages that would simplify a bit the process.
Cheers.