Related
I need to parse all a file into a better format to produce an outcome with columns delimited by a comma, thinking of being able to export the content in CSV file.
This is an example of my input;
. D 0 Mon Dec 10 11:07:46 2018
.. D 0 Mon Feb 19 11:38:06 2018
RJ9-5 D 0 Fri Nov 30 10:34:24 2018
WorkingOnClass D 0 Wed Feb 28 09:37:52 2018
ML-Test001 D 0 Fri Dec 7 16:38:56 2018
TestML4Testing D 0 Wed Aug 22 08:58:42 2018
ML-NewDataSE SetCases1.xlsx A 1415577 Wed Aug 29 14:00:16 2018
DR0001-Dum01 D 0 Thu Aug 16 08:24:25 2018
DR0002-Dum02 D 0 Thu Aug 16 09:04:50 2018
Readme File for Documentation And Data Description.docx A 16136 Wed Aug 29 14:00:24 2018
ML Database Prototype D 0 Thu Dec 6 15:11:11 2018
OneNote D 0 Mon Dec 3 09:39:20 2018
Data A 0 Mon Dec 10 11:07:46 2018
\RJ9-5
. D 0 Fri Nov 30 10:34:24 2018
.. D 0 Mon Dec 10 11:07:46 2018
KLR0151_Set023_Files_RJ9_05.xlsx A 182462 Wed Apr 4 02:48:55 2018
KLR0152_Set023_Files_RJ9_05.xlsx A 525309 Wed Apr 4 02:53:57 2018
\ML-Test001
. D 0 Wed Feb 28 09:37:52 2018
.. D 0 Mon Dec 10 11:07:46 2018
WT_Conforming_Format1_1.docx A 500914 Mon Feb 26 08:50:55 2018
Conforming_Format_1_1.xlsx A 130647 Mon Feb 26 08:52:33 2018
DR0135_Dum01_text.xls A 974848 Mon Feb 12 08:11:11 2018
DR0139_Dum02_body.xls A 1061888 Tue Jun 19 13:43:54 2018
DataSet_File_mod0874953.xlsx A 149835 Mon Feb 26 14:17:02 2018
File Path For Dataset-2018.07.11.xlsx A 34661 Mon Feb 12 09:27:17
This is script right here can make the job:
#!/bin/bash
awk -v OFS=, '
BEGIN { print "PATH, FILENAME, SIZE, TIMESTAMP" }
/[\\]/ { path=$0 }
$2 ~ /A/ {print path"\\"$1,$3,$4 " " $5 " " $6 " " $7 " "$8 }
' "$#"
But is ignoring the names with spaces on it, so I need to validate them with something like:
awk -v FS="\t" '{print $1}'
But I could't integrate into the shell script, because the way the shell script is working, so I was thinking on make AWK to start reading by the end, since the end is always the same, and leave the rest.
The output should something like this:
/RJ9-5/KLR0151_Set023_Files_RJ9_05.xlsx,182462,Wed Apr 4 02:48:55 2018
/RJ9-5/KLR0152_Set023_Files_RJ9_05.xlsx,25309,Wed Apr 4 02:53:57 2018
/ML-Test001/WT_Conforming_Format1_1.docx,500914,Mon Feb 26 08:50:55 2018
/ML-Test001/Format_1_1.xlsx,130647,Mon Feb 26 08:52:33 2018
/ML-Test001/DR0135_Dum01_text.xls,974848,Mon Feb 12 08:11:11 2018
/ML-Test001/DR0139_Dum02_body.xls,1061888,Tue Jun 19 13:43:54 2018
/ML-Test001/DataSet_File_mod0874953.xlsx,149835,Mon Feb 26 14:17:02 2018
/ML-Test001/File Path For Dataset-2018.07.11.xlsx,34661,Mon Feb 12 09:27:17 2018
With GNU awk for the 3rd arg to match() (and far less importantly \s shorthand for [[:space:]]):
$ cat tst.awk
BEGIN { OFS="," }
{ gsub(/^\s+|\s+$/,"") }
sub(/^\\/,"/") { path = $0; next }
path == "" { next }
match($0,/^(.*[^ ]) +A +([^ ]+) +(.*)/,a) { print path "/" a[1], a[2], a[3] }
$ awk -f tst.awk file
/RJ9-5/KLR0151_Set023_Files_RJ9_05.xlsx,182462,Wed Apr 4 02:48:55 2018
/RJ9-5/KLR0152_Set023_Files_RJ9_05.xlsx,525309,Wed Apr 4 02:53:57 2018
/ML-Test001/WT_Conforming_Format1_1.docx,500914,Mon Feb 26 08:50:55 2018
/ML-Test001/Conforming_Format_1_1.xlsx,130647,Mon Feb 26 08:52:33 2018
/ML-Test001/DR0135_Dum01_text.xls,974848,Mon Feb 12 08:11:11 2018
/ML-Test001/DR0139_Dum02_body.xls,1061888,Tue Jun 19 13:43:54 2018
/ML-Test001/DataSet_File_mod0874953.xlsx,149835,Mon Feb 26 14:17:02 2018
/ML-Test001/File Path For Dataset-2018.07.11.xlsx,34661,Mon Feb 12 09:27:17
Try this Perl solution:
$ perl -lane ' if(/^\s*$/) { $x=0;$y=0} if(/^\\/) {$x=1 ;($a=$_)=~s/\s*$//g;$a=~s/\\/\//g; } $y++ if $x==1 ; if($y>3) { s/^\s*//g; $_=~s/(.+?)\s+\S+\s+((\d+)\s+.+)/$1 $2/g;print "$a/$_" } ' essparaq.txt
/RJ9-5/KLR0151_Set023_Files_RJ9_05.xlsx 182462 Wed Apr 4 02:48:55 2018
/RJ9-5/KLR0152_Set023_Files_RJ9_05.xlsx 525309 Wed Apr 4 02:53:57 2018
/ML-Test001/WT_Conforming_Format1_1.docx 500914 Mon Feb 26 08:50:55 2018
/ML-Test001/Conforming_Format_1_1.xlsx 130647 Mon Feb 26 08:52:33 2018
/ML-Test001/DR0135_Dum01_text.xls 974848 Mon Feb 12 08:11:11 2018
/ML-Test001/DR0139_Dum02_body.xls 1061888 Tue Jun 19 13:43:54 2018
/ML-Test001/DataSet_File_mod0874953.xlsx 149835 Mon Feb 26 14:17:02 2018
/ML-Test001/File Path For Dataset-2018.07.11.xlsx 34661 Mon Feb 12 09:27:17
$ cat essparaq.txt
. D 0 Mon Dec 10 11:07:46 2018
.. D 0 Mon Feb 19 11:38:06 2018
RJ9-5 D 0 Fri Nov 30 10:34:24 2018
WorkingOnClass D 0 Wed Feb 28 09:37:52 2018
ML-Test001 D 0 Fri Dec 7 16:38:56 2018
TestML4Testing D 0 Wed Aug 22 08:58:42 2018
ML-NewDataSE SetCases1.xlsx A 1415577 Wed Aug 29 14:00:16 2018
DR0001-Dum01 D 0 Thu Aug 16 08:24:25 2018
DR0002-Dum02 D 0 Thu Aug 16 09:04:50 2018
Readme File for Documentation And Data Description.docx A 16136 Wed Aug 29 14 :00:24 2018
ML Database Prototype D 0 Thu Dec 6 15:11:11 2018
OneNote D 0 Mon Dec 3 09:39:20 2018
Data A 0 Mon Dec 10 11:07:46 2018
\RJ9-5
. D 0 Fri Nov 30 10:34:24 2018
.. D 0 Mon Dec 10 11:07:46 2018
KLR0151_Set023_Files_RJ9_05.xlsx A 182462 Wed Apr 4 02:48:55 2018
KLR0152_Set023_Files_RJ9_05.xlsx A 525309 Wed Apr 4 02:53:57 2018
\ML-Test001
. D 0 Wed Feb 28 09:37:52 2018
.. D 0 Mon Dec 10 11:07:46 2018
WT_Conforming_Format1_1.docx A 500914 Mon Feb 26 08:50:55 2018
Conforming_Format_1_1.xlsx A 130647 Mon Feb 26 08:52:33 2018
DR0135_Dum01_text.xls A 974848 Mon Feb 12 08:11:11 2018
DR0139_Dum02_body.xls A 1061888 Tue Jun 19 13:43:54 2018
DataSet_File_mod0874953.xlsx A 149835 Mon Feb 26 14:17:02 2018
File Path For Dataset-2018.07.11.xlsx A 34661 Mon Feb 12 09:27:17
I'm getting different results in jenkins pipeline when trying to sort a list compared to jenkins script console.
Jenkins version: 2.134
groovy version: 2.4.11
Code
def items = [[1, "09-Aug-2018 11:13" ],
[2, "11-Jul-2018 13:27" ],
[3, "02-Oct-2018 15:48" ],
[4, "03-Sep-2018 14:27" ],
[5, "08-Sep-2018 10:07" ],
[6, "11-Jul-2018 14:06" ],
[7, "08-Sep-2018 10:16" ],
[8, "09-Aug-2018 09:53" ],
[9, "16-Jul-2018 12:59" ],
[10, "08-Aug-2018 15:14" ],
[11, "12-Jul-2018 09:04" ],
[12, "08-Aug-2018 14:32" ],
[13, "08-Sep-2018 10:59" ],
[14, "08-Sep-2018 09:50" ],
[15, "16-Aug-2018 09:12" ],
[16, "19-Sep-2018 09:59" ],
[17, "18-Jul-2018 10:25" ],
[18, "11-Sep-2018 14:45" ],
[19, "12-Jul-2018 09:36" ],
[20, "02-Oct-2018 09:18" ]]
def itemsDateNotString = items.collect { [it[0], new Date().parse("dd-MMM-yyyy H:m", it[1])] }
println(itemsDateNotString)
def itemsSorted = itemsDateNotString.sort{ it[1] }
println(itemsSorted)
def itemsSortedReversed = itemsSorted.reverse()
println(itemsSortedReversed)
Results on my laptop and on Jenkins script console:
itemsDateNotString
[[1, Thu Aug 09 11:13:00 GMT 2018], [2, Wed Jul 11 13:27:00 GMT 2018], [3, Tue Oct 02 15:48:00 GMT 2018], [4, Mon Sep 03 14:27:00 GMT 2018], [5, Sat Sep 08 10:07:00 GMT 2018], [6, Wed Jul 11 14:06:00 GMT 2018], [7, Sat Sep 08 10:16:00 GMT 2018], [8, Thu Aug 09 09:53:00 GMT 2018], [9, Mon Jul 16 12:59:00 GMT 2018], [10, Wed Aug 08 15:14:00 GMT 2018], [11, Thu Jul 12 09:04:00 GMT 2018], [12, Wed Aug 08 14:32:00 GMT 2018], [13, Sat Sep 08 10:59:00 GMT 2018], [14, Sat Sep 08 09:50:00 GMT 2018], [15, Thu Aug 16 09:12:00 GMT 2018], [16, Wed Sep 19 09:59:00 GMT 2018], [17, Wed Jul 18 10:25:00 GMT 2018], [18, Tue Sep 11 14:45:00 GMT 2018], [19, Thu Jul 12 09:36:00 GMT 2018], [20, Tue Oct 02 09:18:00 GMT 2018]]
itemsSorted:
[[2, Wed Jul 11 13:27:00 GMT 2018], [6, Wed Jul 11 14:06:00 GMT 2018], [11, Thu Jul 12 09:04:00 GMT 2018], [19, Thu Jul 12 09:36:00 GMT 2018], [9, Mon Jul 16 12:59:00 GMT 2018], [17, Wed Jul 18 10:25:00 GMT 2018], [12, Wed Aug 08 14:32:00 GMT 2018], [10, Wed Aug 08 15:14:00 GMT 2018], [8, Thu Aug 09 09:53:00 GMT 2018], [1, Thu Aug 09 11:13:00 GMT 2018], [15, Thu Aug 16 09:12:00 GMT 2018], [4, Mon Sep 03 14:27:00 GMT 2018], [14, Sat Sep 08 09:50:00 GMT 2018], [5, Sat Sep 08 10:07:00 GMT 2018], [7, Sat Sep 08 10:16:00 GMT 2018], [13, Sat Sep 08 10:59:00 GMT 2018], [18, Tue Sep 11 14:45:00 GMT 2018], [16, Wed Sep 19 09:59:00 GMT 2018], [20, Tue Oct 02 09:18:00 GMT 2018], [3, Tue Oct 02 15:48:00 GMT 2018]]
itemsSortedReversed:
[[3, Tue Oct 02 15:48:00 GMT 2018], [20, Tue Oct 02 09:18:00 GMT 2018], [16, Wed Sep 19 09:59:00 GMT 2018], [18, Tue Sep 11 14:45:00 GMT 2018], [13, Sat Sep 08 10:59:00 GMT 2018], [7, Sat Sep 08 10:16:00 GMT 2018], [5, Sat Sep 08 10:07:00 GMT 2018], [14, Sat Sep 08 09:50:00 GMT 2018], [4, Mon Sep 03 14:27:00 GMT 2018], [15, Thu Aug 16 09:12:00 GMT 2018], [1, Thu Aug 09 11:13:00 GMT 2018], [8, Thu Aug 09 09:53:00 GMT 2018], [10, Wed Aug 08 15:14:00 GMT 2018], [12, Wed Aug 08 14:32:00 GMT 2018], [17, Wed Jul 18 10:25:00 GMT 2018], [9, Mon Jul 16 12:59:00 GMT 2018], [19, Thu Jul 12 09:36:00 GMT 2018], [11, Thu Jul 12 09:04:00 GMT 2018], [6, Wed Jul 11 14:06:00 GMT 2018], [2, Wed Jul 11 13:27:00 GMT 2018]]
However when I run this code as part of the jenkins pipeline it thows and exception and the results are below:
itemsDateNotString
[[1, Thu Aug 09 11:13:00 GMT 2018], [2, Wed Jul 11 13:27:00 GMT 2018], [3, Tue Oct 02 15:48:00 GMT 2018], [4, Mon Sep 03 14:27:00 GMT 2018], [5, Sat Sep 08 10:07:00 GMT 2018], [6, Wed Jul 11 14:06:00 GMT 2018], [7, Sat Sep 08 10:16:00 GMT 2018], [8, Thu Aug 09 09:53:00 GMT 2018], [9, Mon Jul 16 12:59:00 GMT 2018], [10, Wed Aug 08 15:14:00 GMT 2018], [11, Thu Jul 12 09:04:00 GMT 2018], [12, Wed Aug 08 14:32:00 GMT 2018], [13, Sat Sep 08 10:59:00 GMT 2018], [14, Sat Sep 08 09:50:00 GMT 2018], [15, Thu Aug 16 09:12:00 GMT 2018], [16, Wed Sep 19 09:59:00 GMT 2018], [17, Wed Jul 18 10:25:00 GMT 2018], [18, Tue Sep 11 14:45:00 GMT 2018], [19, Thu Jul 12 09:36:00 GMT 2018], [20, Tue Oct 02 09:18:00 GMT 2018]]
itemsSorted:
Wed Jul 11 13:27:00 GMT 2018
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: java.util.Date.reverse() is applicable for argument types: () values: []
Possible solutions: every(), every(groovy.lang.Closure), before(java.util.Date), parse(java.lang.String), parse(java.lang.String, java.lang.String), parse(java.lang.String, java.lang.String, java.util.TimeZone)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:58)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:49)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at tmpDeleteArtifacts.printItems(<PATH>)
at tmpDeleteArtifacts.call(<PATH>)
at standardPipeline.call(<PATH>)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:77)
at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
Provided that the code works on jenkins script consolei'm not sure why the pipeline is giving different results. I believe it's the sort function because I would expect it to sort the list in date order however it is returning an single date in the pipeline whereas the script console is returning the sorted list.
Thanks in advance,
Jay.
It happens to you, because you run list transformation in Groovy CPS mode, which has plenty of limitations when running in continuation passing style. Solution is fairly simple - extract your Groovy code to a function annotated with #NonCPS so Jenkins executes it outside CPS mode. Take a look at this simple example:
Jenkinsfile:
node {
stage("Test") {
def items = [[1, "09-Aug-2018 11:13" ],
[2, "11-Jul-2018 13:27" ],
[3, "02-Oct-2018 15:48" ],
[4, "03-Sep-2018 14:27" ],
[5, "08-Sep-2018 10:07" ],
[6, "11-Jul-2018 14:06" ],
[7, "08-Sep-2018 10:16" ],
[8, "09-Aug-2018 09:53" ],
[9, "16-Jul-2018 12:59" ],
[10, "08-Aug-2018 15:14" ],
[11, "12-Jul-2018 09:04" ],
[12, "08-Aug-2018 14:32" ],
[13, "08-Sep-2018 10:59" ],
[14, "08-Sep-2018 09:50" ],
[15, "16-Aug-2018 09:12" ],
[16, "19-Sep-2018 09:59" ],
[17, "18-Jul-2018 10:25" ],
[18, "11-Sep-2018 14:45" ],
[19, "12-Jul-2018 09:36" ],
[20, "02-Oct-2018 09:18" ]]
sortExample(items)
}
}
#NonCPS
def sortExample(items) {
def itemsDateNotString = items.collect { [it[0], new Date().parse("dd-MMM-yyyy H:m", it[1])] }
println(itemsDateNotString)
def itemsSorted = itemsDateNotString.sort{ it[1] }
println(itemsSorted)
def itemsSortedReversed = itemsSorted.reverse()
println(itemsSortedReversed)
}
Output:
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/test-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
[[1, Thu Aug 09 11:13:00 UTC 2018], [2, Wed Jul 11 13:27:00 UTC 2018], [3, Tue Oct 02 15:48:00 UTC 2018], [4, Mon Sep 03 14:27:00 UTC 2018], [5, Sat Sep 08 10:07:00 UTC 2018], [6, Wed Jul 11 14:06:00 UTC 2018], [7, Sat Sep 08 10:16:00 UTC 2018], [8, Thu Aug 09 09:53:00 UTC 2018], [9, Mon Jul 16 12:59:00 UTC 2018], [10, Wed Aug 08 15:14:00 UTC 2018], [11, Thu Jul 12 09:04:00 UTC 2018], [12, Wed Aug 08 14:32:00 UTC 2018], [13, Sat Sep 08 10:59:00 UTC 2018], [14, Sat Sep 08 09:50:00 UTC 2018], [15, Thu Aug 16 09:12:00 UTC 2018], [16, Wed Sep 19 09:59:00 UTC 2018], [17, Wed Jul 18 10:25:00 UTC 2018], [18, Tue Sep 11 14:45:00 UTC 2018], [19, Thu Jul 12 09:36:00 UTC 2018], [20, Tue Oct 02 09:18:00 UTC 2018]]
[Pipeline] echo
[[2, Wed Jul 11 13:27:00 UTC 2018], [6, Wed Jul 11 14:06:00 UTC 2018], [11, Thu Jul 12 09:04:00 UTC 2018], [19, Thu Jul 12 09:36:00 UTC 2018], [9, Mon Jul 16 12:59:00 UTC 2018], [17, Wed Jul 18 10:25:00 UTC 2018], [12, Wed Aug 08 14:32:00 UTC 2018], [10, Wed Aug 08 15:14:00 UTC 2018], [8, Thu Aug 09 09:53:00 UTC 2018], [1, Thu Aug 09 11:13:00 UTC 2018], [15, Thu Aug 16 09:12:00 UTC 2018], [4, Mon Sep 03 14:27:00 UTC 2018], [14, Sat Sep 08 09:50:00 UTC 2018], [5, Sat Sep 08 10:07:00 UTC 2018], [7, Sat Sep 08 10:16:00 UTC 2018], [13, Sat Sep 08 10:59:00 UTC 2018], [18, Tue Sep 11 14:45:00 UTC 2018], [16, Wed Sep 19 09:59:00 UTC 2018], [20, Tue Oct 02 09:18:00 UTC 2018], [3, Tue Oct 02 15:48:00 UTC 2018]]
[Pipeline] echo
[[3, Tue Oct 02 15:48:00 UTC 2018], [20, Tue Oct 02 09:18:00 UTC 2018], [16, Wed Sep 19 09:59:00 UTC 2018], [18, Tue Sep 11 14:45:00 UTC 2018], [13, Sat Sep 08 10:59:00 UTC 2018], [7, Sat Sep 08 10:16:00 UTC 2018], [5, Sat Sep 08 10:07:00 UTC 2018], [14, Sat Sep 08 09:50:00 UTC 2018], [4, Mon Sep 03 14:27:00 UTC 2018], [15, Thu Aug 16 09:12:00 UTC 2018], [1, Thu Aug 09 11:13:00 UTC 2018], [8, Thu Aug 09 09:53:00 UTC 2018], [10, Wed Aug 08 15:14:00 UTC 2018], [12, Wed Aug 08 14:32:00 UTC 2018], [17, Wed Jul 18 10:25:00 UTC 2018], [9, Mon Jul 16 12:59:00 UTC 2018], [19, Thu Jul 12 09:36:00 UTC 2018], [11, Thu Jul 12 09:04:00 UTC 2018], [6, Wed Jul 11 14:06:00 UTC 2018], [2, Wed Jul 11 13:27:00 UTC 2018]]
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
"Use #NonCPS-annotated functions for slightly more complex work. This means more involved processing, logic, and transformations. This lets you leverage additional Groovy & functional features for more powerful, concise, and performant code."
Source: https://jenkins.io/blog/2017/02/01/pipeline-scalability-best-practice/
I have a list of transactions in an array.
=> [Wed, 23 Oct 2013, Mon, 18 Nov 2013, Fri, 22 Nov 2013, Mon, 13 Jan 2014, Tue, 28 Jan 2014, Mon, 03 Feb 2014, Mon, 10 Feb 2014, Tue, 18 Feb 2014, Fri, 07 Mar 2014, Mon, 31 Mar 2014, Mon, 07 Apr 2014, Tue, 10 Jun 2014, Mon, 30 Jun 2014, Mon, 22 Sep 2014, Mon, 06 Oct 2014, Fri, 14 Nov 2014, Tue, 18 Nov 2014, Fri, 26 Dec 2014, Thu, 15 Jan 2015, Mon, 23 Mar 2015, Mon, 20 Apr 2015]
I need to compare the dates of each transaction and list any months that are missing in the list of months and year. Here is what I have now...
#find_transactions = (#user.transactions.find_all { |t| (t.name 'name' })
#trans_dates = #find_transactions.map(&:date).sort!.map { |s| Date.strptime(s, '%Y-%m') }.each_cons(2).map{ |d1,d2| d1.next_month == d2 }
This method currently gives me a true or false if each month is there but I need to actually have the method print a list of months that are missing. I would like to have it print the month and year together.
Here is the response this method gives me...
=> [true, false, true, false, false, true, false, true, false, false, false, true, true]
I want a response like this...
=> [March 2015, December 2014, September 2014]
Thanks in advance!
Edit: For array already being composed of date objects you can do:
require 'date'
dates = [Wed, 23 Oct 2013, Mon, 18 Nov 2013, Fri, 22 Nov 2013, Mon, 13 Jan 2014, Tue, 28 Jan 2014, Mon, 03 Feb 2014, Mon, 10 Feb 2014, Tue, 18 Feb 2014, Fri, 07 Mar 2014, Mon, 31 Mar 2014, Mon, 07 Apr 2014, Tue, 10 Jun 2014, Mon, 30 Jun 2014, Mon, 22 Sep 2014, Mon, 06 Oct 2014, Fri, 14 Nov 2014, Tue, 18 Nov 2014, Fri, 26 Dec 2014, Thu, 15 Jan 2015, Mon, 23 Mar 2015, Mon, 20 Apr 2015]
all_dates = []
dates.first.upto(dates.last) {|x| all_dates << x.strftime('%b %Y') if x.day == 1 || x == dates.first}
d = dates.map {|x| x.strftime('%b %Y')}.uniq
p (all_dates - d)
#=> ["Dec 2013", "May 2014", "Jul 2014", "Aug 2014", "Feb 2015"]
Edit: Below methods are for an array of date strings
You can try this:
require 'date'
dates = ["Wed, 23 Oct 2013", "Mon, 18 Nov 2013", "Fri, 22 Nov 2013", "Mon, 13 Jan 2014", "Tue, 28 Jan 2014", "Mon, 03 Feb 2014", "Mon, 10 Feb 2014", "Tue, 18 Feb 2014", "Fri, 07 Mar 2014", "Mon, 31 Mar 2014", "Mon, 07 Apr 2014", "Tue, 10 Jun 2014", "Mon, 30 Jun 2014", "Mon, 22 Sep 2014", "Mon, 06 Oct 2014", "Fri, 14 Nov 2014", "Tue, 18 Nov 2014", "Fri, 26 Dec 2014", "Thu, 15 Jan 2015", "Mon, 23 Mar 2015", "Mon, 20 Apr 2015"]
all_dates = []
d = dates.map {|x| Date.parse(x[8..-1])}.uniq
counter = d.first
until counter == d.last
all_dates << counter
counter = counter.next_month
end
p (all_dates - d).map {|x| x.strftime('%b %Y')}
#=> ["Dec 2013", "May 2014", "Jul 2014", "Aug 2014", "Feb 2015"]
Another (more concise) way would be:
require 'date'
dates = ["Wed, 23 Oct 2013", "Mon, 18 Nov 2013", "Fri, 22 Nov 2013", "Mon, 13 Jan 2014", "Tue, 28 Jan 2014", "Mon, 03 Feb 2014", "Mon, 10 Feb 2014", "Tue, 18 Feb 2014", "Fri, 07 Mar 2014", "Mon, 31 Mar 2014", "Mon, 07 Apr 2014", "Tue, 10 Jun 2014", "Mon, 30 Jun 2014", "Mon, 22 Sep 2014", "Mon, 06 Oct 2014", "Fri, 14 Nov 2014", "Tue, 18 Nov 2014", "Fri, 26 Dec 2014", "Thu, 15 Jan 2015", "Mon, 23 Mar 2015", "Mon, 20 Apr 2015"]
all_dates = []
d = dates.map {|x| Date.parse(x[8..-1])}.uniq
d.first.upto(d.last) {|x| all_dates << x if x.day == 1}
p (all_dates - d).map {|x| x.strftime('%b %Y')}
#=> ["Dec 2013", "May 2014", "Jul 2014", "Aug 2014", "Feb 2015"]
This is one way you could do that.
Code
require 'date'
def missing_months(dates)
a = dates.map { |s| d = Date.strptime(s, '%a, %d %b %Y'); d - d.day + 1 }
(all_months_in_range(*a.minmax) -a).map { |d| d.strftime('%b %Y') }
end
def all_months_in_range(f,l)
(12*(l.year-f.year)+l.month-f.month+1).times.map do |i|
y,m = (f.month+i).divmod(12)
y += f.year
(m=12; y-=1) if m ==0
Date.new(y,m)
end
end
Example
dates = ['Wed, 23 Oct 2013', 'Mon, 18 Nov 2013', 'Fri, 22 Nov 2013',
'Fri, 14 Nov 2014', 'Tue, 18 Nov 2014', 'Fri, 26 Dec 2014',
'Mon, 13 Jan 2014', 'Tue, 28 Jan 2014', 'Mon, 03 Feb 2014',
'Mon, 31 Mar 2014', 'Mon, 07 Apr 2014', 'Tue, 10 Jun 2014',
'Mon, 30 Jun 2014', 'Mon, 22 Sep 2014', 'Mon, 06 Oct 2014',
'Mon, 10 Feb 2014', 'Tue, 18 Feb 2014', 'Fri, 07 Mar 2014',
'Thu, 15 Jan 2015', 'Mon, 23 Mar 2015', 'Mon, 20 Apr 2015']
missing_months(dates)
#=> ["Dec 2013", "May 2014", "Jul 2014", "Aug 2014", "Feb 2015"]
Notice that the dates needn't be sorted.
Explanation
For the example above:
a = dates.map { |s| d = Date.strptime(s, '%a, %d %b %Y'); d - d.day + 1 }
#=> [#<Date: 2013-10-01 ((2456567j,0s,0n),+0s,2299161j)>,
# #<Date: 2013-11-01 ((2456598j,0s,0n),+0s,2299161j)>,
# ...
# #<Date: 2015-04-01 ((2457114j,0s,0n),+0s,2299161j)>]
Notice that each of these dates is on the first of the month. Next, obtain the first and last of these dates:
f,l = a.minmax
f #=> [#<Date: 2013-10-01 ((2456567j,0s,0n),+0s,2299161j)>,
l #=> #<Date: 2015-04-01 ((2457114j,0s,0n),+0s,2299161j)>]
Now pass f and l to all_months_in_range to create an array that contains a date object for the first day of each month between f and l.
b = all_months_in_range(f,l)
#=> [#<Date: 2013-10-01 ((2456567j,0s,0n),+0s,2299161j)>,
# #<Date: 2013-11-01 ((2456598j,0s,0n),+0s,2299161j)>,
# ...
# #<Date: 2015-04-01 ((2457114j,0s,0n),+0s,2299161j)>]
b.size #=> 19
I will skip an explanation of this helper method, as it is quite straightforward.
Compute that difference between arrays b and a to obtain the missing beginning-of-month dates:
c = b-a
#=> [#<Date: 2013-12-01 ((2456628j,0s,0n),+0s,2299161j)>,
# #<Date: 2014-05-01 ((2456779j,0s,0n),+0s,2299161j)>,
# #<Date: 2014-07-01 ((2456840j,0s,0n),+0s,2299161j)>,
# #<Date: 2014-08-01 ((2456871j,0s,0n),+0s,2299161j)>,
# #<Date: 2015-02-01 ((2457055j,0s,0n),+0s,2299161j)>]
Lastly, convert these dates to the desired format:
c.map { |d| d.strftime('%b %Y') }
#=> ["Dec 2013", "May 2014", "Jul 2014", "Aug 2014", "Feb 2015"]
Addendum: after reading #Sid's answer, I see I could have saved myself some trouble in my helper method by using Date#next_month:
def all_months_in_range(f,l)
(12*(l.year-f.year)+l.month-f.month+1).times.map { |i| f.next_month(i) }
end
This isn't very elegant, but it worked. I started with your original code, #SupremeA, and built off of that.
require 'date'
dates = ['Wed, 23 Oct 2013', 'Mon, 18 Nov 2013', 'Fri, 22 Nov 2013', 'Mon, 13 Jan 2014', 'Tue, 28 Jan 2014', 'Mon, 03 Feb 2014', 'Mon, 10 Feb 2014', 'Tue, 18 Feb 2014', 'Fri, 07 Mar 2014', 'Mon, 31 Mar 2014', 'Mon, 07 Apr 2014', 'Tue, 10 Jun 2014', 'Mon, 30 Jun 2014', 'Mon, 22 Sep 2014', 'Mon, 06 Oct 2014', 'Fri, 14 Nov 2014', 'Tue, 18 Nov 2014', 'Fri, 26 Dec 2014', 'Thu, 15 Jan 2015', 'Mon, 23 Mar 2015', 'Mon, 20 Apr 2015']
new_dates = []
dates.each { |d| new_dates.push(Date.parse(d).strftime('%B %Y')) }
sorted_dates = new_dates.map { |s| Date.strptime(s, '%B %Y') }.sort.uniq
missing_months = []
sorted_dates.each_cons(2) do |d1,d2|
d = d1
while d.next_month != d2
missing_months.push(d.next_month.strftime('%B %Y'))
d = d >> 1
end
end
p missing_months
=> ["December 2013", "May 2014", "July 2014", "August 2014", "February 2015"]
I am struggling to sort this hash listed below:
{
17=>{:id=>17, :count=>1, :created_at=>Wed, 05 Sep 2012 19:02:34 UTC +00:00},
14=>{:id=>14, :count=>2, :created_at=>Sun, 02 Sep 2012 19:20:28 UTC +00:00},
9=>{:id=>9, :count=>0, :created_at=>Sun, 02 Sep 2012 17:09:35 UTC +00:00},
10=>{:id=>10, :count=>2, :created_at=>Sat, 01 Sep 2012 17:09:56 UTC +00:00},
11=>{:id=>11, :count=>0, :created_at=>Fri, 31 Aug 2012 19:13:57 UTC +00:00},
12=>{:id=>12, :count=>2, :created_at=>Thu, 30 Aug 2012 19:19:32 UTC +00:00},
13=>{:id=>13, :count=>0, :created_at=>Thu, 23 Aug 2012 19:20:09 UTC +00:00}
}
The above hash should be sorted on count and created_of and look like this hash:
{
12=>{:id=>12, :count=>2, :created_at=>Thu, 30 Aug 2012 19:19:32 UTC +00:00},
10=>{:id=>10, :count=>2, :created_at=>Sat, 01 Sep 2012 17:09:56 UTC +00:00},
14=>{:id=>14, :count=>2, :created_at=>Sun, 02 Sep 2012 19:20:28 UTC +00:00},
17=>{:id=>17, :count=>1, :created_at=>Wed, 05 Sep 2012 19:02:34 UTC +00:00},
13=>{:id=>13, :count=>0, :created_at=>Thu, 23 Aug 2012 19:20:09 UTC +00:00},
11=>{:id=>11, :count=>0, :created_at=>Fri, 31 Aug 2012 19:13:57 UTC +00:00},
9=>{:id=>9, :count=>0, :created_at=>Sun, 02 Sep 2012 17:09:35 UTC +00:00}
}
Assuming you are using Ruby 1.9 where hashes have order:
Hash[data.sort_by { |key, h| [-h[:count], h[:created_at]] }]
In Ruby 1.8, you cannot sort a hash as a hash has no order. In Ruby 1.9, a hash iteration order is defined by the insertion order. You therefore have to create a new hash in which you will insert the elements in the right order.
sorted_keys = hash.keys.sort
sorted_hash = Hash.new
sorted_keys.each do |k|
sorted_hash[k] = hash[k]
end
I need this
require 'date'
DateTime.parse "Mon, Dec 27 6:30pm"
to return a DateTime for 6:30pm in the EDT timezone, but it returns one in UTC. How can I get a EST DateTime or convert the UTC one into an EDT DateTime with a 6:30pm value?
OK I'm going to offer an answer to my own question
require 'time'
ENV["TZ"] = "US/Eastern"
Time.parse("Mon, Dec 27 6:30pm").to_datetime
=> #<DateTime: 2011-12-27T18:30:00-05:00 (117884327/48,-5/24,2299161)>
In Rails, this worked nicely for me
DateTime.parse "Mon, Dec 27 6:30pm #{Time.zone}"
It won't work in vanilla Ruby though.
Final answer ;-)
require 'date'
estHoursOffset = -5
estOffset = Rational(estHoursOffset, 24)
date = (DateTime.parse("Mon, Dec 27 6:30pm") - (estHoursOffset/24.0)).new_offset(estOffset)
(or -4 for EDT)
DateTime#change()
You can try using change() after parsing it to alter the timezone offset:
DateTime.parse( "Mon, Dec 27 6:30pm" ).change( offset: '-0400' )
# => Wed, 27 Dec 2017 18:30:00 -0400
You can also just use the hours:
DateTime.parse( "Mon, Dec 27 6:30pm" ).change( offset: '-4' )
# => Wed, 27 Dec 2017 18:30:00 -0400
But, be careful, you cannot use an integer:
DateTime.parse( "Mon, Dec 27 6:30pm" ).change( offset: -4 )
# => Wed, 27 Dec 2017 18:30:00 +0000
If you need to determine the correct offset to use based on a time zone you can do something like this:
offset = ( Time.zone_offset('EDT') / 1.hour ).to_s
# => "-4"
DateTime.parse( "Mon, Dec 27 6:30pm" ).change( offset: offset )
# => Wed, 27 Dec 2017 18:30:00 -0400
You can also use change() to manually set other parts of the DateTime as well, like setting the hour to noon:
DateTime.parse( "Mon, Dec 27 6:30pm" ).change( offset: '-4', hour: 12 )
# => Wed, 27 Dec 2017 12:00:00 -0400
Be careful with that one because you can see that it's cleared the minutes as well.
Here's the docs for the change() method: http://api.rubyonrails.org/v5.1/classes/DateTime.html#method-i-change
If you're using Rails' ActiveSupport:
"Mon, Dec 27 6:30pm".in_time_zone(-4.hours).to_datetime
# => Mon, 27 Dec 2021 18:30:00 -0400
Time.find_zone(-4.hours).parse("Mon, Dec 27 6:30pm").to_datetime
# => Mon, 27 Dec 2021 18:30:00 -0400
If you want to use the local daylight saving time (DST) rules, you could use:
"Mon, Dec 27 6:30pm".in_time_zone("Eastern Time (US & Canada)")
# => Mon, 27 Dec 2021 18:30:00 EST -05:00
Time.find_zone("Eastern Time (US & Canada)").parse("Mon, Dec 27 6:30pm")
# => Mon, 27 Dec 2021 18:30:00 EST -05:00
Time.find_zone("Eastern Time (US & Canada)").parse("Mon, Dec 27 6:30pm").to_datetime
# => Mon, 27 Dec 2021 18:30:00 -0500
Time.find_zone("Eastern Time (US & Canada)").parse("Mon, Jun 27 6:30pm")
# => Sun, 27 Jun 2021 18:30:00 EDT -04:00
Time.find_zone("Eastern Time (US & Canada)").parse("Mon, Jun 27 6:30pm").to_datetime
# => Sun, 27 Jun 2021 18:30:00 -0400
Time.find_zone("EST5EDT").parse("Mon, Jun 27 6:30pm").to_datetime
# => Sun, 27 Jun 2021 18:30:00 -0400
Notice the date in June, above, is automatically set to EDT (-0400) because this date is in DST, contrary to the December date.
To force EST regardless if date is within DST or not:
Time.find_zone("EST").parse("Mon, Jun 27 6:30pm")
# => Sun, 27 Jun 2021 18:30:00 EST -05:00
Time.find_zone("EST").parse("Mon, Jun 27 6:30pm").to_datetime
# => Sun, 27 Jun 2021 18:30:00 -0500