I know there are various examples on stackoverflow thats discusses this problem but even after going through them I was still unable to find a solution. What is the best way to convert to numeric with regard to OHLCV thats appearing as characters. What am I missing here or doing wrong?
rm(list = ls(all = TRUE))
setwd("~/Documents/R")
library(jsonlite)
library(ggplot2)
library(forecast)
library(xts)
library(quantmod)
library(anytime)
library(highfrequency)
library(lubridate)
setwd("~/Documents/R/STK/csv")
gdax <- read.csv(file="GDAX_BTCUSD.csv", sep="\t", dec=".", header=FALSE, stringsAsFactors = FALSE, na.strings = "NA")
options(scipen=999)
gdax <- na.omit(gdax[,c(9,4,5,6,7,8)])
colnames(gdax) <- c("Date", "Open", "High", "Low", "Close", "Volume")
> class(gdax)
[1] "data.frame"
> head(gdax)
Date Open High Low Close Volume
1 2016-05-16 00:00:00 461.4800000000 461.4900000000 461.4700000000 461.4900000000 5.345398
2 2016-05-16 00:01:00 461.4700000000 461.4900000000 461.4700000000 461.4900000000 4.650470
3 2016-05-16 00:02:00 461.4800000000 461.4900000000 461.4700000000 461.4900000000 1.360030
4 2016-05-16 00:03:00 461.4900000000 461.4900000000 461.4900000000 461.4900000000 0.586860
5 2016-05-16 00:04:00 461.4700000000 461.4700000000 461.4700000000 461.4700000000 0.135180
6 2016-05-16 00:05:00 461.4700000000 461.5000000000 461.4400000000 461.5000000000 11.257970
GDAX_BTC <- xts(gdax[, 2:6], order.by=as.POSIXct(gdax[,1], tz="UTC"))
> head(GDAX_BTC)
Open High Low Close Volume
2016-05-16 00:00:00 "461.4800000000" "461.4900000000" "461.4700000000" "461.4900000000" " 5.34539836"
2016-05-16 00:01:00 "461.4700000000" "461.4900000000" "461.4700000000" "461.4900000000" " 4.65047000"
2016-05-16 00:02:00 "461.4800000000" "461.4900000000" "461.4700000000" "461.4900000000" " 1.36003000"
2016-05-16 00:03:00 "461.4900000000" "461.4900000000" "461.4900000000" "461.4900000000" " 0.58686000"
2016-05-16 00:04:00 "461.4700000000" "461.4700000000" "461.4700000000" "461.4700000000" " 0.13518000"
2016-05-16 00:05:00 "461.4700000000" "461.5000000000" "461.4400000000" "461.5000000000" " 11.25797000"
> class(GDAX_BTC)
[1] "xts" "zoo"
GDAX_BTC <- xts(gdax[,-1], order.by=as.POSIXct(gdax$Date, format = "%Y-%m-%d %H:%M:%S"))
# no changes in the output.
#
R version 3.4.3 (2017-11-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.2
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] lubridate_1.7.1 highfrequency_0.5.2 anytime_0.3.0 quantmod_0.4-12 TTR_0.23-2 xts_0.10-1
[7] zoo_1.8-0 forecast_8.2 ggplot2_2.2.1 jsonlite_1.5
loaded via a namespace (and not attached):
[1] Rcpp_0.12.14 magrittr_1.5 munsell_0.4.3 colorspace_1.3-2 lattice_0.20-35 rlang_0.1.6 quadprog_1.5-5
[8] stringr_1.2.0 plyr_1.8.4 tools_3.4.3 nnet_7.3-12 parallel_3.4.3 grid_3.4.3 RApiDatetime_0.0.3
[15] timeDate_3042.101 gtable_0.2.0 tseries_0.10-42 yaml_2.1.16 lazyeval_0.2.1 lmtest_0.9-35 tibble_1.4.1
[22] curl_3.1 fracdiff_1.4-2 stringi_1.1.6 compiler_3.4.3 pillar_1.0.1 scales_0.5.0
Related
We have a spring boot application which has a API for third party integration.For third party integration we are using apache camel and groovy as a scripting language.We are using producerTemplate for camel and XML based camel and route context. In route context we used groovy for some manipulation.
But after running for few days we noticed that non heap memory keeps increasing.
Here is a snap of non heap memory usage
Then we took a memory-related statistics of that process and found that there are many groovy/lang/GroovyClassLoader#0x0000000100a32e10. Some of them are alive and some of them are dead.There are thousands of them.
class_loader classes bytes parent_loader alive? type
<bootstrap> 3417 5835717 null live <internal>
0x0000000081b805b8 1 1471 null dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082d2ced0 15 40146 0x0000000082d2cf48 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000082c3d2c0 1 1474 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082d3d6d0 13 35357 0x0000000082d3d748 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000083158b10 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000800cd800 120 316382 0x00000000800cd860 live sun/misc/Launcher$AppClassLoader#0x000000010000f6a0
0x00000000833c5330 14 68990 null live org/codehaus/groovy/runtime/callsite/CallSiteClassLoader#0x0000000100a42750
0x0000000081f5e5f8 1 1471 null dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082ff86f0 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082fbeae8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000833c3b28 1 1471 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000830f1f18 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000081e141c8 1 1474 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082c3aee8 1 1474 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000833b6b10 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000830e9b20 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000ea687a58 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000080d928e8 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000083239b10 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000833b4f08 11 26125 0x00000000833b4f80 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000083348300 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000816dfd58 1 1471 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000830f2f38 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082c39ef8 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000833c2700 1 1485 0x0000000082d2af08 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000816df150 1 1471 null dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000080d91498 1 1473 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082d3c690 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082d49a90 12 34183 0x0000000082d49b08 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000083345b70 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000833c2f78 1 1471 0x00000000833b8628 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082fbdab0 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000833b3f70 11 25942 0x00000000833b3fe8 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000082ef0ea0 1 1474 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x000000008309d758 11 26153 0x000000008309d7d0 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000082fbaaa8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082fb9a88 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082fbce88 10 25290 0x0000000082fbcf00 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000830deb78 10 25302 0x00000000830debf0 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000830eab78 17 55775 0x00000000830eabf0 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x000000008308b770 13 34835 0x000000008308b7e8 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000082d35ea8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000081b429c8 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000833c3f40 1 1471 0x00000000833b3f70 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000833c5b40 1 880 null dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082eee290 1 1471 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000830f4f70 11 26125 0x00000000830f4fe8 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000833c43b8 1 1471 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000833c57b8 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000830ecb88 11 26110 0x00000000830ecc00 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000082ff8678 11 26153 0x0000000082ff86f0 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x000000008300eb88 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000083389fb0 11 26125 0x000000008338a028 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000080d92050 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082fbea70 10 25290 0x0000000082fbeae8 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000830f3f80 12 34459 0x00000000830f3ff8 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000833c33a8 1 1471 0x000000008334acb8 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000081b7fd20 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000816dc5f0 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000081b42518 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000833c6790 1 1473 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000081b80108 1 1474 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000080d91c68 1 880 null dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x000000008299be28 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000081b42900 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000083079fb0 11 25942 0x000000008307a028 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000833b4f80 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082eeda50 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x000000008299b658 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000831ae3d8 11 26110 0x00000000831ae450 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000082fbda38 10 25290 0x0000000082fbdab0 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000081f5e530 1 1471 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000830ebbc8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000081b80970 1 878 null dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082d34618 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082d3c618 10 25290 0x0000000082d3c690 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000082fbaa30 11 26153 0x0000000082fbaaa8 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000833c3bf0 1 880 null dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000833b3fe8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082eede38 1 892 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000830edbd8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x000000008299ba40 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x000000008309d7d0 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x000000008308b7e8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000080d92438 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000081e14100 1 1474 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082d35e30 10 25292 0x0000000082d35ea8 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000816df588 1 1497 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000830f4fe8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082fb9a10 11 26153 0x0000000082fb9a88 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000ea686a98 11 26153 0x00000000ea686b10 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000833c4bc8 15 72430 0x00000000800cc640 live org/codehaus/groovy/runtime/callsite/CallSiteClassLoader#0x0000000100a42750
0x00000000830f3ff8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x000000008300eff8 1 880 null dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000ea683290 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000804938b0 0 0 0x00000000800cd800 live java/util/ResourceBundle$RBClassLoader#0x000000010005ef58
0x0000000080d92820 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000831b07e0 14 32627 0x00000000831b0858 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x000000008323abd8 11 26110 0x000000008323ac50 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x000000008334bbc8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000830debf0 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082d39380 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082d3f780 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000830dfe58 11 25944 0x00000000830dfed0 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000083116a48 11 26153 0x0000000083116ac0 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000831af640 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x00000000833c3a60 1 1471 0x00000000831750f8 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000833c5258 12 67191 null live org/codehaus/groovy/runtime/callsite/CallSiteClassLoader#0x0000000100a42750
0x00000000830e9aa8 10 25290 0x00000000830e9b20 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x00000000829f5738 0 0 0x00000000800cc640 live groovy/text/markup/MarkupTemplateEngine$TemplateGroovyClassLoader#0x0000000100a332a8
0x00000000830f1ea0 11 26110 0x00000000830f1f18 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x000000008309b6b8 0 0 0x00000000800cc640 live groovy/lang/GroovyClassLoader#0x0000000100a32e10
0x0000000082c3d770 1 1474 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x00000000833c5e88 1 1471 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000082eee358 1 881 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000080d91560 1 1473 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x000000008299b720 1 880 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
0x0000000083239a98 14 32627 0x0000000083239b10 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000083348288 11 26110 0x0000000083348300 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x0000000082d5fb68 16 49041 0x0000000082d5fbe0 live groovy/lang/GroovyClassLoader$InnerLoader#0x0000000100ba9e58
0x000000008300eac0 1 881 0x00000000800cc640 dead sun/reflect/DelegatingClassLoader#0x0000000100009df8
................................................................................................................................
................................................................................................................................
................................................................................................................................
**total = 13675 90164 204988152 N/A alive=1, dead=13674 N/A**
After running for few days non heap memory usage exceed 1 GB and application hang and could not respond. Then we restart application and this phenomenon recur.
What could be the reason for such a high usage of non heap memory ? Is there something else to mitigate it ?
Update:
Added dependencies in build.gradle file for camel:
compile("org.apache.camel:camel-spring-boot-starter:2.21.2")
compile("org.apache.camel:camel-script:2.21.2")
compile("org.apache.camel:camel-groovy:2.21.2")
compile("org.apache.camel:camel-xstream:2.21.2")
compile("org.apache.camel:camel-jackson:2.21.2")
compile("org.apache.camel:camel-jdbc:2.21.2")
compile("org.apache.camel:camel-jpa:2.21.2")
compile("org.apache.camel:camel-http:2.21.2")
At startup of our application spring boot imports camel_config.xml file.
#ImportResource(locations = "../camel_config.xml}")
Content of camel config context:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.2.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">
<import resource="classpath:../ref_route.xml"/>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<propertyPlaceholder
id="properties"
location="classpath:application.properties"/>
<routeContextRef ref="ref_route"/>
</camelContext>
Calling from service layer to camel:
LinkedHashMap<String, String> response = producerTemplate.requestBodyAndHeaders("direct:ref_route", body, headers, LinkedHashMap.class);
camel route context file: ref_route.xml
I have two text files which has space seperated values, i want to combine the files based on a key column from both files and output in another file.
location.txt
1 21.5 23
2 24.5 20
3 19.5 19
4 22.5 15
5 24.5 12
6 19.5 12
data.txt which has milllion of data, but i will give simple few entries here,
2004-03-31 03:38:15.757551 2 1 122.153 -3.91901 11.04 2.03397
2004-02-28 00:59:16.02785 3 2 19.9884 37.0933 45.08 2.69964
2004-02-28 01:03:16.33393 11 3 19.3024 38.4629 45.08 2.68742
2004-02-28 01:06:16.013453 17 4 19.1652 38.8039 45.08 2.68742
2004-02-28 01:06:46.778088 18 5 19.175 38.8379 45.08 2.69964
2004-02-28 01:08:45.992524 22 6 19.1456 38.9401 45.08 2.68742
What i trying is to combine these two files based on the key value of column 1 from location.txt and column 4 from data.txt and get the result in format as below by combining all the data from data.txt and column 2 and 3 from location.txt..
2004-03-31 03:38:15.757551 2 1 122.153 -3.91901 11.04 2.03397 21.5 23
2004-02-28 00:59:16.02785 3 2 19.9884 37.0933 45.08 2.69964 24.5 20
2004-02-28 01:03:16.33393 11 3 19.3024 38.4629 45.08 2.68742 19.5 19
2004-02-28 01:06:16.013453 17 4 19.1652 38.8039 45.08 2.68742 22.5 15
2004-02-28 01:06:46.778088 18 5 19.175 38.8379 45.08 2.69964 24.5 12
2004-02-28 01:08:45.992524 22 6 19.1456 38.9401 45.08 2.68742 19.5 12
I'm using awk command:
awk -F' ' "NR==FNR{label[$1]=$1;x[$1]=$2;y[$1]=$3;next}; ($2==label[$2]){print $0 "," x[$2] y[$3]}" location.txt data.txt > result.txt
But I'm not getting the output as i expected, Can anyone help me fix this?
can we get the result file in csv format with space replaced with comma?
In awk:
$ awk '
NR==FNR { # process location.txt
a[$1]=$2 OFS $3 # hash using $1 as key
next # next record
}
$4 in a { # process data.txt
print $0,a[$4] # output record and related location
}' location.txt data.txt # mind the file order
2004-03-31 03:38:15.757551 2 1 122.153 -3.91901 11.04 2.03397 21.5 23
2004-02-28 00:59:16.02785 3 2 19.9884 37.0933 45.08 2.69964 24.5 20
2004-02-28 01:03:16.33393 11 3 19.3024 38.4629 45.08 2.68742 19.5 19
2004-02-28 01:06:16.013453 17 4 19.1652 38.8039 45.08 2.68742 22.5 15
2004-02-28 01:06:46.778088 18 5 19.175 38.8379 45.08 2.69964 24.5 12
2004-02-28 01:08:45.992524 22 6 19.1456 38.9401 45.08 2.68742 19.5 12
With bash and join
join -1 1 -2 4 <(sort -k1,1 -n location.txt) <(sort -k4,4 -n data.txt) -o 2.1,2.2,2.3,2.4,2.5,2.6,2.7,2.8,1.2,1.3
Output:
2004-03-31 03:38:15.757551 2 1 122.153 -3.91901 11.04 2.03397 21.5 23
2004-02-28 00:59:16.02785 3 2 19.9884 37.0933 45.08 2.69964 24.5 20
2004-02-28 01:03:16.33393 11 3 19.3024 38.4629 45.08 2.68742 19.5 19
2004-02-28 01:06:16.013453 17 4 19.1652 38.8039 45.08 2.68742 22.5 15
2004-02-28 01:06:46.778088 18 5 19.175 38.8379 45.08 2.69964 24.5 12
2004-02-28 01:08:45.992524 22 6 19.1456 38.9401 45.08 2.68742 19.5 12
See: man join
I'm using golang to call pppd and then kill it after a while. However I got a lot of defunct proccesses in this way.
This is how I run pppd
exec.Command("sh", "-c", "pppd call vpn").CombinedOutput()
This is how I kill it.
exec.Command("sh", "-c", "pkill pppd").CombinedOutput()
Then I got a lot of this
root 31541 23536 0 10:54 ? 00:00:00 [pppd] <defunct>
root 31929 23356 0 10:55 ? 00:00:00 [pptpgw] <defunct>
root 31933 23356 0 10:55 ? 00:00:00 [pptpcm] <defunct>
root 31940 23356 0 10:55 ? 00:00:00 [pppd] <defunct>
root 31993 23536 0 10:55 ? 00:00:00 [pptpgw] <defunct>
root 31997 23536 0 10:55 ? 00:00:00 [pptpcm] <defunct>
root 31998 23536 0 10:55 ? 00:00:00 [pppd] <defunct>
root 32012 23356 0 10:55 ? 00:00:00 [pptpgw] <defunct>
root 32016 23356 0 10:55 ? 00:00:00 [pptpcm] <defunct>
root 32017 23356 0 10:56 ? 00:00:00 [pppd] <defunct>
root 32070 23536 0 10:56 ? 00:00:00 [pptpgw] <defunct>
root 32074 23536 0 10:56 ? 00:00:00 [pptpcm] <defunct>
root 32075 23536 0 10:56 ? 00:00:00 [pppd] <defunct>
root 32083 23356 0 10:56 ? 00:00:00 [pptpgw] <defunct>
root 32087 23356 0 10:56 ? 00:00:00 [pptpcm] <defunct>
root 32089 23356 0 10:56 ? 00:00:00 [pppd] <defunct>
root 32131 23536 0 10:57 ? 00:00:00 [pptpgw] <defunct>
root 32135 23536 0 10:57 ? 00:00:00 [pptpcm] <defunct>
root 32148 23536 0 10:57 ? 00:00:00 [pppd] <defunct>
root 32160 23356 0 10:57 ? 00:00:00 [pptpgw] <defunct>
root 32164 23356 0 10:57 ? 00:00:00 [pptpcm] <defunct>
root 32165 23356 0 10:57 ? 00:00:00 [pppd] <defunct>
root 32177 23536 0 10:57 ? 00:00:00 [pptpgw] <defunct>
root 32181 23536 0 10:57 ? 00:00:00 [pptpcm] <defunct>
How can I avoid defunct processes.
These "zombie" processes are created when a process has finished, but the parent has not read their exit status via the wait system call.
I would guess that all you need to do is call (*Cmd).Wait() on every command structure you create. Obviously This will be less straight forward than you may like, since you probably don't want to call Wait on the first command until after the second command is finished.
EDIT: As is pointed out in the comments, (*Cmd).CombinedOutput() calls (*Cmd).Run(), which calls (*Cmd).Wait()... So the above is wrong. The real answer in this case is that for some reason sh isn't cleaning up, and so the solution is to cut out the midle man and do the call like so:
exec.Command("pppd", "call", "vpn").CombinedOutput()
That'll teach me to read the docs a little closer next time...
A simpler way to cancel your command would be to use exec.CommandContext. e.g.
ctx, cancel := context.WithCancel(context.Background())
exec.CommandContext(ctx, "pppd", "call", "vpn").CombinedOutput()
// in some other goroutine...
cancel()
Maybe this would solve your zombie problem?
run subprocess in a new thread
go exec.Command("sh", "-c", "pppd call vpn").CombinedOutput()
kill subprocess
exec.Command("pkill", "pppd").CombinedOutput().CombinedOutput()
I'm running a validation software and I want all of the output sent to a text file and have the results of multiple files placed/appended to the same file. I thought my code was working, but I just discovered I'm only getting the results from 1 file output to the text file.
java -jar /Applications/epubcheck-3.0.1/epubcheck-3.0.1.jar ~/Desktop/Validator/*.epub 2>&1 | tee -a ~/Desktop/Validator/EPUBCHECK3_results.txt
open ~/Desktop/Validator/EPUBCHECK3_results.txt
EDIT
When I run the same .jar file using Windows command line it will process a batch of files and appeand the results appropriately. I would just do this, but it would mean having to switch work stations and transferring files to validate them. I would like to get this running through the Unix shell on my Mac system so that I don't have to do unnecessary work. Command line that IS working below:
FOR /f %%1 in ('dir /b "C:\Users\scrawfo\Desktop\epubcheck\drop epubs here\*.epub"') do (
echo %%1 >> epubcheck.txt
java -jar "C:\Users\scrawfo\Desktop\epubcheck\epubcheck-3.0.jar" "C:\Users\scrawfo\Desktop\epubcheck\drop epubs here\%%1" 2>> epubcheck.txt
echo. >> epubcheck.txt)
notepad epubcheck.txt
del epubcheck.txt
syntax provided by you is correct there might be some problem with java output or something Try Executing it without redirection
cat test
Output:-
This is Test File ...............
Next Executed Command with same syntax
ps l 2>&1 | tee -a test
Output:-
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME
COMMAND 4 0 3287 1 20 0 4060 572 n_tty_ Ss+ tty2
0:00 /sbin/mingetty /dev/tty2 4 0 3289 1 20 0 4060 572
n_tty_ Ss+ tty3 0:00 /sbin/mingetty /dev/tty3 4 0 3291
1 20 0 4060 576 n_tty_ Ss+ tty4 0:00 /sbin/mingetty
/dev/tty4 4 0 3295 1 20 0 4060 576 n_tty_ Ss+ tty5
0:00 /sbin/mingetty /dev/tty5 4 0 3297 1 20 0 4060 572
n_tty_ Ss+ tty6 0:00 /sbin/mingetty /dev/tty6 4 0 19086
1 20 0 4060 572 n_tty_ Ss+ tty1 0:00 /sbin/mingetty
/dev/tty1 4 0 20837 20833 20 0 108432 2148 wait Ss pts/0
0:00 -bash 4 0 21471 20837 20 0 108124 1036 - R+ pts/0
0:00 ps l 0 0 21472 20837 20 0 100908 664 pipe_w S+ pts/0
0:00 tee -a test
Checked File
cat test
Output:-(Appended properly)
This is Test File ...............
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND 4 0
3287 1 20 0 4060 572 n_tty_ Ss+ tty2 0:00
/sbin/mingetty /dev/tty2 4 0 3289 1 20 0 4060 572
n_tty_ Ss+ tty3 0:00 /sbin/mingetty /dev/tty3 4 0 3291
1 20 0 4060 576 n_tty_ Ss+ tty4 0:00 /sbin/mingetty
/dev/tty4 4 0 3295 1 20 0 4060 576 n_tty_ Ss+ tty5
0:00 /sbin/mingetty /dev/tty5 4 0 3297 1 20 0 4060 572
n_tty_ Ss+ tty6 0:00 /sbin/mingetty /dev/tty6 4 0 19086
1 20 0 4060 572 n_tty_ Ss+ tty1 0:00 /sbin/mingetty
/dev/tty1 4 0 20837 20833 20 0 108432 2148 wait Ss pts/0
0:00 -bash 4 0 21471 20837 20 0 108124 1036 - R+ pts/0
0:00 ps l 0 0 21472 20837 20 0 100908 664 pipe_w S+ pts/0
0:00 tee -a test
Need help with debugging permission issue during hadoop streaming. I try to start awk streaming:
// mkdir to all nodes
[pocal#oscbda01 ~]$ for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 ; do ssh -f oscbda$i mkdir -p /home/pocal/KS/comverse/awk/; done;
// copy streaming files to all nodes
[pocal#oscbda01 ~]$ for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 ; do scp * oscbda$i:/home/pocal/KS/comverse/awk/; done;
// give 777 permission to all files
[pocal#oscbda01 ~]$ for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 ; do ssh -f oscbda$i chmod 777 /home/pocal/KS/comverse/awk/*; done;
//start streaming
[pocal#oscbda01 ~]$ hadoop fs -rm -r /user/pocal/ks/comverse/one/out;\
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.0.0-cdh4.3.0.jar \
-Dmapreduce.job.reduces=0 \
-Dmapred.reduce.tasks=0 \
-mapper "awk -f /home/pocal/KS/comverse/awk/data_change.awk -f /home/pocal/KS/comverse/awk/selfcare.awk -f /home/pocal/KS/comverse/awk/selfcare_secondary_mapping.awk -f /home/pocal/KS/comverse/awk/out_sort.awk" \
-input "/user/pocal/ks/comverse/one/" \
-output "/user/pocal/ks/comverse/one/out"
And get error…
………..
attempt_201311041208_1379_m_000010_2: awk: fatal: can't open source file `/home/pocal/KS/comverse/awk/data_change.awk' for reading (Permission denied)
13/12/12 09:01:32 INFO mapred.JobClient: Task Id : attempt_201311041208_1379_m_000004_2, Status : FAILED
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
attempt_201311041208_1379_m_000004_2: awk: fatal: can't open source file `/home/pocal/KS/comverse/awk/data_change.awk' for reading (Permission denied)
13/12/12 09:01:33 INFO mapred.JobClient: Task Id : attempt_201311041208_1379_m_000003_2, Status : FAILED
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
attempt_201311041208_1379_m_000003_2: awk: fatal: can't open source file `/home/pocal/KS/comverse/awk/data_change.awk' for reading (Permission denied)
13/12/12 09:01:37 INFO mapred.JobClient: Job complete: job_201311041208_1379
13/12/12 09:01:37 INFO mapred.JobClient: Counters: 8
13/12/12 09:01:37 INFO mapred.JobClient: Job Counters
13/12/12 09:01:37 INFO mapred.JobClient: Failed map tasks=1
13/12/12 09:01:37 INFO mapred.JobClient: Launched map tasks=52
13/12/12 09:01:37 INFO mapred.JobClient: Data-local map tasks=12
13/12/12 09:01:37 INFO mapred.JobClient: Rack-local map tasks=40
13/12/12 09:01:37 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=348738
13/12/12 09:01:37 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=2952
13/12/12 09:01:37 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/12/12 09:01:37 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/12/12 09:01:37 ERROR streaming.StreamJob: Job not Successful!
Streaming Command Failed!
Check on one machine:
[pocal#oscbda01 ~]$ ssh oscbda10 ls -l /home/pocal/KS/comverse/awk/data_change.awk
-rwxrwxrwx 1 pocal pocal 1548 Dec 10 10:05 /home/pocal/KS/comverse/awk/data_change.awk
Permissions Ok…
Does anybody has any ideas?
was problem with parent directory:
[pocal#oscbda01 .ssh]$ ssh oscbda05 ls -l /home/|grep pocal
total 24
drwx------ 5 pocal pocal 4096 Dec 10 09:52 pocal
[pocal#oscbda01 .ssh]$ ssh oscbda05 ls -l /home/pocal
total 4
drwxrwxrwx 3 pocal pocal 4096 Dec 10 09:52 KS
[pocal#oscbda01 .ssh]$ ssh oscbda05 ls -l /home/pocal/KS
total 4
drwxrwxrwx 3 pocal pocal 4096 Dec 10 09:52 comverse
[pocal#oscbda01 .ssh]$ ssh oscbda05 ls -l /home/pocal/KS/comverse/
total 4
drwxrwxrwx 2 pocal pocal 4096 Dec 10 10:05 awk
[pocal#oscbda01 .ssh]$ ssh oscbda05 ls -l /home/pocal/KS/comverse/awk/
total 216
-rwxrwxrwx 1 pocal pocal 4398 Dec 10 10:05 calltype_checker.awk
-rwxrwxrwx 1 pocal pocal 16173 Dec 10 10:05 ch_rebuild.c
-rwxrwxrwx 1 pocal pocal 14643 Dec 10 10:05 ch_rebuild.dat
-rwxrwxrwx 1 pocal pocal 1548 Dec 10 10:05 data_change.awk
-rwxrwxrwx 1 pocal pocal 4080 Dec 10 10:05 decompress_incomming_data.sh
-rwxrwxrwx 1 pocal pocal 720 Dec 10 10:05 fms.awk
-rwxrwxrwx 1 pocal pocal 2502 Dec 10 10:05 load_func
-rwxrwxrwx 1 pocal pocal 1308 Dec 10 10:05 load_vars
-rwxrwxrwx 1 pocal pocal 199 Dec 10 10:05 load_vars_dynamic
-rwxrwxrwx 1 pocal pocal 1358 Dec 10 10:05 out.awk
-rwxrwxrwx 1 pocal pocal 1296 Dec 10 10:05 out_nosort.awk
-rwxrwxrwx 1 pocal pocal 1358 Dec 10 10:05 out_sort.awk
-rwxrwxrwx 1 pocal pocal 70041 Dec 10 10:05 selfcare.awk
-rwxrwxrwx 1 pocal pocal 54204 Dec 10 10:05 selfcare_secondary_mapping.awk
-rwxrwxrwx 1 pocal pocal 1847 Dec 10 10:05 stat.awk
Add permission:
[pocal#oscbda01 .ssh]$ for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 ; do ssh -f oscbda$i chmod +x /home/pocal; done;
Check:
[pocal#oscbda01 .ssh]$ ssh oscbda05 ls -l /home/|grep poc
drwx--x--x 5 pocal pocal 4096 Dec 10 09:52 pocal
And it works!