I am getting the error:
2014-07-30 14:34:07,104 [main] ERROR org.apache.pig.Main - ERROR 2997: Encountered IOException. org.apache.pig.tools.parameters.ParameterSubstitutionException: Undefined parameter : year found when trying to find the value of input.
after I do ./run_pig_script 2014-07-30?
I am trying to pass in year, month, and day to the input parameter in Pig.
Here is a condensed version of my shell script, which is named run_pig_script:
#!/bin/bash
# Hadoop Job Queue to use
JOB_QUEUE=LongRun
array=( "$#" )
time=${array[0]}
IFS='-' read -ra timestamp <<< "$time"
year=${timestamp[0]}
month=${timestamp[1]}
day=${timestamp[2]}
echo $year
echo $month
echo $day
pig -Dmapred.job.queue.name=$JOB_QUEUE -param "input=hdfs:///raw/tool/year=$year/month=$month/day=$day/hour=00/min=00/" -f process_data.pig
The error message appears in org.apache.pig.tools.parameters.PreprocessorContext.java:
while (bracketKeyMatcher.find()) {
if ( (bracketKeyMatcher.start() == 0) || (line.charAt( bracketKeyMatcher.start() - 1)) != '\\' ) {
key = bracketKeyMatcher.group(1);
if (!(param_val.containsKey(key))) {
String message;
if (parentKey == null) {
message = "Undefined parameter : " + key;
} else {
message = "Undefined parameter : " + key + " found when trying to find the value of " + parentKey + ".";
}
throw new ParameterSubstitutionException(message);
}
val = param_val.get(key);
if (val.contains("$")) {
val = val.replaceAll("(?<!\\\\)\\$", "\\\\\\$");
}
replaced_line = replaced_line.replaceFirst("\\$\\{"+key+"\\}", val);
}
}
Related
I have written a shell script in a groovy function which should return the the output (in a single line) as abcd-def-chart but, I am getting the output as shown below in 2 different lines:
abcd-def
-chart
My groovy code:
String getChartName(Map configuration = [:]) {
if (configuration.chartName != null) {
return configuration.chartName
}
chartName = ""
if (configuration.buildSystem == 'maven') {
chartName = getMavenProjectName() + "-chart"
}
echo "chartName: ${chartName}"
return chartName
}
String getMavenProjectName() {
echo "inside getMavenProjectName +++++++"
def mavenChartName = sh returnStdout:true, script: '''
#!/bin/bash
GIT_LOG=$(env -i git config --get remote.origin.url)
basename "$GIT_LOG" .git; '''
echo "mavenChartName: ${mavenChartName}"
return mavenChartName
}
I am trying to execute the open source code which finds the list of tables involved in SQL.
I am working on Retrieve table names from Oracle queries.
I understood the expressions and commands to some extent and tried it.
Details of my execution:
GetTable.pl file
same as in the link
test.sql file
I didn't use the one in link. Instead I had only a single SQL for testing.
SELECT emp_name FROM load_tables.temp;
Executed in Strawberry Perl
I tried the following
$ perl GetTable.pl
Usage : GetTable <sql query file>
$ perl test.sql
Can't locate object method "FROM" via package "load_tables" (perhaps you forgot to load "load_tables"?) at test.sql line 1
Can someone help me in executing it? I'm not sure if there is problem with code as I could see two people have executed successfully.
Perl code
#!/usr/bin/perl
use warnings;
#Function which gets the table names and formats and prints them.
sub printTable {
my $tab = shift;
$tab =~ s/,\s+/,/g;
$tab =~ s/\s+,/,/g;
my #out = split( /,/, $tab );
foreach ( #out ) {
$_ =~ s/ .*//;
print $opr, $_, "\n";
}
}
# Function which gets the indivdual queries and separtes the table
# names from the queries. Sub-Queries, co-related queries, etc..
# will also be handled.
sub process {
local $opr;
my $line = shift;
$line =~ s/\n/ /g;
if ( $line =~ m/^\s*(select|delete)/i ) {
if ( $line =~ m/^\s*select/i ) {
$opr = "SELECT: ";
}
else {
$opr = "DELETE: ";
}
if ( $line =~ m/from.*where/i ) {
while ( $line =~ m/from\s+(.*?)where/ig ) {
&printTable( $1 );
}
}
elsif ( $line =~ m/from.*;/i ) {
while ( $line =~ m/from\s+(.*);/ig ) {
&printTable( $1 );
}
}
}
elsif ( $line =~ m/^\s*update\s+(\w+)\s+/i ) {
$opr = "UPDATE: ";
&printTable( $1 );
}
elsif ( $line =~ m/^\s*insert\s+into\s+(\w+)\s+/i ) {
$opr = "INSERT: ";
&printTable( $1 );
}
}
#The main function which reads the files and reads the
#query into a variable and sends it to process function.
if ( #ARGV != 1 ) {
print "Usage: GetTable <sql query file>\n";
exit 1;
}
open QFILE, $ARGV[0] or die "File $ARGV[0]: $! \n";
my $flag = 0;
my $query = "";
my $conds = "select|insert|update|delete";
while ( <QFILE> ) {
next if ( /^$/ );
if ( $flag == 1 ) {
$query .= $_;
if ( /;\s*$/ ) {
$flag = 0;
&process( $query );
}
}
elsif ( /^\s*($conds).*;\s*/i ) {
&process( $_ );
}
elsif ( /^\s*($conds)/i ) {
$flag = 1;
$query = $_;
}
}
close QFILE;
Two important skills to learn as a programmer are a) accuracy in following instructions and b) reading the error message carefully.
You started by running GetTable.pl. But that program requires a parameter (the name of an SQL file to analyse) and the error message tried to tell you that.
I don't know why, but instead of doing what the error message told you to do (which would have been to run perl GetTable.pl test.sql) you decided to ask Perl to run your SQL file.
The second error message you got was the Perl compiler trying to make sense of the SQL that you asked it to run. But the Perl compiler doesn't understand SQL, it understands Perl. So it's no surprise that it got confused.
To fix it, do what your first error message suggested—run the command
$ perl GetTable.pl test.sql
My shell script is written in cygwin for windows:
// main.sh
#!/bin/bash
[ "$#" -lt 1 ] && echo "Usage: thisscript.sh <filename.txt>" && exit 0
filename=`basename -s .txt $1`
i=0
while [ $i == 0 ]
do
phantomjs --web-security=no myXHR.js $filename.txt
logLastLine=`tail -n 1 $filename.log`
if [[ "$logLastLine" =~ "Error" ]]; then
echo "Error occurs, now keep looping it..."
elseif [[ "$logLastLine" =~ "503" ]]; then
echo "Error occurs, now keep looping it..."
elseif [[ "$logLastLine" =~ "500" ]]; then
echo "Error occurs, now keep looping it..."
else
echo "Complete! Exiting the execution..."
i=1
fi
done
And here are the codes contained in the myXHR.js
// myXHR.js
phantom.onError = function(msg, trace) {
console.log("PhantomJS Error");
phantom.exit();
};
var fs = require('fs'), system = require('system');
if (system.args.length < 2) {
console.log("Usage: myXHR.js <FILE>");
}
var content = '',
f = null,
lines = null,
eol = "\n";
try {
f = fs.open(system.args[1], "r");
filename=system.args[1].replace(/\.txt/,"");
content = f.read();
} catch (e) {
console.log(e);
}
if (f) {
f.close();
}
var request = new XMLHttpRequest();
if (content) {
lines = content.split(eol);
for (i=0; i<(lines.length-1);i++) {
request.open('GET', "http://stackoverflow.com/", false);
request.send();
if （request.status === 200) {
try {
fs.write($filename.log, line[i] + "Succeed!", 'a');
} catch(e) {
console.log(e);
}
} else {
try {
fs.write($filename.log, line[i] + "Error!", 'a');
} catch(e) {
console.log(e);
}
}
}
phantom.exit();
To illustrate, the javascript, executed by PhantomJS, are reading 1st argument(a filename.txt file), passed into the shell script, line by line. For each line it sends a XMLHttpRequest to check the request status and writes it into filename.log file.
Error status number includes 503 and 500. Luckily these statuses are less likely to occur again if I resend the same XMLHttpRequest. So what I need to do is to set up a error handler which is for resend the same XMLHttpRequest when errors occur.
In this error handler, I use X=${tail -n 1 log} to see if there is a error status number(containing "503" or "500" string). For instance, if [[ "$X" =~ "503" ]]; then restart the execution of the javascript, by not giving i=1 and while loop never exits. Until it has finished reading the last line of the imported file without any error status numbers.
(I know it is awkward to handle error like this, but it was a quick solution that came to my mind.)
But this is theoretical. In practice, this script ended with an error "Memory exhausted". I reckon this error is triggered by the large amount of lines(>100k) in the $1 file, and it occurs in the JavaScript execution part. I used free -m command to get memory usage information, and I noticed that when Javascript is running, the used swap is increasing!
Could anybody teach me how to release the memory when the scripts is being executed.
i try to convert csv file to tsv using below code.
in my csv the first word has no value.
when i run the code it generate tsv file correctly , but it gives above error. please help
Thanks
$myfile = "file path";
function convert($filename)
{
if(#$fh_in = fopen("{$filename}.csv","r"))
{
$fh_out = fopen("{$filename}.tsv","a");
while(!feof($fh_in))
{
$line = array();
$line = fgetcsv($fh_in,1024);
fwrite($fh_out,implode("\t",$line)."\n");
}
fclose($fh_in);
fclose($fh_out);
}
else {
echo "File doesn’t exist\n";
return false;
}
echo "Conversion completed!\n";
return true;
}
convert($myfile);
According to PHP Manual, implode can be
string implode ( string $glue , array $pieces )
string implode ( array $pieces )
So in your case if $line is empty then it will execute with second prototype and consider \n as $pieces which is wrong. So check for value in $line before calling implode
I am writing a simple java and bash program, but it is not working. Let me know where is wrong.
Bash:
for i in [1..100]; do
echo $i
java prob2 $i
done
Java:
import java.io.*;
public class prob2
{
public static void main( String[] args )
{
int l = args.length;
if ( l == 1 )
{
int num = Integer.parseInt(args[0]);
while ( num != 0 && num != 1)
num = num - 2;
if ( num == 0 )
System.out.println("Even");
else if ( num == 1 )
System.out.println("Odd");
}
}
}
The error I'm getting is:
Exception in thread "main" java.lang.NumberFormatException: For input string: "[1..100]" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:492) at java.lang.Integer.parseInt(Integer.java:527) at prob2.main(prob2.java:10)
That's not how you would do a bash loop. Try this:
for i in `seq 1 100`; do
echo $i
java prob2 $i
done
As an aside, a faster algorithm for determining if a number is odd or even is to take it modulo 2:
if (num % 2 == 0) {
System.out.println("Even");
} else {
System.out.println("Odd");
}
You have to use curly braces, not array brackets:
for i in {1..100}; do
echo $i
java prob2 $i
done