I have a Cshell script that I am modifying to have related input and output locations.
the functionality all happens in a foreach loop like so:
set INPUT_LOCATION_LIST = "loc1 loc2 loc3 loc4"
foreach location ($INPUT_LOCATION_LIST)
#***Do some stuff ***
end
I would like to have an output list with different values than the input list but traverse through it each iteration through the foreach loop. The man for foreach simply has
foreach name (wordlist)
as the definition. so only dealing with a single one. my current thought on dealing with it is have the wordlist contain both input and output location and then parse it out in the script:
set INPUT_LOCATION_LIST = "loc1;out1 loc2;out2 loc3;out3 loc4;out4"
so im wondering if anyone has a better way to do that.
You can iterate through one list using foreach and through the other one by treating at like an array and using shift:
set INPUT_LOCATION_LIST = "loc1 loc2 loc3 loc4"
set OUT_LIST = (out1 out2 out3 out4)
foreach location ($INPUT_LOCATION_LIST)
do_something $location $OUT_LIST[1]
shift OUT_LIST
end
I don't normally use csh, but your question caught my eye. There's probably a solution with less steps, but this kind of thing worked in my version of csh:
foreach location ($INPUT_LOCATION_LIST)
set one_word_with_space = ${location:s/;/ /}
set loc_out = ($one_word_with_space)
set loc = ${loc_out[1]}
set out = ${loc_out[2]}
...
end
Basic idea is just to change the semi-colon separated string into a space-separated string, then parse that into an array.
Related
I want to open many files to do an analysis through python.
The title changes in three ways, ID, intervention and drug_type
So I define the path where the files are and create three list variables which contain the names I want to swap parts of the string with.
path = r'R:\Storage\healthsciences\SPRH\pwelch'
ID = ['P1','P2','P3','P4','P5']
intervention = ['intervention1','intervention2']
drug_type = ['drug1','drug2','drug3','drug4']
filename = P1_intervention1_drug2_pwelch_output
How would I get the different sections to change in a for loop?
So..
filename =
[ID]'_'[intervention]'_'drug_type'_pwelch_output.mat'
with each list variable changing as it loops over
Any push in the right direction is much appreciated :)
ids = ['P1','P2','P3','P4','P5']
interventions = ['intervention1', 'intervention2']
drug_types = ['drug1','drug2','drug3','drug4']
for id in ids:
for intervention in interventions:
for drug_type in drug_types:
filename = f"{id}_{intervention}_{drug_type}_pwelch_output.mat"
# use the filename
I'm trying to create a random number generator in Lua. I found out that I can just use math.random(1,100) to randomize a number between 1 and 100 and that should be sufficient.
But I don't really understand how to use the randomize number as variables in the script.
Tried this but of course it didn't work.
$randomCorr = math.random(1,100);
http.request_batch({
{"POST", "https://store.thestore.com/priceAndOrder/selectProduct", headers={["Content-Type"]="application/json;charset=UTF-8"}, data="{\"ChoosenPhoneModelId\":4,\"PricePlanId\":\"phone\",\"CorrelationId\":\"$randomCorr\",\"DeliveryTime\":\"1 vecka\",\"$$hashKey\":\"006\"},\"ChoosenAmortization\":{\"AmortizationLength\":0,\"ChoosenDataPackage\":{\"Description\":\"6 GB\",\"PricePerMountInKr\":245,\"DataAmountInGb\":6,\"$$hashKey\":\"00W\"},\"ChoosenPriceplan\":{\"IsPostpaid\":true,\"Title\":\"Fastpris\",\"Description\":\"Fasta kostnader till fast pris\",\"MonthlyAmount\":0,\"AvailiableDataPackages\":null,\"SubscriptionBinding\":0,\"$$hashKey\":\"00K\"}}", auto_decompress=true},
{"GET", "https://store.thestore.com/api/checkout/getproduct?correlationId=$randomCorr", auto_decompress=true},
})
In Lua, you can not start a variable name with $. This is where your main issue is at. Once the $ is removed from your code, we can easily see how to refer to variables in Lua.
randomCorr = math.random(100)
print("The random number:", randomCorr)
randomCorr = math.random(100)
print("New Random Number:", randomCorr)
Also, concatenation does not work the way you are implying it into your Http array. You have to concatenate the value in using .. in Lua
Take a look at the following example:
ran = math.random(100)
data = "{\""..ran.."\"}"
print(data)
--{"14"}
The same logic can be implied into your code:
data="{\"ChoosenPhoneModelId\":4,\"PricePlanId\":\"phone\",\"CorrelationId\":\""..randomCorr.."\",\"DeliveryTime\":\"1 vecka\",\"$$hashKey\":\"006\"},\"ChoosenAmortization\":{\"AmortizationLength\":0,\"ChoosenDataPackage\":{\"Description\":\"6 GB\",\"PricePerMountInKr\":245,\"DataAmountInGb\":6,\"$$hashKey\":\"00W\"},\"ChoosenPriceplan\":{\"IsPostpaid\":true,\"Title\":\"Fastpris\",\"Description\":\"Fasta kostnader till fast pris\",\"MonthlyAmount\":0,\"AvailiableDataPackages\":null,\"SubscriptionBinding\":0,\"$$hashKey\":\"00K\"}}"
Or you can format the value in using one of the methods provided by the string library
Take a look at the following example:
ran = math.random(100)
data = "{%q}"
print(string.format(data,ran))
--{"59"}
The %q specifier will take whatever you put as input, and safely surround it with quotations
The same logic can be applied to your Http Data.
Here is a corrected version of the code snippet:
local randomCorr = math.random(1,100)
http.request_batch({
{"POST", "https://store.thestore.com/priceAndOrder/selectProduct", headers={["Content-Type"]="application/json;charset=UTF-8"}, data="{\"ChoosenPhoneModelId\":4,\"PricePlanId\":\"phone\",\"CorrelationId\":\"" .. randomCorr .. "\",\"DeliveryTime\":\"1 vecka\",\"$$hashKey\":\"006\"},\"ChoosenAmortization\":{\"AmortizationLength\":0,\"ChoosenDataPackage\":{\"Description\":\"6 GB\",\"PricePerMountInKr\":245,\"DataAmountInGb\":6,\"$$hashKey\":\"00W\"},\"ChoosenPriceplan\":{\"IsPostpaid\":true,\"Title\":\"Fastpris\",\"Description\":\"Fasta kostnader till fast pris\",\"MonthlyAmount\":0,\"AvailiableDataPackages\":null,\"SubscriptionBinding\":0,\"$$hashKey\":\"00K\"}}", auto_decompress=true},
{"GET", "https://store.thestore.com/api/checkout/getproduct?correlationId=" .. randomCorr, auto_decompress=true},
})
There is something called $$hashKey also, in the quoted string. Not sure if that is supposed to be referencing a variable or not. If it is, it also needs to be concatenated into the resulting string, using the .. operator (just like with the randomCorr variable).
Split the group record in to different records :
for eg :
Input : (A,(3,2,3))
Output in to 3 new lines:
A,3
A,2
A,3
Can any one let me know the option to do this please?
The problem is when you convert the output of Arraylist to tuple then it will be difficult to achieve what you want, so I recommend this approach, so it will be easy to get the output .
In your UDF code, instead of creating Arraylist, append the output into string seperated by comma and return back to pig script.
You final output should be like this from UDF as a string ie "3,2,3"
Then use the below code to get the result
C = FOREACH B GENERATE $0,NewRollingCount(BagToString($1)) AS rollingCnt
D = FOREACH C GENERATE $0,FLATTEN(TOKENIZE(rollingcnt));
DUMP D;
I have a variable titled F.
Describe F returns:
F: {group: bytearray,indexkey: {(indexkey: chararray)}}
Dump F returns:
(321,{(CHOW),(DREW)})
(5011,{(CHOW),(DREW)})
(5825,{(TANNER),(SPITZENBERGER)})
(16631,{(CHOW),(DREW)})
(34299,{(CHOW),(DREW)})
(35044,{(TANNER),(SPITZENBERGER)})
(65623,{(CHOW),(DREW)})
(74597,{(SPITZENBERGER),(TANNER)})
(83499,{(SPITZENBERGER),(TANNER)})
(90257,{(SPITZENBERGER),(TANNER)})
What I need is to produce an output that looks like this (only 1st row as an example):
(321,DREW,{(CHOW)})
I've tried using deference to pull out the first element by using this:
G = FOREACH F generate indexkey.$0;
But, this still returns the whole tuple.
Can anyone suggest a method for doing this? I was under the impression that the deference operator should allow me to do this.
Thanks in advance!
Daniel
You can't index into bags like that. The reason for that is bags don't have any notion of ordering. Selecting the first item in a bag should be treated as picking a random one.
Either way, if you want only one item instead of all of them you can used a nested FOREACH to pull a LIMIT of 1:
first = FOREACH F {
lim = LIMIT indexkey 1;
GENERATE group, lim;
}
(disclaimer: I can't test this code right now, if it doesn't work let me know. Hopefully you can get the gist)
You can take this a bit further and FLATTEN it to remove the bag of one item entirely, but be careful in that if the bag is empty i think you throw away the entire record in this case.
first = FOREACH F {
lim = LIMIT indexkey 1;
GENERATE group, FLATTEN(lim);
}
I am writing kettle transformation.
My input file looks like following
sessionId=40936a7c-8af9|txId=40936a7d-8af9-11e|field3=val3|field4=val4|field5=myapp|field6=03/12/13 15:13:34|
Now, how do i process this file? I am completely at loss.
First step is CSV file input with | as delimiter
My analysis will be based on "Value" part of name value pair.
Has anyone processes such files before?
Since you have already splitted the records into fields of 'key=value' you could use an expression transform to cut the string into two by locating the position of the = character and create two out ports where one holds the key and the other the value.
From there it depends what you want to do with the information, if you want to store them as key/value route them trough a union, or use a router transform to send them to different targets.
Her is an example of an expression to split the pairs:
You could use the Modified Javascript Value Step, add this step after this grouping with pipes.
Now do some parsing javascript like this:
var mainArr = new Array();
var sessionIdSplit = sessionId.toString().split("|");
for(y = 0; y < sessionIdSplit.length; y++){
mainArr[y] = sessionIdSplit[y].toString();
//here you can add another loop to parse again and split the key=value
}
Alert("mainArr: "+ mainArr);