I need to find out an array of lat lon within Canada Province shapefile. I'm quite new to geopandas
latlon = np.array([[ 41.68254852, -140.93778992],
[ 51.04940796, -140.93778992],
[ 60.4166674 , -140.93778992],
[ 69.7831192 , -140.93778992],
[ 79.14993286, -140.93778992],
[ 46.99043655, -140.62530518],
[ 56.35729599, -140.62530518],
[ 65.72415161, -140.62530518],
[ 75.09098816, -140.62530518],
[ 42.93146133, -140.31280518],
[ 52.29832458, -140.31280518],
[ 61.66518021, -140.31280518]])
geom = [Point(lat, lon) for lat, lon in latlon]
crs = None
geo_df = GeoDataFrame(latlon, crs=crs, geometry=geom)
Read in Canada Province shapefile from here:
http://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2016-eng.cfm
ca_sh = gpd.read_file("gpr_000b11a_e.dbf")
d = gpd.sjoin(geo_df, ca_sh, how = 'left', op='within')
But I got following warning and error:
Warning: CRS does not match!
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
in ()
----> 1 d = gpd.sjoin(geo_df, ca_sh, how = 'left', op='within')
/usr/local/lib/python3.4/site-packages/geopandas/tools/sjoin.py in
sjoin(left_df, right_df, how, op, lsuffix, rsuffix)
55 idxmatch = idxmatch[idxmatch.apply(len) > 0]
56
---> 57 r_idx = np.concatenate(idxmatch.values)
58 l_idx = np.concatenate([[i] * len(v) for i, v in
idxmatch.iteritems()])
59
ValueError: need at least one array to concatenate
Is there any suggestion?
Related
I want to get the value of rsi on h1 for each m15 candle, this is how I do this. However, with data larger than 500000 lines, this is very time consuming, is there any better way. Note that it is mandatory to resample each row to get the correct result
import talib
import pandas as pd
import numpy as np
def Data(df):
df['RSI1'] = talib.RSI(df['close'], timeperiod=13)
df['RSI2'] = talib.RSI(df['close'], timeperiod=21)
return df
#len(df) > 555555
df = pd.read_csv('m15_candle.csv')
for i in range(0, len(df)):
t = df.at[i, 'time']
if t.hour == 0 and t.minute == 0:
df = df[i:]
break
df = df.set_index('time')
ohlc = {
'open': 'first',
'high': 'max',
'low': 'min',
'close': 'last'
}
rsi1 = [0]*len(df)
rsi2 = [0]*len(df)
for i in range(100000, len(df)):
h1 = Data(df[:i].resample("1h", offset=0).apply(ohlc).dropna())
rsi1[i] = h1.iloc[-1]['RSI1']
rsi2[i] = h1.iloc[-1]['RSI2']
df['RSI1_h1'] = rsi1
df['RSI2_h1'] = rsi2
df = df.reset_index()
df.to_csv("data.csv", index = False)
For my studying purposes I am following along a very popular notebook for sentiment classification with Bert.
Kaggle notebook for sentiment classification with BERT
But in place of train the model like in notebook, i just load another model
MODEL_NAME = "nlptown/bert-base-multilingual-uncased-sentiment"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME)
and want to test this on my data, to get a heatmap and accuracy score likde on the end of this notebook.
But when i am at the step of evalution i get
TypeError: max() received an invalid combination of arguments - got (SequenceClassifierOutput, dim=int), but expected one of:
* (Tensor input)
* (Tensor input, Tensor other, *, Tensor out)
* (Tensor input, int dim, bool keepdim, *, tuple of Tensors out)
* (Tensor input, name dim, bool keepdim, *, tuple of Tensors out)
in evaluation function where it says
_, preds = torch.max(outputs, dim=1)
I tried to change this to
_, preds = torch.max(torch.tensor(outputs), dim=1)
But then a got another issue:
RuntimeError: Could not infer dtype of SequenceClassifierOutput
the method for evaluation looks like this:
def eval_model(model, data_loader, loss_fn, device, n_examples):
model = model.eval()
losses = []
correct_predictions = 0
with torch.no_grad():
for d in data_loader:
input_ids = d["input_ids"].to(device)
attention_mask = d["attention_mask"].to(device)
targets = d["targets"].to(device)
# Get model ouptuts
outputs = model(
input_ids=input_ids,
attention_mask=attention_mask,
)
_, preds = torch.max(outputs, dim=1)
loss = loss_fn(outputs, targets)
correct_predictions += torch.sum(preds == targets)
losses.append(loss.item())
return correct_predictions.double() / n_examples, np.mean(losses)
And outputs it self in the code above looks like this
SequenceClassifierOutput(loss=None, logits=tensor([[ 2.2241, 1.2025, 0.1638, -1.4620, -1.6424],
[ 3.1578, 1.3957, -0.1131, -1.8141, -1.9536],
[ 0.7273, 1.7851, 1.1237, -0.9063, -2.3822],
[ 0.9843, 0.9711, 0.5067, -0.7553, -1.4547],
[-0.4127, -0.8895, 0.0572, 0.3550, 0.7377],
[-0.4885, 0.6933, 0.8272, -0.3176, -0.7546],
[ 1.3953, 1.4224, 0.7842, -0.9143, -2.2898],
[-2.4618, -1.2675, 0.5480, 1.4326, 1.2893],
[ 2.5044, 0.9191, -0.1483, -1.4413, -1.4156],
[ 1.3901, 1.0331, 0.4259, -0.8006, -1.6999],
[ 4.2252, 2.6539, -0.0392, -2.6362, -3.3261],
[ 1.9750, 1.8845, 0.6779, -1.3163, -2.5570],
[ 5.1688, 2.2360, -0.6230, -2.9657, -2.9031],
[ 1.1857, 0.4277, -0.1837, -0.7163, -0.6682],
[ 2.1133, 1.3829, 0.5750, -1.3095, -2.2234],
[ 2.3258, 0.9406, -0.0115, -1.1673, -1.6775]], device='cuda:0'), hidden_states=None, attentions=None)
How i can make it work?
Kind regards
unfile('C:/Users/ttcho/Desktop/1234.py', wdir='C:/Users/ttcho/Desktop')
File "C:\ProgramData\Anaconda3\lib\site-packages\pytube\extract.py", line 132
cipher_url = [ parse_qs(formats[i]["signatureCipher"]) for i, data in enumerate(formats) ] except Exception: cipher_url = [ parse_qs(formats[i]["cipher"]) for i, data in enumerate(formats) ]
cipher_url = [ parse_qs(formats[i]["signatureCipher"]) for i, data in enumerate(formats) ] except Exception: cipher_url = [ parse_qs(formats[i]["cipher"]) for i, data in enumerate(formats) ]
does anyone can help me how to fix it? please
hi im using Maatwebsite with laravel everything working so good so far ..
but i want to update excel like this ..
number date amount
1 2020-01-01 10
1 2020-01-01 -5
1 2020-01-01 -5
2 2020-02-02 20
2 2020-02-02 -20
3 2020-03-03 50
3 2020-03-03 -50
3 2020-03-03 40
3 2020-03-03 -40
what i want thats the sum of the amount with number 1 sould return 0 or end the import
number date amount
1 2020-01-01 10
1 2020-01-01 -5
1 2020-01-01 -5
how can i check this (10 + -5 + -5 == 0) or fail
and for 2 and 3 ect ..
thanks a lot
I explained all the steps in comments but I will explain here as well.
When you loop through each row, you first need to grab all the rows where number field is 1, 2 or 3, what ever the number is currently in the iteration.
In this case, the first loop results will be:
number date amount
1 2020-01-01 10
1 2020-01-01 -5
1 2020-01-01 -5
Then you sum (add up) the values from the amount column.
In this case, it will sum like this:
10
-5
-5
-----
0
If the answer is not 0, take the current sheet ($sheet) and return what you currently have, to be exported.
Full class:
...
use Maatwebsite\Excel\Concerns\FromArray;
class ExampleExport implements FromArray
{
public function array() : array
{
// EXAMPLE hardcoded data for the purpose of this demo
$rows = [
[
'number' => 1,
'date' => '2020-01-01',
'amount' => 10
],
[
'number' => 1,
'date' => '2020-01-01',
'amount' => -5
],
[
'number' => 1,
'date' => '2020-01-01',
'amount' => -5
],
[
'number' => 2,
'date' => '2020-02-02',
'amount' => 20
],
[
'number' => 2,
'date' => '2020-02-02',
'amount' => -20 // <-------- Change this to -21 to test
],
[
'number' => 3,
'date' => '2020-03-03',
'amount' => 50
],
[
'number' => 3,
'date' => '2020-03-03',
'amount' => -50
],
[
'number' => 3,
'date' => '2020-03-03',
'amount' => 40
],
[
'number' => 3,
'date' => '2020-03-03',
'amount' => -40
],
];
// EXAMPLE hardcoded data for the purpose of this demo
$rows = collect($rows);
// You probably going to get this from your database so should look like this:
// $rows = \App\Model::all(['number', 'date', 'amount']); // OR below
// $rows = DB::table('examples')->select(['number', 'date', 'amount'])->get();
// Blank sheet
$sheet = [];
foreach ($rows as $row) {
// Get all the rows where the `number` field has the same value,
// for example 1, 2, or 3, then sum the `amount` field (add up).
//
// If the amount does not add up to 0, stop and return what you currently got
if ($rows->where('number', $row['number'])->sum('amount') !== 0) {
return $sheet;
}
// Else, add them to the sheet and continue.
$sheet[] = $row;
}
return $sheet;
}
}
Results:
When amount adds up to 0:
When amount does not add up to 0:
Here is an example of the file.
powrup.asm POWER_UP
......EXTERNAL_RAM_ADDRESSING_CHECK powrup.asm:461
......EXRAM powrup.asm:490
......INRAM powrup.asm:540
......OUTPUT_TEST powrup.asm:573
............AD_READ douttst.asm:276
............AD_READ douttst.asm:366
......OUTPUT2_TEST powrup.asm:584
............AD_READ douttst2.asm:253
............AD_READ douttst2.asm:342
......OUTPUT3_TEST powrup.asm:599
............AD_READ douttst3.asm:307
............AD_READ douttst3.asm:398
......INPUT_TEST powrup.asm:614
......PROGRAM_PINS2_INPUT powrup.asm:629
......ARINC_TEST powrup.asm:633
............ARINC_LEVEL_TEST artest.asm:178
..................AD_READ arltst.asm:204
..................AD_READ arltst.asm:250
..................AD_READ arltst.asm:300
..................AD_READ arltst.asm:346
..................AD_READ arltst.asm:396
..................AD_READ arltst.asm:442
............ARINC_READ artest.asm:209
............ARINC_WORD_TXRX_TEST artest.asm:221
..................ARINC_OUT artxrx.asm:207
..................ARINC_READ artxrx.asm:221
............ARINC_READ artest.asm:251
............ARINC_WORD_TXRX_TEST artest.asm:263
..................ARINC_OUT artxrx.asm:207
..................ARINC_READ artxrx.asm:221
......PROGRAM_PINS2_INPUT powrup.asm:640
......PROGRAM_PIN_TEST powrup.asm:642
......PT_RCVR_BITE powrup.asm:645
............AD_READ10 ptbite.asm:225
..................AD_READ adread10.asm:141
............AD_READ10 ptbite.asm:308
..................AD_READ adread10.asm:141
............AD_READ10 ptbite.asm:384
..................AD_READ adread10.asm:141
............AD_READ10 ptbite.asm:467
..................AD_READ adread10.asm:141
............AD_READ10 ptbite.asm:542
..................AD_READ adread10.asm:141
............AD_READ10 ptbite.asm:622
..................AD_READ adread10.asm:141
......PROGRAM_PINS2_INPUT powrup.asm:653
......EXEC_INIT powrup.asm:663
The ... represents the call depth. The file name after the line indicates the file name
and the line number it was called from in the parent. I can parse the file. What I am trying to do once I have parsed the file is put the data in a n-ary tree.
I am doing a Data coupling and Control Coupling analysis and have already collected all the set/use data for the all the variables in the build. I need to now be able to traverse the tree and based on the depth figure out if there are any set before use situations or any set but not used situations. I thought a tree traversal would make the most sense.
Here is an example of the of the collected data:
$hash_DB{'alt_deviation_evaluation.asm->ALT_STATUS'} = [
'alt_deviation_evaluation.asm',
'ALT_STATUS',
'1.1',
1,
"",
"",
"135,188,202,242",
"130,144"
];
'alt_deviation_evaluation.asm->ALT_STATUS' is the file name and variable name.
'alt_deviation_evaluation.asm', File name
'ALT_STATUS', Variable name
'1.1', versions of file
1, indicates has been processed
"", not used (maybe in future)
"", not used (maybe in future)
"135,188,202,242", variable Set line numbers for this fileVariable
"130,144" Variable Use line number for this file/Variable
I also have an array with all the variable names. Shortened example:
our #vars_list = (
'A429_TX_BUFFER_LENGTH',
'A429_TX_INPUT_BUFFER',
'A429_TX_INT_MASK',
'ABS_ALT_DIFF',
'ACTUAL_ALT',
'ADDRESS_FAIL',
'AD_CONV_FAIL',
'AD_CONV_SIGNAL',
'AD_DATA',
'AD_FAIL',
'AD_STATUS',
'AIR_MODE',
'AIR_MODE_COUNT',
'AIR_MODE_LAST',
'ALPHA_COR_SSM',
'ALPHA_EC_SSM',
'ALPHA_GRAD_SSM',
'ALPHA_LE_SSM',
'ALPHA_LG_SSM',
'ALPHA_MAX_MC_SSM'
};
My biggest hurdle is figuring out the proper data structures and algorithms to accomplish this task.
I figured a depth first search of a n-ary tree would give me what I want.
Here is my final solution:
#!/usr/local/bin/perl
# !/usr/bin/perl
use Data::Dumper; #!!!
sub Create_Tree;
sub Treverse;
#for my $node (#TREE) {
# print_node($node[0], 1);
#}
#Main
our #TREE;
Create_Tree("call_tree.small_01.txt");
my $str = Dumper #TREE;
$str =~ s/^(\s+)/' 'x(length($1)>>2)/meg;
#print "\n\n=======================================\n$str"; #!!!
#print "\n\n=======================================\n" . (Dumper #TREE); #!!!
#print "Arr = #TREE, SZ = $#TREE\n\n";
Treverse(\#TREE,1);
sub Create_Tree
{
my ($call_tree) = #_;
my #stack;
my ($old_depth, $p_arr) = (0, \#TREE);
open(IN, "< $call_tree" ) or die "Can not open '$call_tree' for input.\n";
for (<IN>)
{
if (m/^(\s*)(\S+)\s*=>\s*(\S+):(\d+)/ or m/^(\s*)(\S+)()()/)
{
my ($depth, $callee_fn, $caller_fn, $ln, $diff) = ($1, $2, $3, $4, 0);
$depth = int(length($depth) / 6);
$diff = $depth - $old_depth;
if ($diff == 1)
{
push #stack, $p_arr;
$p_arr = \#{$$p_arr[$#{$p_arr}]{children}};
}
elsif ($diff < 0)
{
$p_arr = pop #stack while ++$diff <= 0;
}
elsif ($diff > 1)
{
die "Incorrectly formated call tree:\n $_\n";
}
push #$p_arr, {
caller => $caller_fn,
called_by => $callee_fn,
at_line => $ln
};
$old_depth = $depth;
}
}
close IN;
}
exit;
OUTPUT look like this:
......file1
............file1 101:A
..................XXX.AA 102:AA
........................XXX.AAA 103:AAA
........................XXX.AAB 104:AAB
..............................XXX.AABA 105:AABA
..................XXX.AB 106:AB
........................XXX.ABA 107:ABA
............file1 108:B
..................XXX.BA 109:BA
........................XXX.BAA 110:BAA
........................XXX.BAB 111:BAB
From this call_tree.txt file:
file1
A => file1:101
AA => XXX.AA:102
AAA => XXX.AAA:103
AAB => XXX.AAB:104
AABA => XXX.AABA:105
AB => XXX.AB:106
ABA => XXX.ABA:107
B => file1:108
BA => XXX.BA:109
BAA => XXX.BAA:110
BAB => XXX.BAB:111
Using this subroutine:
sub Treverse
{
my ($p_arr, $level) = #_;
for (my $ind=0; $ind<=$#{$p_arr}; $ind++)
{
print "." x ($level*6);
if ($$p_arr[$ind]{'caller'} ne "") {print "$$p_arr[$ind]{'caller'}" . " " x 4;}
if ($$p_arr[$ind]{'at_line'} ne "") {print "$$p_arr[$ind]{'at_line' }" . ":";}
if ($$p_arr[$ind]{'called_by'} ne "") {print "$$p_arr[$ind]{'called_by'}" . "\n";}
Treverse(\#{$$p_arr[$ind]{children}}, $level +1) if defined $$p_arr[$ind]{children};
}
}
# END of Treverse
Here is how to print the structure you built using Wes's answer.
Once processed the data, you end up with something like this:
my #nodes = (
{ name => 'ARINC_TEST', file => 'powrup.asm', line => 633,
children => [
{ name => 'ARINC_LEVEL_TEST', file => 'artest.asm', line => 178,
children => [
{ name => 'AD_READ', file => 'arltst.asm', line => 204 },
{ name => 'AD_READ', file => 'arltst.asm', line => 250 },
{ name => 'AD_READ', file => 'arltst.asm', line => 300 },
{ name => 'AD_READ', file => 'arltst.asm', line => 346 },
{ name => 'AD_READ', file => 'arltst.asm', line => 396 },
{ name => 'AD_READ', file => 'arltst.asm', line => 442 },
],
},
{ name => 'ARINC_READ', file => 'artest.asm', line => 209,
children => [],
},
{ name => 'ARINC_WORD_TXRX_TEST', file => 'artest.asm', line => 221,
children => [
{ name => 'ARINC_OUT', file => 'artxrx.asm', line => 207 },
{ name => 'ARINC_READ', file => 'artxrx.asm', line => 221 },
],
}
]
}
);
The structure is recursive and children key point to arrayref of another hash. To print this out, you need recursive code:
for my $node (#nodes) {
print_node($node, 1);
}
sub print_node {
my ($node, $level) = #_;
# the node itself
print "." x ($level*6)
, $node->{name}, " " x 4
, $node->{file}, ":"
, $node->{line}, "\n";
# recurse for children
if(defined $node->{children}) {
for my $child (#{ $node->{children} }) {
print_node($child, $level + 1);
}
}
}
For data above, the code outputs
......ARINC_TEST powrup.asm:633
............ARINC_LEVEL_TEST artest.asm:178
..................AD_READ arltst.asm:204
..................AD_READ arltst.asm:250
..................AD_READ arltst.asm:300
..................AD_READ arltst.asm:346
..................AD_READ arltst.asm:396
..................AD_READ arltst.asm:442
............ARINC_READ artest.asm:209
............ARINC_WORD_TXRX_TEST artest.asm:221
..................ARINC_OUT artxrx.asm:207
..................ARINC_READ artxrx.asm:221
For data structures, one of the biggest powers of perl is arbitrary nesting of structures. Thus, rather than have a single variable that contains all the data for all the notes you can have "subnodes" inside their parents.
Lets say you had a hash for one entry:
%node1 = (
name => 'ALPHA_MAX_MC_SSM',
file => 'arltst.asm',
line => 42
);
The above code will create a nice simply node to store data in. But you can actually store more data inside that one itself. A "child node":
%node2 = (
name => 'ACTUAL_ALT',
file => 'foo.asm',
line => 2001
);
$node1{children}[0] = \%node2;
Then you have a child node ('children') in the first node that is an array of all it's children. You can access date in the child directly like:
$node1{'children'}[0]{'name'};
To understand this and how it works you need to read up on perl references and the perl data types. It takes a bit as a new perl programmer to get the concepts, but once you get it you can do amazingly powerful quick programs snarfing up complex hierarchical data and processing it.