Under a database, I am currently creating several tables and inserting several cvs files into these tables.
While I succeeded in inserting some CVS files, some files I failed.
The code I failed is as below
import sys
sys.path.append('/Users/leigh/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages')
import mysql.connector
import config
import csv
import os
dbcon = mysql.connector.connect(database=config.db, user=config.user, password=config.passwd, host=config.host)
dbcur = dbcon.cursor()
dbcur.execute('CREATE TABLE parking (Place_ID int, Parking varchar(50))')
with open('/Users/leigh/documents/RCdata/chefmozparking.csv') as csvfile:
reader = csv.reader(csvfile)
next(reader, None)
for row in reader:
dbcur.execute('INSERT INTO parking (Place_ID, Parking) VALUES(%s, "%s")' % tuple(row))
dbcon.commit()
I used the same format (same code function on another datasets with same data structure) on the succeeded ones, but this one just failed with the error written:
dbcur.execute('INSERT INTO parking (Place_ID, Parking) VALUES(%s, "%s")' % tuple(row))
TypeError: not all arguments converted during string formatting
The columns are perfectly matched and the data type cannot be wrong.
I am not very experienced in MySQL, and I have looked through the internet and still cannot find what is wrong here.
Could some one please kindly tell me what is going on?
Thanks!
// If further information is needed, please just let me know.
Related
I am trying to use mutliprocessing Pool.map() to speed up my code. In the function where I have computation occurring for each process I reference an xarray.DataArray that was opened using xarray.open_rasterio(). However, I receive errors similar to this:
rasterio.errors.RasterioIOError: Read or write failed. /net/home_stu/cfite/data/CDL/2019/2019_30m_cdls.img, band 1: IReadBlock failed at X offset 190, Y offset 115: Unable to open external data file: /net/home_stu/cfite/data/CDL/2019/
I assume this is some issue related to the same file being referenced simultaneously while another worker is opening it too? I use DataArray.sel() to select small portions of the raster grid that I work with since the entire .img file is way to big to load all at once. I have tried opening the .img file in the main code and then just referencing to it in my function, and I've tried opening/closing it in the function that is being passed to Pool.map() - and receive errors like this regardless. Is my file corrupted, or will I just not be able to work with this file using multiprocessing Pool? I am very new to working with multiprocessing, so any advice is appreciated. Here is an example of my code:
import pandas as pd
import xarray as xr
import numpy as np
from multiprocessing import Pool
def select_grid(x,y):
ds = xr.open_rasterio('myrasterfile.img') #opening large file with xarray
grid = ds.sel(x=slice(x,x+50), y=slice(y,y+50))
ds.close()
return grid
def myfunction(row):
x = row.x
y = row.y
mygrid = select_grid(x,y)
my_calculation = mygrid.sum() #example calculation, but really I am doing multiple calculations
my_calculation.to_csv('filename.csv')
with Pool(30) as p:
p.map(myfunction, list_of_df_rows)
How can I choose an index dynamically using python. My current code is as follows:
import os
import time
import datetime
from datetime import datetime
import shutil
from elasticsearch import Elasticsearch
from esindex import createIndex
client=Elasticsearch()
I have 1000's of images and the name of the image would be: 1559624525_cb704087042c76bf.jpg. I am splitting the name into two parts, timestamp (1559624525) and machid (cb704087042c76bf) and writing it into an es index.
path="/home/ubuntu/images/"
for root, dirs, files in os.walk(path):
for name in files:
try:
dat=name.split('_')[0]
machid=name.split('_')[1]
machid=machid.split('.')[0]
createIndex(machid) ##this creates index dynamically
dat=int(dat)
dat=datetime.utcfromtimestamp(dat).strftime('%d-%m-%Y %H:%M:%S')
dte=dat.split(' ')[0]
id=id+1
doc = {'sequence_id':id,'imagename': name, 'time_stamp': dat, 'date'=dte}
#print(doc)
client.index(index=machid, body=doc) ##this line is not working. If i comment this out, shutil works.
shutil.copy('/home/ubuntu/images/'+ name, '/home/ubuntu/test/')
os.remove('/home/ubuntu/images/' + name)
except Exception:
continue
how can i input a document into elasticsearch by dynamically choosing the index.
print the Exception in except block
Check machid is not empty
Since there is not any errors use Cat APIS to check any indeces created in your ES
curl -X GET "localhost:9200/_cat/indices"
Apparently I figured that my code works. In createIndex function I had a date field, which was set to date, however the above code parses the date as text. To test, when I changed the date to text in createIndex, everything fell in place. Replying this to complete the QA loop.
I want to download multiple xml files from web service API. I have a query that gets a JSON document:
= Json.Document(Web.Contents("http://reports.sem-o.com/api/v1/documents/static-reports?DPuG_ID=BM-086&page_size=100"))
and manipulates it to get list of file names such as: PUB_DailyMeterDataD1_201812041627.xml in a column on an excel spreadsheet.
I hoped to get a function to run against this list of names to get all the data, so first I worked on one file: PUB_DailyMeterDataD1_201812041627
= Xml.Tables(Web.Contents("https://reports.sem-o.com/documents/PUB_DailyMeterDataD1_201812041627.xml"))
This gets an xml table which I manipulate to get the data I want (the half hourly metered MWh for generator GU_401970
Now I want to change the query into a function to automate the process across all xml files avaiable from the service. The function requires a variable to be substituted for the filename. I try this as preparation for the function:
let
Filename="PUB_DailyMeterDataD1_201812041627.xml",
Source = (Web.Contents("https://reports.sem-o.com/documents/Filename")),
(followed by the manipulating Mcode)
This doesnt work.
then this:
let
Filename="PUB_DailyMeterDataD1_201812041627.xml",
Source = Xml.Tables(Web.Contents("https://reports.sem-o.com/documents/[Filename]")),
I get:
DataFormat.Error: Xml processing failed. Either the input is invalid or it isn't supported. (Internal error: Data at the root level is invalid. Line 1, position 1.)
Details:
Binary
So stuck here. Can you help.
thanks
Conor
You append strings with the "&" symbol in Power Query. [Somename] is the format for referencing a field within a table, a normal variable is just referenced with it's name. So in your example
let Filename="PUB_DailyMeterDataD1_201812041627.xml",
Source = Xml.Tables(Web.Contents("https://reports.sem-o.com/documents/" & Filename)),
Would work.
It sounds like you have an existing query that drills down to a list of filenames and you are trying to use that to import them from the url though, so assuming that the column you have gotten the filenames from is called "Filename" then you could add a custom column with this in it
Xml.Tables(Web.Contents("https://reports.sem-o.com/documents/" & [Filename]))
And it will load the table onto the row of each of the filenames.
My SAS session details are:-
ENCODING=LATIN1
LOCALE=EN_US
I am reading encrypted customer names, decrypting them using a java code that has
been implemented in the sas code using javaobj.
data outlib.X1 ( ENCODING = 'asciiany' );
set inlib.INP1 (ENCODING = 'asciiany' ) end=eof;
length decryptedvalue $100;
declare javaobj jObj("<jarname>");
jObj.callstringmethod("<method>", FIRST_NAME , decryptedvalue);
run;
The jObj.callstringmethod returns the decrypted first_name value in the string decryptedvalue. I do a proc export at the end of my SAS code and store all the decrypted names as csv files.
In the last run some of the names have special characters e.g. RÉAL.
This is causing SAS execution to fail with following error :-
ERROR: Transcoding failure at line 30 column 2.
ERROR: DATA STEP Component Object failure. Aborted during the EXECUTION phase.
Is there some way to make SAS session (LATIN1) accept these unicode characters? Can i set
the encoding of the decryptedvalue variable? I dont want to run my entire SAS session in unicode using sas_u8. I only want to accept these characters
Even reading these problematic values as blank is ok.
I have already tried following things :-
Set inencoding= utf8 for the libname
Make ENCODING = 'asciiany' for the input dataset.
Any inputs would be useful.
Thanks in advance!
SAS Tech support suggested this :-
Adding export EASYSOFT_UNICODE=YES in the sasenv_local file .
We are now able to run the entire SAS code (including SQL queries) in the sas u8 session.
Thanks to all for your support.
Still using bloody OpenOffice Writer to customize my sale_order.rml report.
In my sale order I have 6 order lines with 6 different lead time to delivery. I need to show the maximum out of the six values.
After many attempt I have abandoned using the reduce function as it works erratically or not at all most of the time. I have never seen anything like this.
So I thought I'd give a try using max encapsulating a loop such as:
[[ max(repeatIn(so.order_line.delay,'d')) ]]
My maximum lead time being 20, I would expect to see 20 (yes well that would be too easy, wouldn't it!).
It returns
{'d': 20.0}
At least it contains the value I am after.
But; if I try and manipulate this result, it disappears altogether.
I have tried:
int(re.findall(r'[0-9]+', max(repeatIn(so.order_line.delay,'d')))[0])
which works great from the python window, but returns absolutely nothing in OpenERP.
I import the re from my sale_order.py file, which I have recompiled into sale_order.pyo:
import time
import re
from datetime import datetime, timedelta
from report import report_sxw
class order(report_sxw.rml_parse):
def __init__(self, cr, uid, name, context=None):
super(order, self).__init__(cr, uid, name, context=context)
self.localcontext.update({
'time': time,
'datetime': datetime,
'timedelta': timedelta,
're': re,
})
I have of course restarted the server many times. My test install sits on windows.
So can anyone tell me what I am doing wrong, because I can make it work from Python but not from OpenOffice Writer!
Thanks for your help!
EDIT 1:
The format
{'d': 20.0}
is, according to python, a dictionary. Still in Python, to extract the integer from a dictionary it is possible to do it like so:
>>> dict={'d': 20.0}
>>> print(dict['d'])
20.0
But how can I transpose this to OpenERP writer???
I have manage to get the result I wanted by importing functools and declaring the reduce function within the parameters of the sale_order.py file.
I then simply used a combination of reduce and max function and it works exactly as expected.
The correct syntax is as follow:
repeatIn(objects,'o')
reduce(lambda x, y: max(x, y.delay), o.order_line, 0)
Nothing else is required.
Enjoy!