I want to loop post requests for data from a csv.
csv file (has 2 columns )=
agentLicenseID(x) licenseExpirationDate(y)
271844 6/20/2021
271847 6/30/2021
271848 5/21/2021
body = {'sid':API_SID,'key':API_KEY, 'agentLicenseID':x,'licenseExpirationDate':y }
response = requests.post(url=UPD_URL,data=body)
I intend on looping the response for different values of x and y ( agentLicenseID and licenseExpirationDate) from the csv file
With help of pandas:
import pandas as pd
df = pd.read_csv("your_file.csv", sep=r"\s+") # <-- change the separator if it's different
for x, y in zip(df["agentLicenseID"], df["licenseExpirationDate"]):
body = {
"sid": API_SID,
"key": API_KEY,
"agentLicenseID": x,
"licenseExpirationDate": y,
}
response = requests.post(url=UPD_URL, data=body)
# ...
Related
I am trying to store dictionaries in a JSON file which user input. The code is:
#client.command()
async def shibaku0(ctx, coin1, coin2, coin3, coin4, coin5, coin6, shibakunumber, oslink):
await ctx.message.delete()
with open('Shibaku0.json', 'r') as f:
coins_data = json.load(f)
coins_data[str(ctx.author.id)]["coins"] = (coin1, coin2, coin3, coin4, coin5, coin6)
shibakunumber[str(ctx.author.id)]["shibakunumber"] = (shibakunumber)
oslink[str(ctx.author.id)]["oslink"] = (oslink)
with open('Shibaku0.json', 'w') as f:
json.dump(coins_data, f)
embed=discord.Embed(title="Shibaku0", url=f'{oslink}', description=f'No. {shibakunumber}')
embed.add_field(name="Coins: ", value=f'{coin1} {coin2} {coin3} {coin4} {coin5} {coin6}', inline=True)
embed.set_footer(text=f"{ctx.author.name}'s Shibaku")
await ctx.send(embed=embed)
I want to store the "coins", "shibakunumber, "oslink", but I am getting this error message when I try and run the code:
TypeError: list indices must be integers or slices, not str
To make a json that is an associative array (JSON object / dict). You need to have { or a single object within the file. Each object associates userid with the data for that user, i.e, a nested object.
Here's a sample :
import json
# sample_data/Obj.json
# {
# "user1": {"coins": "coinstringvalue", "shibakunumber": 1, "oslink": "linkstringvalue"},
# "user2": {"coins": "coinstringvalue2", "shibakunumber": 2, "oslink": "linkstringvalue2"}
# }
mydata = json.load(open('sample_data/Obj.json', 'r'))
print(mydata['user1'])
print(mydata['user2'])
print(mydata['user2']['shibakunumber'])
This gives the output:
{'coins': 'coinstringvalue', 'shibakunumber': 1, 'oslink': 'linkstringvalue'}
{'coins': 'coinstringvalue2', 'shibakunumber': 2, 'oslink': 'linkstringvalue2'}
2
Note that the mydata is a dict type in python.
Here's an example of starting with an empty json file.
import json
# sample_data/empty.json
# {}
mydata = json.load(open('sample_data/empty.json', 'r'))
print(mydata)
# author_id = str(ctx.author.id)
author_id = str(1161214)
if author_id not in mydata:
mydata[author_id] = dict()
mydata[author_id]['coins'] = ("coin1", "coin2")
mydata[author_id]['shibakunumber'] = 2
mydata[author_id]['oslink'] = "somelink"
json.dump(mydata, open('sample_data/new_file.json', 'w'))
loaded_data = json.load(open('sample_data/new_file.json', 'r'))
print(loaded_data)
Produces:
mydata {}
loaded_data {'1161214': {'coins': ['coin1', 'coin2'], 'shibakunumber': 2, 'oslink': 'somelink'}}
#i have scraped data below is my code, now i want to add a column of symbols to the respective company data, plz guide me how the symbol can be added to the respective firm data
#code below
from time import sleep
import pandas as pd
import os
import numpy as np
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
browser = webdriver.Chrome(ChromeDriverManager().install())
symbols =['FATIMA',
'SSGC',
'FCCL',
'ISL',
'KEL',
'NCL',
'DGKC',
'SNGP',
'NML',
'ENGRO',
'HUMNL',
'CHCC',
'ATRL',
'HUBC',
'ASTL',
'PIBTL',
'OGDC',
'EFERT',
'FFC',
'NCPL',
'KTML',
'PSO',
'LUCK',
'SEARL',
'KOHC',
'ABOT',
'AICL',
'HASCOL',
'PTC',
'KAPCO',
'PIOC',
'POL',
'SHEL',
'GHGL',
'HCAR',
'DCR',
'BWCL',
'MTL',
'GLAXO',
'PKGS',
'SHFA','MARI',
'ICI',
'ACPL',
'PSMC',
'SPWL',
'THALL',
'BNWM',
'EFUG',
'GADT',
'AABS']
company = 1
for ThisSymbol in symbols :
# Get first symbol from the above python list
company = 2
# In the URL, make symbol as variable
url = 'http://www.scstrade.com/stockscreening/SS_CompanySnapShotYF.aspx?symbol=' + ThisSymbol
browser.get(url)
sleep(2)
# The below command will get all the contents from the url
html = browser.execute_script("return document.documentElement.outerHTML")
# So we will supply the contents to beautiful soup and we tell to consider this text as a html, with the following command
soup = BeautifulSoup (html, "html.parser")
for rn in range(0,9) :
plist = []
r = soup.find_all('tr')[rn]
# Condition: if first row, then th, otherwise td
if (rn==0) :
celltag = 'th'
else :
celltag = 'td'
# Now use the celltag instead of using fixed td or th
col = r.find_all(celltag)
print()
if col[i] == 0:
print ("")
else:
for i in range(0,4) :
cell = col[i].text
clean = cell.replace('\xa0 ', '')
clean = clean.replace (' ', '')
plist.append(clean)
# If first row, create df, otherwise add to it
if (rn == 0) :
df = pd.DataFrame(plist)
else :
df2 = pd.DataFrame(plist)
colname = 'y' + str(2019-rn)
df[colname] = df2
if (company == 1):
dft = df.T
# Get header Column
head = dft.iloc[0]
# Exclude first row from the data
dft = dft[1:]
dft.columns = head
dft = dft.reset_index()
# Assign Headers
dft = dft.drop(['index'], axis = 'columns')
else:
dft2 = df.T
# Get header Column
head = dft2.iloc[0]
# Exclude first row from the data
dft2 = dft2[1:]
dft2.columns = head
dft2 = dft2.reset_index()
# Assign Headers
dft2 = dft2.drop(['index'], axis = 'columns')
dft['Symbol'] = ThisSymbol
dft = dft.append(dft2, sort=['Year','Symbol'])
company = company +1
dft
my output looks this, i want to have a symbol column to each respective firm data
Symbol,i have added
dft['Symbol'] = ThisSymbol
but it add just first company from the list to all companies data
enter image description here
I am trying to send image through API in Micropython. still no solution how to do it. please help
import urequests
import json
URL = 'https://example.com/test'
datas = json.dumps({"auth_key": "43435", "mac": "abcd", "name": "washid"})
filep = 'OBJ.jpg'
filess = {'odimg': open(filep, 'rb')}
try:
response = urequests.post(URL,data=datas,files=files)
print(response.json())
except Exception as e:
print(e)
Maybe this template can help you :
import ubinascii
import uos
import urequests
def make_request(data, image=None):
boundary = ubinascii.hexlify(uos.urandom(16)).decode('ascii')
def encode_field(field_name):
return (
b'--%s' % boundary,
b'Content-Disposition: form-data; name="%s"' % field_name,
b'',
b'%s'% data[field_name]
)
def encode_file(field_name):
filename = 'latest.jpeg'
return (
b'--%s' % boundary,
b'Content-Disposition: form-data; name="%s"; filename="%s"' % (
field_name, filename),
b'',
image
)
lines = []
for name in data:
lines.extend(encode_field(name))
if image:
lines.extend(encode_file('file'))
lines.extend((b'--%s--' % boundary, b''))
body = b'\r\n'.join(lines)
headers = {
'content-type': 'multipart/form-data; boundary=' + boundary,
'content-length': str(len(body))}
return body, headers
def upload_image(url, headers, data):
http_response = urequests.post(
url,
headers=headers,
data=data
)
if http_response.status_code == 204:
print('Uploaded request')
else:
raise UploadError(http_response)
http_response.close()
return http_response
You need to declare an header for your request
I used Jonathan's answer. Had to modify the code a bit (-thanks to Akshay to figure this out). A fixed boundary is used instead of generating a new one every time. Also, there needs to be an additional \r\n at the end of the file. I have used this to upload photos to Telegram Bot using ESP-32 CAM.
def make_request(data, image=None):
boundary = '---011000010111000001101001'
#boundary fixed instead of generating new one everytime
def encode_field(field_name): # prepares lines that include chat_id
return (
b'--%s' % boundary,
b'Content-Disposition: form-data; name="%s"' % field_name,
b'',
b'%s'% data[field_name] #field_name conatains chat_id
)
def encode_file(field_name): # prepares lines for the file
filename = 'latest.jpg' # dummy name is assigned to uploaded file
return (
b'--%s' % boundary,
b'Content-Disposition: form-data; name="%s"; filename="%s"' % (
field_name, filename),
b'',
image
)
lines = [] # empty array initiated
for name in data:
lines.extend(encode_field(name)) # adding lines (data)
if image:
lines.extend(encode_file('photo')) # adding lines image
lines.extend((b'--%s--' % boundary, b'')) # ending with boundary
body = b'\r\n'.join(lines) # joining all lines constitues body
body = body + b'\r\n' # extra addtion at the end of file
headers = {
'content-type': 'multipart/form-data; boundary=' + boundary
} # removed content length parameter
return body, headers # body contains the assembled upload package
def upload_image(url, headers, data):
http_response = urequests.post(
url,
headers=headers,
data=data
)
print(http_response.status_code) # response status code is the output for request made
if (http_response.status_code == 204 or http_response.status_code == 200):
print('Uploaded request')
else:
print('cant upload')
#raise UploadError(http_response) line commneted out
http_response.close()
return http_response
# funtion below is used to set up the file / photo to upload
def send_my_photo(photo_pathstring): # path and filename combined
token = 'authentication token or other data' # this my bot token
chat_id= 999999999 # my chat_id
url = 'https://api.telegram.org/bot' + token
path = photo_pathstring # this is the local path
myphoto = open(path , 'rb') #myphoto is the photo to send
myphoto_data = myphoto.read() # generate file in bytes
data = { 'chat_id' : 999999999 }
body, headers = make_request(data, myphoto_data) # generate body to upload
url = url + '/sendPhoto'
headers = { 'content-type': "multipart/form-data; boundary=---011000010111000001101001" }
upload_image(url, headers, body) # using function to upload to telegram
Spark Version: '2.0.0.2.5.0.0-1245'
So, my original question changed a bit but it's still the same issue.
What I want to do is load a huge amount of JSON files and transform those to a DataFrame - also probably save them as CSV or parquet file for further processing. Each JSON file represents one row in the final DataFrame.
import os
import glob
HDFS_MOUNT = # ...
DATA_SET_BASE = # ...
schema = StructType([
StructField("documentId", StringType(), True),
StructField("group", StringType(), True),
StructField("text", StringType(), True)
])
# Get the file paths
file_paths = glob.glob(os.path.join(HDFS_MOUNT, DATA_SET_BASE, '**/*.json'))
file_paths = [f.replace(HDFS_MOUNT + '/', '') for f in file_paths]
print('Found {:d} files'.format(len(file_paths))) # 676 files
sql = SQLContext(sc)
df = sql.read.json(file_paths, schema=schema)
print('Loaded {:d} rows'.format(df.count())) # 9660 rows (what !?)
Besides the fact that there are 9660 rows instead of 676 (number of available files) I also have the problem that the content seems to be None:
df.head(2)[0].asDict()
gives
{
'documentId': None,
'group': None,
'text': None,
}
Example Data
This is just fake data of course but it resembles the actual data.
Note: Some fields may be missing e.g. text must not always be present.
a.json
{
"documentId" : "001",
"group" : "A",
"category" : "indexed_document",
"linkIDs": ["adiojer", "asdi555", "1337"]
}
b.json
{
"documentId" : "002",
"group" : "B",
"category" : "indexed_document",
"linkIDs": ["linkId", "1000"],
"text": "This is the text of this document"
}
assuming that all your files has the same structure and are in the same directory:
df = sql_cntx.read.json('/hdfs/path/to/folder/*.json')
There might be a problem if any of the columns has Null values for all rows. Then spark will not be able to determine schema, so you have an option to tell spark which schema to use:
from pyspark import SparkContext, SQLContext
from pyspark.sql.types import StructType, StructField, StringType, LongType
sc = SparkContext(appName="My app")
sql_cntx = SQLContext(sc)
schema = StructType([
StructField("field1", StringType(), True),
StructField("field2", LongType(), True)
])
df = sql_cntx.read.json('/hdfs/path/to/folder/*.json', schema=schema)
UPD:
in case if file has multirows formatted json you can try this code:
sc = SparkContext(appName='Test')
sql_context = SQLContext(sc)
rdd = sc.wholeTextFiles('/tmp/test/*.json').values()
df = sql_context.read.json(rdd, schema=schema)
df.show()
Example I have my JDBC Request and the response is like:
X Y Z
a1 b1 c1
a2 b2 c2
a3 b3 c3
a4 b4 c4
a5 b5 c5
. . .
. . .
. . .
How can I get all the value of x, y and z?
then I have HTTP request and I'm going to assert if all the response is matched to the data selected from JDBC.
example response:
{
{
"x":"a1",
"y":"b1",
"z": "c1"
},
{
"x":"a2",
"y":"b2",
"z": "c2"
},
{
"x":"a3",
"y":"b3",
"z": "c4"
},
{
"x":"a4",
"y":"b4",
"z": "c4"
},
{
"x":"a5",
"y":"b5",
"z": "c5"
},
{
"x":"a6",
"y":"b6",
"z": "c6"
},
{
"x":"a7",
"y":"b7",
"z": "c7"
},
{
"x":"a8",
"y":"b8",
"z": "c8"
},
.
.
.
.
}
As per JDBC Request sampler documentation:
If the Variable Names list is provided, then for each row returned by a Select statement, the variables are set up with the value of the corresponding column (if a variable name is provided), and the count of rows is also set up. For example, if the Select statement returns 2 rows of 3 columns, and the variable list is A,,C, then the following variables will be set up:
A_#=2 (number of rows)
A_1=column 1, row 1
A_2=column 1, row 2
C_#=2 (number of rows)
C_1=column 3, row 1
C_2=column 3, row 2
So given you provide "Variable Names" as X,Y,Z you should be able to access the values as ${X_1}, ${Y_2}, etc.
See Debugging JDBC Sampler Results in JMeter for more detailed information on working with JDBC Test Elements results and result sets.
You should declare the "Variable Names" field and also declare a result variable name as shown below.
Then you can access them using the _1 _2 method. Please find below the sample code that you can use in the beanshell post processor.
import java.util.ArrayList;
import net.minidev.json.parser.JSONParser;
import net.minidev.json.JSONObject;
import net.minidev.json.JSONArray;
ArrayList items = vars.getObject("result1");
for ( int i = items.size() - 1; i >= 0; i--) {
JSONObject jsonitemElement = new JSONObject();
jsonitemElement.put("x", vars.get("x_" + (i + 1)));
jsonitemElement.put("y", vars.get("y_" + (i + 1)));
jsonitemElement.put("z", vars.get("z_" + (i + 1)));
log.info(jsonitemElement.toString());
}
Since you are getting these values as the response from the response payload of the HTTP request, you should add the code to parse that JSON response in an assertion or post processor and compare it with the elements from the above sample code.
A point to note - Different applications send the target JSON in any order. So, there is no guarantee that the HTTP response will always send the response as A1,B1,C1 - A2,B2,C2 etc. It can send them in any order starting with A5,B5,C5 etc. It is better to then use a hashmap or write your array comparison to ensure that your result set completely matches the HTTp response.