I try to train ResNet 50 for Image classification using this code: https://github.com/mlperf/training/blob/master/image_classification/tensorflow/official/resnet/imagenet_main.py, but in process i have error in line 322 seed = int(argv[1]) IndexError: list index out of range. How can I fix this error? I am using Tenserflow 2.0.
Your error message IndexError: list index out of range suggests that the list argv has length < 2 (since index 1 is already out of bounds).
sys.argv contains the list of argument values that you pass on the command line. The first element, argv[0] is always the name of the script you invoke.
In your case this means that you did not pass any arguments.
So looking at the code, it tries to read - as the first command line option - a seed that it can use to initialise various random number generators (RNG) with.
So what you want to do is invoke your script with something like:
python imagenet_main.py r4nd0ms33d
where r4nd0ms33d is some random value.
Sometimes explicitly seeding the RNG with a fixed value can be useful, e.g. when you want your training results to be reproducable.
You can play around with this simple Python program:
import sys
if __name__ == '__main__':
print(sys.argv)
Save it into test.py and see what it prints if you run it as python test.py versus python test.py random versus python test.py random seed
Related
When I create a list from values in a CSV file, I am able to print out the contents of the list using the built in print function, but when I try to enter the elements of the list into an input box using pyautogui,only the first element of the list is printed.
Here is the code which is not working with screenshot of error attached.
Idle Error Message
import csv, pyautogui, time
file = open('serials.csv', 'r')
serial_list = list(csv.reader(file))
file.close
print(serial_list)
time.sleep(5)
i = 0
for serial in serial_list:
pyautogui.typewrite(serial_list[i][i])
i +=1
I was able to get the desired results using the Python code below which I generated using VBA, but would like to learn how to do this properly using Python.
import pyautogui, time
time.sleep(3)
pyautogui.write("2300-xxxx1")
pyautogui.press('tab')
pyautogui.write("2300-xxxx2")
pyautogui.press('tab')
pyautogui.write("2300-xxxx3")
pyautogui.press('tab')
pyautogui.write("2300-xxxx4")
pyautogui.press('tab')
pyautogui.write("2300-xxxx5")
yo, I think issue is in this line of code:
pyautogui.typewrite(serial_list[i][i])
Here, you trying to access a specific element of the 2D list serial_list using serial_list[i][i] where i is the index variable. However, this will only give you the first element of the list.
You should use the variable serial that you defined in the for loop to access the current element of the list:
pyautogui.typewrite(serial)
Also, since the serial_list is a 2D list of strings, you want to convert the elements of list to strings before passing it to the pyautogui.typewrite() function like this:
for serial in serial_list:
pyautogui.typewrite(str(serial))
Also, you should call the file.close method with paranthesis file.close() to close the file properly.
Also, you should use the pyautogui.press('tab') function after each pyautogui.typewrite() function call to move to the next input box.
So the final code will look like:
import csv, pyautogui, time
file = open('serials.csv', 'r')
serial_list = list(csv.reader(file))
file.close()
print(serial_list)
time.sleep(5)
for serial in serial_list:
pyautogui.typewrite(str(serial))
pyautogui.press('tab')
Now it will work.
why? cuz 🅱️
I have prepared a script for Pymol that works well in evaluating RMSD values for a list of residues of proteins of interest (targetted residues are generated by script embedded commands). However, I wished to implement the script to allow the user to select the analysed residues. I have tried to use the input function, but it fails to work. Since the code is a bit long, I simplified progressively the scripts. In that way, I ended up with the following two scripts:
"test.py":
import Bio.PDB
import numpy as np
from pymol import cmd
import os,glob
from LFGA_functions import user_entered
List_tot = user_entered()
print (List_tot)
which calls the simple "user_entered" function from inside the "My_function.py" script:
def user_entered():
List_res = []
List_tot = []
a = input("Please indicate the number of residues to analyze/per monomer:")
numberRes =int(a)
for i in range(numberRes):
Res = input("Please provide each residue NUMBER and hit ENTER:")
Res1 = int(Res)
Res2 = Res1+2000
Res3 = Res1+3000
Res4 = Res1+4000
Res5 = Res1+5000
Res6 = Res1+6000
List_res = (str(Res1),str(Res2),str(Res3),str(Res4),str(Res5),str(Res6))
List_tot.append(List_res)
return List_tot
The script "test.py" works well when executed by Python (3.7.5, which is installed with Pymol 2.3.4,under Windows 7 Prof) from a windows command line. An example of result:
[first input to indicate that 2 cases will be treated, followed by the identification number for each case][1]
However, when the script is run from Pymol GUI, I get the following error message: input():lost sys.stdin
Does anybody know what is the problem. Obviously, I am a very primitive Python user...
I profit also to ask something related to this problem... in the original script that works well in Pymol, before trying to implement the "input" option, I store transiently some data (residue name, type and chain) in the form of a "set". I need a set, and not a list, because it removes automatically data repeatitions. However, when I try to run it from PYthon (thinking of overcoming the abovementioned problem of input), it stops with a message: name 'close_to_A' not defined. However, it is as shown in the portion of code shown here below, and indeed works in Pymol.
...
# Step 3:
# listing of residues sufficiently close to chainA
# that will be considered in calculation of RMSD during trajectory
cmd.load("%s/%s" %(path3,"Average.pdb"), "Average", quiet=0)
cmd.select("around_A", "Average and chain B+C near_to 5 of chain A")
cmd.select("in_A", "Average and chain A near_to 5 of chain B+C")
close_to_A = set()
cmd.iterate("(around_A)","close_to_A.add((chain,resi,resn))")
cmd.iterate("(in_A)","close_to_A.add((chain,resi,resn))")
cmd.delete("around_A")
cmd.delete("in_A")
...
How shall I define a set in Python 3 ? Why does it work when run from Pymol ?
I would really appreciate if you could help me to solve these two problems. Thank you in advance,
Best regards,
Luis
I'm doing a check in my syntax to see if all of the string fields have values. This looks something like this:
IF(STRING ~= "").
Now, instead of filtering or computing a variable, I would like to force an error if there are fields with no values. I'm running a lot of these scripts and I don't want to keep checking the datasets manually. Instead, I just want to receive an error and stop execution.
Is this possible in SPSS Syntax?
First, you can efficiently count the blanks in your string variables with COUNT:
COUNT blanks = A B C(" ").
where you list the string variables to be checked. So if the sum of blanks is > 0, you want the error to be raised. First aggregate to a new dataset and activate it:
DATASET DECLARE sum.
AGGREGATE /OUTFILE=sum /count=sum(blanks).
The hard part is getting the job to stop when the blank count is greater than 0. You can put the job in a syntax file and run it using INSERT with ERROR=STOP, or you can run it as a production job via Utilities or via the -production flag on the command line of an spj (production facility) job.
Here is how to generate an error on a condition.
DATASET ACTIVATE sum.
begin program.
import spssdata
curs = spssdata.Spssdata()
count = curs.fetchone()
curs.CClose()
if count[0] > 0:
spss.Submit("blank values exist")
end program.
DATASET CLOSE sum.
This code reads the aggregate file and if the blank count is positive issues an invalid command causing an error, which stops the job.
I have written a Python 2.7 script that reads a CSV file and then does some standard deviation calculations . It works absolutely fine however it is very very slow. A CSV I tried with 100 million lines took around 28 hours to complete. I did some googling and it appears that maybe using the pandas module might makes this quicker .
I have posted part of the code below, since i am a pretty novice when it comes to python , i am unsure if using pandas would actually help at all and if it did would the function need to be completely re-written.
Just some context for the CSV file, it has 3 columns, first column is an IP address, second is a url and the third is a timestamp.
def parseCsvToDict(filepath):
with open(csv_file_path) as f:
ip_dict = dict()
csv_data = csv.reader(f)
f.next() # skip header line
for row in csv_data:
if len(row) == 3: #Some lines in the csv have more/less than the 3 fields they should have so this is a cheat to get the script working ignoring an wrong data
current_ip, URI, current_timestamp = row
epoch_time = convert_time(current_timestamp) # convert each time to epoch
if current_ip not in ip_dict.keys():
ip_dict[current_ip] = dict()
if URI not in ip_dict[current_ip].keys():
ip_dict[current_ip][URI] = list()
ip_dict[current_ip][URI].append(epoch_time)
return(ip_dict)
Once the above function has finished the data is parsed to another function that calculates the standard deviation for each IP/URL pair (using numpy.std).
Do you think that using pandas may increase the speed and would it require a complete rewrite or is it easy to modify the above code?
The following should work:
import pandas as pd
colnames = ["current_IP", "URI", "current_timestamp", "dummy"]
df = pd.read_csv(filepath, names=colnames)
# Remove incomplete and redundant rows:
df = df[~df.current_timestamp.isnull() & df.dummy.isnull()]
Notice this assumes you have enough RAM. In your code, you are already assuming you have enough memory for the dictionary, but the latter may be significatively smaller than the memory used by the above, for two reasons.
If it is because most lines are dropped, then just parse the csv by chunks: arguments skiprows and nrows are your friends, and then pd.concat
If it is because IPs/URLs are repeated, then you will want to transform IPs and URLs from normal columns to indices: parse by chunks as above, and on each chunk do
indexed = df.set_index(["current_IP", "URI"]).sort_index()
I expect this will indeed give you a performance boost.
EDIT: ... including a performance boost to the calculation of the standard deviation (hint: df.groupby())
I will not be able to give you an exact solution, but here are a couple of ideas.
Based on your data, you read 100000000. / 28 / 60 / 60 approximately 1000 lines per second. Not really slow, but I believe that just reading such a big file can cause a problem.
So take a look at this performance comparison of how to read a huge file. Basically a guy suggests that doing this:
file = open("sample.txt")
while 1:
lines = file.readlines(100000)
if not lines:
break
for line in lines:
pass # do something
can give you like 3x read boost. I also suggest you to try defaultdict instead of your if k in dict create [] otherwise append.
And last, not related to python: working in data-analysis, I have found an amazing tool for working with csv/json. It is csvkit, which allows to manipulate csv data with ease.
In addition to what Salvador Dali said in his answer: If you want to keep as much of the current code of your script, you may find that PyPy can speed up your program:
“If you want your code to run faster, you should probably just use PyPy.” — Guido van Rossum (creator of Python)
I know that there are various ways to get random numbers, eg, from the shell. However, I'm running vim on an android phone with very little compiled in. Also, it does not have to be rigorously random. The point is, what's an interesting, or concise, or fast (that is, with vim native functions), or short way to get a sequence of reasonably good random numbers in Vim?
Try something like
function Rand()
return str2nr(matchstr(reltimestr(reltime()), '\v\.#<=\d+')[1:])
endfunction
. I know no better option then using some of the time functions (there are two of them: reltime() and localtime(), but the latter is updated only each second). I would prefer to either avoid random numbers or use pyeval('random.randint(1, 10)') (preceded by python import random), because shell is slow and I don’t trust time-based solutions.
Note: documentation says that format of the item returned by reltime() depends on the system, thus I am using reltimestr(), not doing something with reltime()[1] which looks like if it contains nanoseconds.
I've recently played around with random numbers in Vim script myself. Here are some resources that I found in the process.
No Vim script
By all means, use an external random number generator if you can. As a rule, they are better and faster than anything that could be done in Vim script.
For example, try
:python import random; print random.randrange(1, 7)
:echo system('echo $RANDOM')
another scripting language, for example Ruby
Libraries
Vim script libraries. These hopefully strive to provide decent quality RNG implementations.
vital.vim is an excellent and comprehensive library created by the vim-jp user group. Their random number generator sports an impressive array of functionality and is the best pure Vim script RNG I know of. vital.vim uses an Xorshift algorithm. Check it out!
Rolling a die with vital.vim:
let Random = vital#of('vital').import('Random')
echo Random.range(1, 7)
vim-rng is a small random number generator plugin. It exports a couple of global functions that rely on a multiply-with-carry algorithm. This project seems to be a work in progress.
Rolling a die with rng:
echo RandomNumber(1, 6)
magnum.vim is my own little big integer library. I've recently added a random number generator that generates integers of any size. It uses the XORSHIFT-ADD algorithm.
Rolling a die with magnum.vim:
let six = magnum#Int(6)
echo magnum#random#NextInt(six).Add(magnum#ONE).Number()
Rndm has been around for much longer than the other libraries. Its functionality is exposed as a couple of global functions. Rolling a die with Rndm:
echo Urndm(1, 6)
Discussion and snippets
Finally, a few links to insightful discussion and Vim script snippets.
ZyX's reltime snippet on this page.
loreb's vimprng project on GitHub has an impressive number of RNG implementations in Vim script. Very useful.
This old mailing list discussion has a couple of Vim script snippets. The first one given by Bee-9 is limited to 16 bit but I found it quite effective. Here it is:
let g:rnd = localtime() % 0x10000
function! Random(n) abort
let g:rnd = (g:rnd * 31421 + 6927) % 0x10000
return g:rnd * a:n / 0x10000
endfunction
Another script, found in a person named Bart's personal config files.
Episode 57 on Vimcasts.org discusses Vim's 'expression register' and refers to random number examples throughout. Refers to this Stackoverflow question and ZyX's snippet. Recommended.
The Vim wiki on wikia has an article 'Jump to a random line' that has a few resources not mentioned yet.
Based on others' answers and other resources from the internet, I have written
two functions to generate a random integer in the given range [Low, High].
Both the two functions receive two arguments: Low and High and return a
random number in this range.
Combine Python and Vim script
The first function combines Python and Vim script.
" generate a random integer from range [Low, High] using Python
function! RandInt(Low, High) abort
" if you use Python 3, the python block should start with `python3` instead of
" `python`, see https://github.com/neovim/neovim/issues/9927
python3 << EOF
import vim
import random
# using vim.eval to import variable outside Python script to python
idx = random.randint(int(vim.eval('a:Low')), int(vim.eval('a:High')))
# using vim.command to export variable inside Python script to vim script so
# we can return its value in vim script
vim.command("let index = {}".format(idx))
EOF
return index
endfunction
Pure Vim script
The second function I propose uses pure vim script:
function! RandInt(Low, High) abort
let l:milisec = str2nr(matchstr(reltimestr(reltime()), '\v\.\zs\d+'))
return l:milisec % (a:High - a:Low + 1) + a:Low
endfunction
Use luaeval() (Neovim only)
The third way to generate random number is to use lua via luaeval().
" math.randomseed() is need to make the random() function generate different numbers
" on each use. Otherwise, the first number it generate seems same all the time.
luaeval('math.randomseed(os.time())')
let num = luaeval('math.random(1, 10)')
If you want to generate random number in non-serious occasions, you may use the
these methods as a starter.