Improve speed for csv.reader/csv.writer in the csv module - performance

Question

I'm doing a pretty simple operation; opening a csv file, deleting the first column, and writing out to a new file. The following code works fine, but it takes 50-60 seconds on my 700 MB file:
import csv
from time import time
#create empty output file
f = open('testnew.csv',"w")
f.close()
t = time()
with open('test.csv',"rt") as source:
rdr= csv.reader( source )
with open('testnew.csv',"a") as result:
wtr= csv.writer( result )
for r in rdr:
del r[0]
_ = wtr.writerow( r )
print(round(time()-t))
By contrast, the following shell script does the same thing in 7-8 seconds:
START_TIME=$SECONDS
cut -d',' -f2- < test.csv > testnew.csv
echo $(($SECONDS - $START_TIME))
Is there a way I can get comparable performance in Python?

Answer 1

If I understand correctly, the shell script is simply splitting lines at the first ,, regardless of whether it is enclosed in quotes or not, and writing out the second part. (I do not know what the shell script does if there is no ,.) The csv method does much more, which is useless for you. To just do the same thing as the shell in python, skip the csv module.
for line in source:
parts = line.split(',', maxsplit=1)
source.write(parts[len(parts)-1])
This passes lines without a comma as is. It leaves spaces after the comma (I do not know what cut does. If you do not want that, you can either use re.split instead of line.split or add .lstrip() just before the closing ) on the last line.

Answer 2

Your bash script not parse csv file, only split and cut. So, in python we can do the same:
with open('test.csv',"r") as source:
with open('testnew.csv',"w") as result:
for l in source:
_, tail = l.split(',', 1)
result.write(tail)
My simple profiling (4Mb file):
bash - 193 ms
python csv parsing - 2391 ms
python string splitting - 620 ms
Python 2 is faster for some reason:
bash - 193 ms
python csv parsing - 1471 ms
python string splitting - 373 ms