How to wrap the first 10 commas with double quotes? - bash
Here is my input.csv file
dealerid,address,city,state,zip,vin,stocknumber,type,color,year,make,model,trim,bodystyle,fueltype,mileage,transmission,interiorcolor,interiorfabric,price,titlestatus,warranty,options_text,cylinders,engine,engineaspiration,enginetext,drivetrain,transmissiontext,mpgcity,mpghighway,features_text,vdc_url,images
TS06095298,999 wanna Road,Windsor,CT,06095,22HDT13S922218113,298,Used,Red,2002,OLDSMOBILE,BRAVADA,,,,136000,AUTOMATIC,,,2200,Clear,Available,"This vehicle is offered for sale by a verified private seller and features: FREE vehicle history & title report. Original window sticker available. Seller`s identity, email and phone verified. Secure cashless transactions. No cash needed. Pay securely by debit card or ACH. Bill of Sale and receipt issued for completed transactions. Vehicle financing options may be available.",,,,,,,,,,https://www.example.com/listings/298,"https://www.example.com/rails/00008.jpg,https://www.example.com/rails/AM00010.jpg"
TS06095298,999 wanna Road,Windsor,CT,06095,22HDT13S922123453,307,Used,Brown,2008,HONDA,599,,,,217538,AUTOMATIC,,,3500,Clear,Available,"This vehicle is offered for sale by a verified private seller and features: FREE vehicle history & title report. Original window sticker available. Seller`s identity, email and phone verified. Secure cashless transactions. No cash needed. Pay securely by debit card or ACH. Bill of Sale and receipt issued for completed transactions. Vehicle financing options may be available.",,,,,,,,,,https://www.example.com/listings/211,"https://www.example.com/rails/00008.jpg,https://www.example.com/rails/AM00010.jpg"
I need to wrap all columns with double quotes so I end up with a file like so:
"dealerid","address","city","state","zip","vin","stocknumber","type","color","year","make","model","trim","bodystyle","fueltype","mileage","transmission","interiorcolor","interiorfabric","price","titlestatus","warranty","options_text","cylinders","engine","engineaspiration","enginetext","drivetrain","transmissiontext","mpgcity","mpghighway","features_text","vdc_url","images"
"TS06095298","999 wanna Road,Windsor","CT","06095","22HDT13S922218113","298","Used","Red","2002","OLDSMOBILE","BRAVADA","","","","136000,AUTOMATIC","","","2200","Clear","Available","This vehicle is offered for sale by a verified private seller and features: FREE vehicle history & title report. Original window sticker available. Seller`s identity, email and phone verified. Secure cashless transactions. No cash needed. Pay securely by debit card or ACH. Bill of Sale and receipt issued for completed transactions. Vehicle financing options may be available.","","","","","","","","","","https://www.example.com/listings/298","https://www.example.com/rails/00008.jpg,https://www.example.com/rails/AM00010.jpg"
"TS06095298","999 wanna Road,Windsor","CT","06095","22HDT13S922123453","307","Used","Brown","2008","HONDA","599","","","","217538","AUTOMATIC","","","3500","Clear","Available","This vehicle is offered for sale by a verified private seller and features: FREE vehicle history & title report. Original window sticker available. Seller`s identity, email and phone verified. Secure cashless transactions. No cash needed. Pay securely by debit card or ACH. Bill of Sale and receipt issued for completed transactions. Vehicle financing options may be available.","","","","","","","","","","https://www.example.com/listings/211","https://www.example.com/rails/00008.jpg,https://www.example.com/rails/AM00010.jpg"
The file is pretty constant throughout with the same missing data from certain columns.
The images column and the features text column come wrapped already.
Seeing the same info was missing throughout I decided to add double quotes to the beginning of each line and started to replace the commas with double quotes but started running into some issues.
Here is what I have so far. I know the code is not very efficient but it's a start.
#!/bin/bash
#- Temp Directories
tmp_dir="$(mktemp -d -t 'csv.XXXXX' || mktemp -d 2>/dev/null)"
tmp_input1="${tmp_dir}/temp_input1.csv"
tmp_input2="${tmp_dir}/temp_input2.csv"
tmp_input3="${tmp_dir}/temp_input3.csv"
#- Variables
client="00000"
wDir="$(pwd)"
ftpDir="${wDir}/.clientftp"
clientDir="${ftpDir}/${client}"
csvFile="${clientDir}/final.csv"
inputCsv="${wDir}/input.csv"
# Lets Begin
cd "$wDir" || exit
cp "$inputCsv" "$tmp_input1"
dos2unix "$tmp_input1"
# place first line to a temp file , surrounding commas with double quotes , adding double quotes to the front and end of line
head -1 "$tmp_input1" | sed -e 's/,/","/g;s/.*/"&"/' > "$tmp_input2"
# place remainding lines to a temp file
sed 1,1d "$tmp_input1" | sed "s/^/\"/" > "$tmp_input3"
sed -i 's/",,,,,,,,,,https/","","","","","","","","","","https/g' "$tmp_input3"
sed -i 's/,Clear,Available,"/","Clear","Available","/g' "$tmp_input3"
sed -i 's/,,,,/","","","","/g' "$tmp_input3"
sed -i 's/,,,/","","","/g' "$tmp_input3"
# Create final file
cat "$tmp_input2" > "$csvFile"
cat "$tmp_input3" >> "$csvFile"
rm -rf "$tmp_dir"
{ clear; echo ""; echo ""; echo "nano $csvFile"; echo ""; }
nano "$csvFile"
This script produces:
"dealerid","address","city","state","zip","vin","stocknumber","type","color","year","make","model","trim","bodystyle","fueltype","mileage","transmission","interiorcolor","interiorfabric","price","titlestatus","warranty","options_text","cylinders","engine","engineaspiration","enginetext","drivetrain","transmissiontext","mpgcity","mpghighway","features_text","vdc_url","images"
"TS06095298,999 wanna Road,Windsor,CT,06095,22HDT13S922218113,298,Used,Red,2002,OLDSMOBILE,BRAVADA","","","","136000,AUTOMATIC","","","2200","Clear","Available","This vehicle is offered for sale by a verified private seller and features: FREE vehicle history & title report. Original window sticker available. Seller`s identity, email and phone verified. Secure cashless transactions. No cash needed. Pay securely by debit card or ACH. Bill of Sale and receipt issued for completed transactions. Vehicle financing options may be available.","","","","","","","","","","https://www.example.com/listings/298,"https://www.example.com/rails/00008.jpg,https://www.example.com/rails/AM00010.jpg"
"TS06095298,999 wanna Road,Windsor,CT,06095,22HDT13S922123453,307,Used,Brown,2008,HONDA,599","","","","217538,AUTOMATIC","","","3500","Clear","Available","This vehicle is offered for sale by a verified private seller and features: FREE vehicle history & title report. Original window sticker available. Seller`s identity, email and phone verified. Secure cashless transactions. No cash needed. Pay securely by debit card or ACH. Bill of Sale and receipt issued for completed transactions. Vehicle financing options may be available.","","","","","","","","","","https://www.example.com/listings/211,"https://www.example.com/rails/00008.jpg,https://www.example.com/rails/AM00010.jpg"
So now I have a few issues:
1- The vdc_url column does not have a closing double quote
2- first 10 commas need to be wrapped with double quotes
The last column can contain more than 3 images
Any help would be appreciated.
I like ruby for quick CVS conversions:
ruby -rcsv -e '
out = CSV.instance($stdout, {force_quotes: true})
CSV.foreach(ARGV.shift) {|row| out << row}
' input.csv
make sure you don't have any trailing whitespace on any line.
csvkit would also be a good solution.
With GNU awk for FPAT:
$ awk -v FPAT='[^,]*|"[^"]*"' -v OFS=',' '
{ for (i=1;i<=NF;i++) {gsub(/^"|"$/,"",$i); $i="\"" $i "\""} }
1' file
"dealerid","address","city","state","zip","vin","stocknumber","type","color","year","make","model","trim","bodystyle","fueltype","mileage","transmission","interiorcolor","interiorfabric","price","titlestatus","warranty","options_text","cylinders","engine","engineaspiration","enginetext","drivetrain","transmissiontext","mpgcity","mpghighway","features_text","vdc_url","images"
"TS06095298","999 wanna Road","Windsor","CT","06095","22HDT13S922218113","298","Used","Red","2002","OLDSMOBILE","BRAVADA","","","","136000","AUTOMATIC","","","2200","Clear","Available","This vehicle is offered for sale by a verified private seller and features: FREE vehicle history & title report. Original window sticker available. Seller`s identity, email and phone verified. Secure cashless transactions. No cash needed. Pay securely by debit card or ACH. Bill of Sale and receipt issued for completed transactions. Vehicle financing options may be available.","","","","","","","","","","https://www.example.com/listings/298","https://www.example.com/rails/00008.jpg,https://www.example.com/rails/AM00010.jpg"
"TS06095298","999 wanna Road","Windsor","CT","06095","22HDT13S922123453","307","Used","Brown","2008","HONDA","599","","","","217538","AUTOMATIC","","","3500","Clear","Available","This vehicle is offered for sale by a verified private seller and features: FREE vehicle history & title report. Original window sticker available. Seller`s identity, email and phone verified. Secure cashless transactions. No cash needed. Pay securely by debit card or ACH. Bill of Sale and receipt issued for completed transactions. Vehicle financing options may be available.","","","","","","","","","","https://www.example.com/listings/211","https://www.example.com/rails/00008.jpg,https://www.example.com/rails/AM00010.jpg"
Related
NEAR Marketplace - How should I charge the transaction fee on each sales?
We're building a marketplace in NEAR. We have two smart contracts for token and marketplace. Same as NEAR example. In our platform, we have successfully implemented below features: Token mint (used contract method: nft_mint) Token listing for sale in marketplace (used contract methods: storage_deposit nft_approve) Token purchase (used contract method: offer) Everything is working fine. Now, we want to charge the transaction fee (2.5%) on each sales for our marketplace. -- I did one mint and buy test with Paras Marketplace to observe the royalty and transaction fee distribution. Here is the test details: With seller.near account, I minted an NFT in Paras Marketplace. And added 3% royalty. And listed it #1 Ⓝ for sale. With buyer.near account, bought it with another account. NFT Details: Name : My Non-fungible token #1 Listed Price : 1 Ⓝ Royalty : 3% Sale Breakdown: Receive (Seller) : 0.95 Ⓝ Royalty (Creator) : 0.03 Ⓝ Fee (Paras) : 0.02 Ⓝ Before purchase - Account Balance buyer.near : 50.03995 Ⓝ seller.near : 4.896 Ⓝ After purchase (1st sale after minted) - Account Balance buyer.near : 49.03388 Ⓝ | Difference : -1.00607 Ⓝ seller.near : 5.82587 Ⓝ | Difference : +0.92987 Ⓝ # NFT Create Transaction This is nft_create_series transaction. Where Paras is sending "transaction_fee": "200" to charge 2% service fee on each sale: # NFT Buy Transaction This is buy transaction. Where Paras charged 2% service fee: Question: We want to charge 2.5% service fee on each sales. We want to implement "transaction_fee": "250" object in our marketplace contract. How to do the same with our marketplace?
There are several ways that you can go about doing this. The two that I would recommend are the following (I saw you were following along with the zero to hero tutorial): Option A (simplest) - store your account in the perpetual royalties object for the token when it is minted. This is the easiest to implement as you don't need to change anything in your contracts and you simply need to alter what is passed into the royalties object on mint. This requires that you have control over the NFT contract. This allows you to receive royalties on whatever marketplace the token is sold on since the royalties are stored on the token level in the NFT contract. This allows you to specify which account should receive the royalties instead of forcing the marketplace contract to store the funds. Option B - before calling nft_transfer_payout in your marketplace contract, subtract 2.5% from the price object. This will result in your marketplace contract keeping the 2.5% and paying out the remaining 97.5% to the respective accounts. With this approach, since many accounts are paying for storage, your contract has funds that aren't all withdraw-able by you. You should keep a tally on how much $NEAR you've received so that you can withdraw that amount and not accidentally withdraw too many funds that may have "belonged" to someone else via storage deposits.
Transaction declined in magento 1.9 when testing with DPS PxPay2.0
I am getting transaction declined errors (The transaction was Declined (BH)) when using a test with DPS Payment Express account (PxPay 2.0) for processing credit cards on a website.I am using test credit card no 4111111111111111, expiry date is any future date ,any name for card holder name and 3 digit csv numer. I am getting following error(response) Response:DECLINED Response Code:BH Currency:USD Amount: 2016.92 Card:411111........11 Card Holder:MANISH KUMAR Card Type:Visa Date:20141007 Time:060747 Transaction Type:Purchase Help Text:The transaction was Declined (BH) Can any one please help me....
It is solved by change the currency to NZD.
For others that are using PXPOST "The transaction was Declined (BH)" means that the port is not configured for the Currency you are trying to charge in. In our case we already had a multi-currency merchant account with our bank but we had to call Payment Express and ask them to enable the new currency we were sending.
Credit card processing: duplicate data 1 with # one empty
I have magento 1.8 with a 1.7 ves_t-shirt theme installed. I have Swift Checkout and onestepcheckout The problem is that after hitting “Purchase” the credit card data is getting posted twice. payment[cc_cid] 567 payment[cc_cid] payment[cc_exp_month] 6 payment[cc_exp_year] 2016 payment[cc_number] 4111111111111 payment[cc_number] payment[cc_type] VI payment[method] braintree this is the post data from firebug. You can see that the CID and CC# are being posted twice. The second is blank which then triggers this error: There was an error capturing the transaction. (Transaction declined: Credit card number is required. Credit card must include either number or venmo_sdk_payment_method_code. Credit card type is not accepted by this merchant account.) Can anyone tell me how to stop the second set of data?
Custom shipping - Free with upgrade option
We offer 2 type of delivery to uk customers spend over £100 and get it free special delivery spend under £100 and get it free 1st class record but option of Special Delivery for extra £3. Cannot work out how to enable to in Magento (1.7.0.2) So how can I add the free with £3 option for carts less than £100?
So I found this extension: http://www.webshopapps.com/uk/free/matrixrate-shipping-extension.html This is the table I used: "Country","Region/State","City","Zip/Postal Code From","Zip/Postal Code To","Order Subtotal From","Order Subtotal To","Shipping Price","Delivery Type" "GBR","","","","","0","99.99","0","Free 1st Class Recorded" "GBR","","","","","0","99.99","3","Upgrade to Special Delivery Next Day before 1pm" "GBR","","","","","100","9999","0","Free Special Delivery Next Day before 1pm" Which works like a charm - hope this helps
How to verify web text while excluding random vars with ruby/watir
I have this text and need to verify the bolded words only in each line: Insured Direct Bill / 16.67%: $xxx.xx down, includes the Policy Fee and Financial Responsibility Fee, and 5 installments of $xxx.xx (does not include $x.xx installment fee) Insured Direct Bill / 25%: $xxx.xx down, includes the Policy Fee and Financial Responsibility Fee, and 5 installments of $xxx.xx (does not include $x.xx installment fee) Electronic Funds Transfer: $xxx.xx down, includes the Policy Fee and Financial Responsibility Fee, and 5 installments of $xxx.xx (does not include $x.xx installment fee) I'm assuming I'll need to use: foo = REGEX #browser.text.include?(foo) I'm stuck at how to get it to see everything other than the currency amounts. [^$\d+.\d+] excludes all the currency but includs the 5 and 16.67 Thanks
regex = /Insured Direct Bill \/ 25%: \$(\d+\.\d+) down, includes the Policy Fee and Financial Responsibility Fee, and 5 installments of \$(\d+\.\d+) \(does not include \$(\d+\.\d+) installment fee\)/ #browser.text.match(regex)
There are some regex lessons in Marick's book Everyday Scripting with Ruby. I recommend them to strengthen your capability if you need to use regex from time to time.