so I'm making a little ruby script to parse a .csv and output a large MySQL query. The problem is some of the names I'm working with have single quotes, which messes up my SQL...
My attempt to gsub the quotes into escape quotes has failed, though I'm not sure why. But sure enough when I open my query.txt file a name like D'vinci would be just that, and not D\'vinci.
What am I doing wrong?
require 'csv'
filename = ARGV[0]
q = "INSERT INTO `table` (`username`, `password`, `firstname`, `lastname`, `email`) VALUES "
CSV.foreach(filename) do |row|
last_name, first_name, email, id = row.each {|c| c.gsub "'", "\'"}
q += "('#{id}', SHA1('password'), '#{first_name}', '#{last_name}', '#{email}'),\n"
end
q += ";"
File.open("query.txt", 'w').write(q)
Do not write your own SQL escaping. Please, always use the methods provided by the appropriate database driver.
If you're using MySQL, either mysql or the newer mysql2 gem, there's an escape function that should handle this for you.
It's not entirely clear why you're writing SQL in the first place when most databases have some kind of import-from-file function. MySQL in particular has LOAD DATA INFILE which can read in many different formats if used correctly.
each returns the receiver. So, your gsub inside that block is doing nothing. Change it to map, and it shall be fine. Or, you can keep the each and use gsub! instead.
Related
I am writing a script that exports results into a SQLite database. I can not get the code to work when I use variables.
Here is an excerpt:
require 'sqlite3'
require 'shodan'
table_name = "#{Date.today.strftime("%B")}#{Time.now.year}_Findings"
db = SQLite3::Database.new "shodan_test.db"
db.execute ("CREATE TABLE #{table_name} (ip string , ports string)")
results = api.host(target)
ip = results["ip_str"].to_s
ports = results["ports"].to_s
db.execute ("insert into #{table_name} (ip, ports) values (#{ip}, #{ports})")
The code fails at the last line. I can remove the variables and the code works. I'm a bit confused since the CREATE TABLE works with a variable.
Your problem is that you're not quoting your strings. The result is that SQLite sees things like
insert into ... values (192.168.0.6, whatever is in ports)
and that's not valid SQL.
SQLite3::Database#execute understands placeholders so you should use them:
db.execute("insert into #{table_name} (ip, ports) values (?, ?)", ip, ports)
The placeholders (?) in the query will be replaced with properly quoted and escaped versions of their values. Using placeholders avoids all the usual quoting and injection problems that plague using string interpolation for SQL.
Also note that I've removed the space between execute and (. If you put a space there then Ruby thinks your "method calling parentheses" are actually "expression grouping parentheses" and it will complain about not understanding what those commas are doing. If you're only passing one argument (such as in db.execute ("CREATE ...")) then it won't matter because the parentheses are effectively ignored but it does matter when there are multiple arguments. In any case, o.m (arg) is a bad habit to get into, you should say o.m(arg) or leave the parentheses out.
I import a text file and save each row as a new record:
CSV.foreach(csv_file_path) do |row|
# saving each row to a new record
end
Strangely enough, the following escapes double quotes, but I have no clue how to escape different characters:
CSV.foreach(csv_file_path, {quote_char: "\""}) do |row|
How do I escape both the characters " and '?
Note that you have additional options available to configure the CSV handler. The useful options for specifying character delimiter handling are these:
:col_sep - defines the column separator character
:row_sep - defines the row separator character
:quote_char - defines the quote separator character
Now, for traditional CSV (comma-separated) files, these values default to { col_sep: ",", row_sep: "\n", quote_char: "\"" }. These will satisfy many needs, but not necessarily all. You can specify the right set to suit your well-formed CSV needs.
However, for non-standard CSV input, consider using a two-pass approach to reading your CSV files. I've done a lot of work with CSV files from Real Estate MLS systems, and they're basically all broken in some fundamental way. I've used various pre- and post-processing approaches to fixing the issues, and had quite a lot of success with files that were failing to process with default options.
In the case of handling single quotes as a delimiter, you could possibly strip off leading and trailing single quotes after you've parsed the file using the standard double quotes. Iterating on the values and using a gsub replacement may work just fine if the single quotes were used in the same way as double quotes.
There's also an "automatic" converter that the CSV parser will use when trying to retrieve values for individual columns. You can specify the : converters option, like so: { converters: [:my_converter] }
To write a converter is pretty simple, it's just a small function that checks to see if the column value matches the right format, and then returns the re-formatted value. Here's one that should strip leading and trailing single quotes:
CSV::Converters[:strip_surrounding_single_quotes] = lambda do |field|
return nil if field.nil?
match = field ~= /^'([^']*)'$/
return match.nil? ? field : match[1]
end
CSV.parse(input, { converters: [:strip_surrounding_single_quotes] }
You can use as many converters as you like, and they're evaluated in the order that you specify. For instance, to use the pre-defined :all along with the custom converter, you can write it like so:
CSV.parse(input, { converters: [:all, :strip_surrounding_single_quotes] }
If there's an example of the input data to test against, we can probably find a complete solution.
In general, you can't, because that will create a CSV-like record that is not standard CSV (Wikipedia has the rules in a bit easier to read format). In CSV, only double quotes are escaped - by doubling, not by using a backslash.
What you are trying to write is not a CSV; you should not use a CSV library to do it.
using raw SQL when I use the IN statement inside a query using sequel's fetch function, I can't escape a single quote by writing where stuff IN ...
#values='stuff1\'','stuff2\''
db.fetch("query...where IN (?)", "#{#values}")
outputs query...where stuff IN ('stuff1'',''stuff2') instead of ('stuff1','stuff2')
Quite frustrating that I'd probably have to write a Sequel equivalent for the raw query or use a different ORM just because of this escape issue. Any thoughts?
You should probably do something like:
#values = ['stuff1', 'stuff2']
db.fetch("query...where IN ?", #values)
If I understand the Sequel documentation correctly, using String#lit or Sequel.lit should turn a Ruby string into a literal string and bypass the automatic escaping mechanism; therefore, this should work (untested):
#values='stuff1\'','stuff2\''.lit
db.fetch("query...where IN (?)", "#{#values}")
The usual caveats when working with raw SQL strings (SQL injection attacks, inefficient SQL due to forced re-parsing of statements etc.) apply :-)
This works:
values = Sequel.lit("'stuff1', 'stuff2'")
db.fetch("SELECT * FROM TABLE1 WHERE COL1 IN ?", values)
See also: How to pass a list to an IN clause via a placeholder with Ruby Sequel
I want to execute my INSERT strings within a Ruby file that has the following array:
names_arr = ["Jack", "Susan", "Peter"]
In the same Ruby file, I've required the SQLite3 gem and have this code:
names_arr.each do |each_name|
db.execute('INSERT INTO users (name) VALUES ("#{each_name}");')
end
I am expecting this in my "users" table:
id name
1 Jack
2 Susan
3 Peter
Instead, I'm getting this:
id name
1 #{each_name}
2 #{each_name}
3 #{each_name}
It's clear I'm not interpolating properly, but I've tried a lot of different things and can't figure out where I'm wrong.
I am using SQLite3 and Ruby.
You have several problems in this code:
names_arr.each do |each_name|
db.execute('INSERT INTO users (name) VALUES ("#{each_name}");')
end
You're trying to use Ruby's string interpolation inside a single quoted string but string interpolation only works in double quoted strings (and heredocs, %Q{...} strings, regex literals, ...).
Standard SQL uses single quotes for string literals, not double quotes. Some databases let you get away with double quotes but it is a really bad habit to get into.
You shouldn't use string interpolation to build SQL anyway, you're not hacking PHP in 1999 so you should use placeholders and let the database library deal with the quoting.
The SQLite driver's execute understands ? placeholders so you can say this:
names_arr.each do |each_name|
db.execute('INSERT INTO users (name) VALUES (?)', each_name)
end
Using Ruby how would I be able to automatically escape single and double quotes in some of the variables being written to the output file. Coming from PHP I'm looking for an addslashes type function, but there doesn't seem to be a simple solution for this in Ruby.
require "csv"
def generate_array( file )
File.open("#{file}" + "_output.txt", 'w') do |output|
CSV.foreach(file) do |img, _, part, focus, country, loc, lat, lon, desc, link|
output.puts("[#{lat}, #{lon}, '#{img.downcase}', '#{part}', '#{loc}', '#{focus}', '#{country}', '#{desc}', '#{link}'],")
end
end
end
ARGV.each do |file|
generate_array(file)
end
I suppose you can emulate PHP addslashes functionality with this Ruby construct:
.gsub(/['"\\\x0]/,'\\\\\0')
For example:
slashed_line = %q{Here's a heavily \s\l\a\s\h\e\d "string"}
puts slashed_line.gsub(/['"\\\x0]/,'\\\\\0')
# Here\'s a heavily \\s\\l\\a\\s\\h\\e\\d \"string\"
There is also String#dump:
slashed_line = %q{Here's a heavily \s\l\a\s\h\e\d "string"}
puts slashed_line.dump
#=> "Here's a heavily \\s\\l\\a\\s\\h\\e\\d \"string\""
I don't know Ruby, but I know that in PHP addslashes is pretty much deprecated.
Every time that you need to escape data, it requires a different escape routine. HTML needs different encoding and handling over database work, and each database has its own special rules.
I assume by your question that you are looking to output things to a CSV file. That, again, opens up a whole kettle of fish as there is no standard CSV. You'll need to do some research on both what will be making the data (and if it will be strict ASCII or Unicode or something else) and which format of escaping quotes will be needed. Most CSV consumers use a two double quotes to replace a single double quote. If you need " in your string, you write "".