Codeigniter CSV upload then explode - codeigniter

I have some code that uploads the CSV file to the specified folder, but it doesn't update the database.
public function do_upload()
{
$csv_path = realpath(APPPATH . '/../assets/uploads/CSV/');
$config['upload_path'] = $csv_path;
$config['allowed_types'] = '*'; // All types of files allowed
$config['overwrite'] = true; // Overwrites the existing file
$this->upload->initialize($config);
$this->load->library('upload', $config);
if ( ! $this->upload->do_upload('userfile'))
{
$error = array('error' => $this->upload->display_errors());
$this->layout->buffer('content', 'program/upload', $error);
$this->layout->render();
}
else
{
$image_data = $this->upload->data();
$fname = $image_data['file_name'];
$fpath = $image_data['file_path'].$fname;
$fh = fopen($fpath, "r");
$insert_str = 'INSERT INTO wc_program (JobRef, Area, Parish, AbbrWorkType, WorkType, Timing, TrafficManagement, Location, Duration, Start, Finish) VALUES '."\n";
if ($fh) {
// Create each set of values.
while (($csv_row = fgetcsv($fh, 2000, ',')) !== false) {
foreach ($csv_row as &$row) {
$row = strtr($row, array("'" => "\'", '"' => '\"'));
}
$insert_str .= '("'
// Implode the array and fix pesky apostrophes.
.implode('","', $csv_row)
.'"),'."\n";
}
// Remove the trailing comma.
$insert_str = rtrim($insert_str, ",\n");
// Insert all of the values at once.
$this->db->set($insert_str);
echo '<script type="text/javascript">
alert("Document successfully uploaded and saved to the database.");
location = "program/index";
</script>';
}
else {
echo '<script type="text/javascript">
alert("Sorry! Something went wrong please proceed to try again.");
location = "program/upload";
</script>';
}
}
}
When I run var_dump($fh); it shows: resource(89) of type (stream)
When I run var_dump($fpath) it shows: string(66) "/Applications/MAMP/htdocs/site/assets/uploads/CSV/wc_program.csv"
So it all uploads but what is wrong with it not updating the database?
I have tried all kinds of changing the fopen method but still no joy, I really need it to add to the database and the insert query and set query should do the trick but it doesn't.
Any help greatly appreciated!

You are not running any query on the database. You are mixing active record syntax with simple query syntax. The active record insert query will be executed by calling.
$this->db->insert('my_table');
db::set() does not actually query the database. It takes in a key/value pair that will be inserted or updated after db::insert() or db::update() is called. If you build the query yourself you need to use the db::query() function.
Review the active directory documentation.
You can use $this->db->query('put your query here'), but you lose the benefit of CodeIgniter's built in security. Review CodeIgniter's query functions.
I'll give you examples of just a few of the many ways you can insert into a database using CodeIgniter. The examples will generate the query from your comment. You will need to adjust your code accordingly.
EXAMPLE 1:
$result = $this->db
->set('JobRef', 911847)
->set('Area', 'Coastal')
->set('Parish', 'Yapton')
->set('AbbrWorkType', 'Micro')
->set('WorkType', 'Micro-Asphalt Surfacing')
->set('Timing', 'TBC')
->set('TrafficManagement', 'No Positive Traffic Management')
->set('Location', 'Canal Road (added PMI 16/07/12)')
->set('Duration', '2 days')
->set('Start', '0000-00-00')
->set('Finish', '0000-00-00')
->insert('wc_program');
echo $this->db->last_query() . "\n\n";
echo "RESULT: \n\n";
print_r($result);
EXAMPLE 2 (Using an associative array):
$row = array(
'JobRef' => 911847,
'Area' => 'Coastal',
'Parish' => 'Yapton',
'AbbrWorkType' => 'Micro',
'WorkType' => 'Micro-Asphalt Surfacing',
'Timing' => 'TBC',
'TrafficManagement' => 'No Positive Traffic Management',
'Location' => 'Canal Road (added PMI 16/07/12)',
'Duration' => '2 days',
'Start' => '0000-00-00',
'Finish' => '0000-00-00'
);
$this->db->insert('wc_program', $row);
// This will do the same thing
// $this->db->set($row);
// $this->db->insert('wc_program');
echo $this->db->last_query();
Example 1 and 2 are using the Active Record. The information is stored piece by piece and then the query is built when you make the final call. This has several advantages. It allows you to build queries dynamically without worrying about SQL syntax and order of the keywords. It also escapes your data.
EXAMPLE 3 (Simple Query):
$query = 'INSERT INTO
wc_program
(JobRef, Area, Parish, AbbrWorkType, WorkType, Timing, TrafficManagement, Location, Duration, Start, Finish)
VALUES
("911847","Coastal","Yapton","Micro","Micro-Asphalt Surfacing","TBC","No Positive Traffic Management","Canal Road (added PMI 16/07/12)","2 days","0000-00-00","0000-00-00")';
$result = $this->db->query($query);
echo $this->db->last_query() . "\n\n";
echo "RESULT: \n";
print_r($result);
This way leaves all the protection against injection up to you, can lead to more errors, and is harder to change/maintain.
If you are going to do it this way you should use the following syntax, which will protect against injection.
EXAMPLE 4:
$query = 'INSERT INTO
wc_program
(JobRef, Area, Parish, AbbrWorkType, WorkType, Timing, TrafficManagement, Location, Duration, Start, Finish)
VALUES
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);';
$row = array(
911847,
'Coastal',
'Yapton',
'Micro',
'Micro-Asphalt Surfacing',
'TBC',
'No Positive Traffic Management',
'Canal Road (added PMI 16/07/12)',
'2 days',
'0000-00-00',
'0000-00-00'
);
$result = $this->db->query($query, $row);
echo $this->db->last_query() . "\n\n";
echo "RESULT: \n";
print_r($result);
CodeIgniter will replace each "?" in the query with the corresponding value from the array after it is escaped. You can use this to run many queries that are of the same form, but have different data just by updating the $row array and benefit from CI's built in security.

Related

Codeigniter query with alias and number as column

I would like to replace column name with numbers in codeigniter
It does work well with letters but not with numbers
$this->db->select ('Observation.id_observation as 1');
$this->db->from ( 'Observation' );
$this->db->join ( 'Enregistrement', 'Enregistrement.id_observation =
Observation.id_observation' , 'left' );
$this->db->where('Observation.id_parcelle', $id_parcelle);
$this->db->where('Observation.id_observateur', $user_id);
$this->db->where('Observation.date_observation >=', $date_start);
$this->db->where('Observation.date_observation <=', $date_end);
return $this->db->get()->result();
Try the following
$this->db->select ('Observation.id_observation as `1`');
This will generate query as
Select `observation` as `1`....
From the Docs
Identifiers may begin with a digit but unless quoted may not consist solely of digits.
You can set alias name as string so you need to give within single quotes otherwise you will get syntax error
$this->db->select ('Observation.id_observation as `1`',FALSE);
$this->db->from ( 'Observation' );
$this->db->join ( 'Enregistrement', 'Enregistrement.id_observation =
Observation.id_observation' , 'left' );
$this->db->where('Observation.id_parcelle', $id_parcelle);
$this->db->where('Observation.id_observateur', $user_id);
$this->db->where('Observation.date_observation >=', $date_start);
$this->db->where('Observation.date_observation <=', $date_end);
return $this->db->get()->result();
OR
$var = '1';
$this->db->select ('Observation.id_observation as `'.$var.'`',FALSE);
OR
$query=$this->db->query('SELECT Observation.id_observation as "1" FROM Observation LEFT JOIN Enregistrement ON Enregistrement.id_observation =
Observation.id_observation WHERE Observation.id_parcelle = '.$id_parcelle.' AND Observation.id_observateur = '.$user_id.' AND Observation.date_observation >= "'.$date_start.'" AND Observation.date_observation <= "'.$date_end.'" ');
return $query->result();
You can try this:
$this->db->select ("Observation.id_observation as '1'");
Hope It will work.
This is what i had to do in the controller ! I very don't like this solultion but it does work:
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
$sql = "SELECT
Observation.id_observation as '0',
Observation.checkseries as '1'
FROM Observation
LEFT JOIN Enregistrement ON Enregistrement.id_observation = Observation.id_observation
WHERE
Observation.id_parcelle = '$parcellecultures->id_parcelle'
AND Observation.id_observateur = '$user'
AND Observation.date_observation >= '$date_start'
AND Observation.date_observation <= '$date_end'
";
//echo $sql;
$result = $conn->query($sql);
//print_r($result);exit;
$conn->close();
I still look for another solution ....

Scraping multiple items off of a Page into a Neat Row

As an example:
I load in the input from a .txt:
Benjamin,Schuvlein,Germany,1912,M,White
I do some code that I will not post here for brevity and get to the link:
https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ
I want to scrape multiple things from that page. In the code below, I only do 1.
I'd also like to make each item be separated by a , in the output .txt.
And, I'd like the output to be preceded by the input.
I'm using the following packages in the code:
use strict;
use warnings;
use WWW::Mechanize::Firefox;
use Data::Dumper;
use LWP::UserAgent;
use JSON;
use CGI qw/escape/;
use HTML::DOM;
Here's the relevant code:
my $ua = LWP::UserAgent->new;
open(my $o, '>', 'out2.txt') or die "Can't open output file: $!";
# Here is the url, although in practice, it is scraped itself using different code
my $url = 'https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ';
print "My URL is <$url>\n";
my $request = HTTP::Request->new(GET => $url);
$request->push_header('Content-Type' => 'application/json');
my $response = $ua->request($request);
die "Error ".$response->code if !$response->is_success;
my $dom_tree = new HTML::DOM;
$dom_tree->write($response->content);
$dom_tree->close;
my $str = $dom_tree->getElementsByTagName('table')->[0]->getElementsByTagName("td")->[10]->as_text();
print $str;
print $o $str;
Desired Output (from that link) is something like:
Benjamin,Schuvlein,Germany,1912,M,White,Queens,New York,Married,Same Place,Head, etc ....
(How much of that output section is scrapable?)
Any help on how to get the link within the link would be much appreciated!
This is fairly simply done using HTML::TreeBuilder::XPath to access the HTML. This program builds a hash of the data using the labels as keys, so any of the desired information can be extracted. I have enclosed in quotes any fields that contain commas or whitespace.
I don't know whether you have the permission of this web site to extract data this way, but I should draw your attention to this X-Copyright header in the HTTP responses. This approach clearly falls under the header of programmatic access.
X-Copyright: COPYRIGHT WARNING Data accessible through the FamilySearch API is protected by copyright. Any programmatic access, reformatting, or rerouting of this data, without permission, is prohibited. FamilySearch considers such unauthorized use a violation of its reproduction, derivation, and distribution rights. Contact devnet (at) familysearch.org for further information.
Am I to expect an email from you? I replied to your first mail but haven't heard since.
use strict;
use warnings;
use URI;
use LWP;
use HTML::TreeBuilder::XPath;
my $url = URI->new('https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ');
my $ua = LWP::UserAgent->new;
my $resp = $ua->get($url);
die $resp->status_line unless $resp->is_success;
my $tree = HTML::TreeBuilder::XPath->new_from_content($resp->decoded_content);
my #results = $tree->findnodes('//table[#class="result-data"]//tr[#class="result-item"]');
my %data;
for my $item (#results) {
my ($key, $val) = map $_->as_trimmed_text, $item->content_list;
$key =~ s/:$//;
$data{$key} = $val;
}
my $record = join ',', map { local $_ = $data{$_}; /[,\s]/ ? qq<"$_"> : $_ }
'name', 'birthplace', 'estimated birth year', 'gender', 'race (standardized)',
'event place', 'marital status', 'residence in 1935',
'relationship to head of household (standardized)';
print $record, "\n";
output
"Benjamin Schuvlein",Germany,1912,Male,White,"Assembly District 2, Queens, New York City, Queens, New York, United States",Married,"Same Place",Head
Try this
use LWP::Simple;
use LWP::UserAgent;
use HTML::TableExtract;
$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0;
$ua = LWP::UserAgent->new;
$ua->agent("Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.91 Safari/537.11");
$req = HTTP::Request->new(GET => "https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ");
$res = $ua->request($req);
$content = $res->content;
#$content = get("https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ") or die "Couldn't get it! $!";
$te = HTML::TableExtract->new( attribs => { 'class' => 'result-data' } );
# $te = HTML::TableExtract->new( );
$te->parse($content);
$table = $te->first_table_found;
# print $content; exit;
# $te->tables_dump(1);
#print Dumper($te);
#print Dumper($table);
print $table->cell(4,0) . ' = ' . $table->cell(4,1), "\n"; exit;
Which prints out
event place: = Assembly District 2, Queens, New York City, Queens, New York, United States
I also noticed this header:
X-Copyright:COPYRIGHT WARNING Data accessible through the FamilySearch API is protected by copyright. Any programmatic access, reformatting, or rerouting of this data, without permission, is prohibited. FamilySearch considers such unauthorized use a violation of its reproduction, derivation, and distribution rights. Contact devnet (at) familysearch.org for further information.
See also http://metacpan.org/pod/HTML::Element#SYNOPSIS
I thought I had answered your question.
The problem is that you are trying to fetch the webpage with LWP. Why are try to doing that if you already have WWW::Mechanize::Firefox?
Did you tried this?
It will retrieve and save each link for further analyses. A small change and you 'get' the DOM tree. Sorry, I do not have acccess to this page, so I just hope it will work.
my $i=1;
for my $link (#links) {
print Dumper $link->url;
print Dumper $link->text;
my $tempfile = './$i.html';$i++;
$mech->get( $link, ':content_file' => $tempfile, synchronize => 1 );
my $dom_tree = $mech->document();
my $str = $dom_tree->getElementsByTagName('table')->[0]->getElementsByTagName("td")->[9]->as_text();
}
EDIT:
Process the page content with regexp (Everyone: Please remember, there is always more than one way to do something wwith Perl!. It works, it is easy...)
it tried it out with this cmd:
wget -nd 'https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ' -O 1.html|cat 1.html|1.pl
use Data::Dumper;
use strict;
use warnings;
local $/=undef;
my $html = <>;#read from file
#$html = $mech->content( format => 'html' );# read data from mech object
my $data = {};
my $current_label = "not_defined";
while ($html =~ s!(<td[^>]*>.*?</td>)!!is){ # process each TD
my $td = $1;
print "td: $td\n";
my $td_val = $td;
$td_val =~ s!<[^>]*>!!gis;
$td_val =~ s!\s+! !gs;
$td_val =~ s!(\A\s+|\s+\z)!!gs;
if ($td =~ m!result-label!){ #primitive state machine, store the current label
print "current_label: $current_label\n";
$current_label = $td_val;
} elsif ($td =~ m!result-value!){ #add each data to current label
push(#{$data->{$current_label}},$td_val);
} else {
warn "found something else: $td\n";
}
}
#process it using a white lists of known entries (son,race, etc).Delete from the result if you find it on white list, die if you find something new.
#multi type
foreach my $type (qw(son wife daughter head)){
process_multi($type,$data->{$type});
delete($data->{$type});
}
#simple type
foreach my $type (qw(birthplace age)){
process_simple($type,$data->{$type});
delete($data->{$type});
}
die "Unknown label!".Dumper($data) if scalar(keys %{$data})>0;
Output:
'line number:' => [
'28'
],
'estimated birth year:' => [
'1912'
],
'head' => [
'Benjamin Schuvlein',
'M',
'28',
'Germany'
],

Timepicker that removes times as they're selected (ajax)

I'm building a booking form for a moving business that uses a calendar combined with a start and end time. I built the timepicker with Formidable Pro, and it allows me to check "unique" on time fields which automatically removes them on the selected date. However it doesn't automatically remove the times from within the range between start and end times (ie: if someone chooses to rent a truck from 1am-3am I need 1am,2am,and 3am to be removed from future options but right now it only removes 1am and 3am) . I need to write ajax to remove the in-between times from the options. I'm not sure where to begin. This is the current ajax_time_ options function. Any push in the right direction would be appreciated.
function ajax_time_options(){
global $frmpro_settings, $frmdb, $wpdb;
//posted vars = $time_field, $date_field, $step, $start, $end, $date, $clock
extract($_POST);
$time_key = str_replace('field_', '', $time_field);
$date_key = str_replace('field_', '', $date_field);
if (!preg_match('/^\d{4}-\d{2}-\d{2}$/', trim($date)))
$date = FrmProAppHelper::convert_date($date, $frmpro_settings->date_format, 'Y-m-d');
$date_entries = FrmEntryMeta::getEntryIds("fi.field_key='$date_key' and meta_value='$date'");
$opts = array('' => '');
$time = strtotime($start);
$end = strtotime($end);
$step = explode(':', $step);
$step = (isset($step[1])) ? ($step[0] * 3600 + $step[1] * 60) : ($step[0] * 60);
$format = ($clock) ? 'H:i' : 'h:i A';
while($time <= $end){
$opts[date($format, $time)] = date($format, $time);
$time += $step;
}
if($date_entries and !empty($date_entries)){
$used_times = $wpdb->get_col("SELECT meta_value FROM $frmdb->entry_metas it LEFT JOIN $frmdb->fields fi ON (it.field_id = fi.id) WHERE fi.field_key='$time_key' and it.item_id in (". implode(',', $date_entries).")");
if($used_times and !empty($used_times)){
$number_allowed = apply_filters('frm_allowed_time_count', 1, $time_key, $date_key);
$count = array();
foreach($used_times as $used){
if(!isset($opts[$used]))
continue;
if(!isset($count[$used]))
$count[$used] = 0;
$count[$used]++;
if((int)$count[$used] >= $number_allowed)
unset($opts[$used]);
}
unset($count);
}
}
echo json_encode($opts);
die();
}

Export comments from fb:comments

I'm a little desperate about this. I have a site with a comments box with around 183000 comments that I want to backup. But every time I do an fql.query, I only get 100 items. I had tried:
SELECT xid, object_id, post_id, time, text, id FROM comment WHERE xid='bdatapoyo' ORDER BY time DESC
and after that, using the last "time"
SELECT xid, object_id, post_id, time, text, id FROM comment WHERE xid='bdatapoyo' AND time >= xxxxxxx ORDER BY time DESC
but the latter only gives me one comment..
I made an PHP page with this code, but again, it only grabs 100 items:
<?php
require 'php-sdk/src/facebook.php';
$facebook = new Facebook(array(
'appId' => 'XXXXXXXXXX',
'secret' => 'XXXXXXXXXX',
'cookie' => true,
));
$conexion = mysql_connect("localhost", "XXXX", "XXXX");
mysql_select_db("TVN", $conexion);
$time_old = "1315070888";
do {
$fql = "SELECT fromid,time,text FROM comment WHERE xid='bdatapoyo' AND time >= " . $time_old . " ORDER BY time";
$response = $facebook->api(array(
'method' => 'fql.query',
'query' =>$fql,
));
foreach($response as $key => $value) {
$sql = "INSERT INTO comment (text, uuid, time) VALUES ('" . $value["text"] . "', '" . $value["fromid"] . "', '" . $value["time"] . "')";
$result = mysql_query($sql);
}
$time_old = $response[count($response)-1]["time"];
print_r($time_old);
} while (count($response) > 0);
?>
OK, I was able to pull this out with LIMIT and OFFSET. The correct FQL is
SELECT fromid,time,text FROM comment WHERE xid='bdatapoyo' LIMIT 100 OFFSET 0
and then paging with the OFFSET
is this code setup to import every single comment to a db from all users on your site? What is xid=bdatapoyo?

Magento, 1 db field not saved

I have a problem with one field of the DB. With this code:
$expireMonth = Mage::getStoreConfig('points_options/config_points/expiration_period', Mage::app()->getStore()->getId());
if (!is_null($expireMonth) && ($expireMonth > 0)) {
$expireDate = date("Y-m-d H:i:s", strtotime("+" . $expireMonth . " month"));
} else {
$expireDate = NULL;
}
//die($expireDate);
//store in points history table
$this->_pointsModel->setCustomerId($this->_customer->getId())
->setOrdersId('welcome')
->setPointsPending($pointsForNewCustomer)
->setPointsComment(Mage::helper('points')->__('welcome points'))
->setDateAdded(date('Y-m-d H:i:s'))
->setPointsStatus(2)//confirmed
->setPointsType('WE')
->setStoreId(Mage::app()->getStore()->getId())
->setExpireDate($expireDate)
->save();
Every field is saved in the table, except for expire_date. If I uncomment the die($expireData), I see the correct value, something like 2012-01-13 13:21:12. The field is defined as:
`expire_date` datetime NULL
Any thoughts?
edit: The solution is:
$expireDate = date("Y-m-d H:i:s", strtotime("+" . $expireMonth . " months"));
Check out the "s" in my strtotime expression.
I know many attributes allow for some sort of formatting before writing and after reading. Have you tried setting the value as a unix timestamp instead?

Resources