I am working on a project in which i have to generate QR codes based on number of members in database. Each qrcode will have a unique string so i am generating a random string of 30 chracters. As of now i have 5000 members in database and i need to generate 5000 QR Codes.
When i run the application it fetches total number of records from database and in foreach loop i am generating QR Code, but the server responds to 503 error after 40-50 seconds though i set the limit in php ini setting for max execution time at 3600. below is the my code. I need advice on how can i achieve my goal what is best practice and how i can generate QR code in bulk with unique string and avoid duplicating qr code and store the qr image in storage folder as well its filename in database for later use to print it.
$data = \DB::connection('mysql2')->table('basicinfo')->where('is_approved','=','1')->whereIn('classOfMembership',$member_arr)->get(); //this return 4808 records
foreach($data as $d){
$randomStr = Str::random(33);
$image = QrCode::format('png')
->size(200)->errorCorrection('H')
->generate('https://example.com/'.$randomStr);
$time_now = time();
$output_file = '/qr-code/election-'.$request->election_year.'/img-' . $time_now . '.png';
$image_name = 'img-' . $time_now . '.png';
$insert_arr[] = [
'election_year' => $request->election_year,
'position' => $request->post,
'qr_hash' => $randomStr,
'qr_code_image' => $image_name
];
Storage::disk('local')->put($output_file, $image);
}
Ballot::insert($insert_arr);
Related
This is my code in Controller. Now I want to set CustomPaper for printing because A4 size is not Portable for large width file in domPDF.
$category = Category::where('id',$request->category_id)->first();
$file = 'Client Ledger Report - '.date('d-m-Y').'.pdf';
$pdf = PDF::loadView('admin.reports.client_statement_report',compact('records','category','type','from','to'));
return $pdf->download($file);
Just write custom dimension in an array variable and by using setPaper() method and chain it with PDF::loadView()->setPaper($dimension).
Example:
$category = Category::where('id',$request->category_id)->first();
/* Custom Dimensions */
$dimension= array(0,0,680,920);
$file = 'Client Ledger Report - '.date('d-m-Y').'.pdf';
$pdf = PDF::loadView('admin.reports.client_statement_report',compact('records','category','type','from','to'))->setPaper($dimension);
return $pdf->download($file);
I'm trying to loop over multiple servers that need to run 1 at a time using Laravel's withChain. The first job completes just fine but the data I'm passing within the chained jobs gives me the
Attempt to assign property of non-object
When I log out the initial dispatched data it looks just like the constructed data in my array so I'm not sure what I'm doing wrong.
$new_jobs_array = [];
foreach ($this->wasRequest->nodes->sortByDesc('pivot.node_type') as $node) {
if ($node->pivot->node_type != 'WAS_DMGR')
{
$snode = strtolower($node->hostname);
$shortname = strtok($snode, '.');
$fileName = strtolower($mnemonic).'_'.$shortname.'_'.$reqId.'.json';
$sourceJsonPath = base_path() . "/json/was/" . $fileName;
$new_job = 'new BootStrapWasNode('. $node .', '. $this->wasRequest .', '.$sourceJsonPath.')';
array_push($new_jobs_array, $new_job);
} else {
$dmgr_node = $node;
}
}
//Log::info($new_jobs_array);
$dmgr_node_sname = strtok($this->wasRequest->nodes->where('pivot.node_type', 'WAS_DMGR')->pluck('hostname')[0], '.');
$fileName = strtolower($mnemonic).'_'.$dmgr_node_sname.'_'.$reqId.'.json';
$sourceJsonPath = base_path() . "/json/was/" . $fileName;
$this->wasRequest->status = 'Bootstrapping Nodes';
$this->wasRequest->save();
//Log::info("DMGR-------------------".$dmgr_node.", ".$this->wasRequest.", ".$sourceJsonPath);
BootStrapWasNode::withChain($new_jobs_array)->dispatch($dmgr_node, $this->wasRequest, $sourceJsonPath);
I can attach the log view if needed but there is a lot of data for each node. The issue is with the $new_nodes_array, the initial dispatch($dmgr_node,$this->wasRequest,$sourceJsonPath) completes without issue.
Was able to figure out the issue.
This line was incorrect
$new_job = 'new BootStrapWasNode('. $node .', '. $this->wasRequest .','.$sourceJsonPath.')';
It should be
$new_job = new BootStrapWasNode($node, $this->wasRequest, $sourceJsonPath);
i am trying to move five different files in one form at the same time but only one image moves. I have five columns for each filename in my database table. The uploaded image names saves in the database column but the only problem is only one file is moved instead of all the files uploaded. Below is code:
//form isValid...
$post = $form2->getData();
$imageonetmp = $post["imageone"]["tmp_name"];
$imageonename = $post["imageone"]["name"];
$imagetwotmp = $post["imagetwo"]["tmp_name"];
$imagetwoname = $post["imagetwo"]["name"];
$imagethreetmp = $post["imagethree"]["tmp_name"];
$imagethreename = $post["imagethree"]["name"];
$imagefourtmp = $post["imagefour"]["tmp_name"];
$imagefourname = $post["imagefour"]["name"];
$imagefivetmp = $post["imagefive"]["tmp_name"];
$imagefivename = $post["imagefive"]["name"];
$filepath = $this->_dir.$imageonename;
$filepath = $this->_dir.$imagetwoname;
$filepath = $this->_dir.$imagethreename;
$filepath = $this->_dir.$imagefourname;
$filepath = $this->_dir.$imagefivename;
move_uploaded_file($imageonetmp, $filepath);
move_uploaded_file($imagetwotmp, $filepath);
move_uploaded_file($imagethreetmp, $filepath);
move_uploaded_file($imagefourtmp, $filepath);
move_uploaded_file ($imagefivetmp, $filepath);
/*set image names to db here*/
$entityManager->persist($autos);
$entityManager->flush();
Hope someone can help.
You are using the same variable name $filepath to write the files. Meaning you are moving each file but overwriting the previous move.
You should use different variable names e.g. $filepath1, $filepath2, etc...
Or
$filepath = file location...
Move file...
And repeat.
I have some code that uploads the CSV file to the specified folder, but it doesn't update the database.
public function do_upload()
{
$csv_path = realpath(APPPATH . '/../assets/uploads/CSV/');
$config['upload_path'] = $csv_path;
$config['allowed_types'] = '*'; // All types of files allowed
$config['overwrite'] = true; // Overwrites the existing file
$this->upload->initialize($config);
$this->load->library('upload', $config);
if ( ! $this->upload->do_upload('userfile'))
{
$error = array('error' => $this->upload->display_errors());
$this->layout->buffer('content', 'program/upload', $error);
$this->layout->render();
}
else
{
$image_data = $this->upload->data();
$fname = $image_data['file_name'];
$fpath = $image_data['file_path'].$fname;
$fh = fopen($fpath, "r");
$insert_str = 'INSERT INTO wc_program (JobRef, Area, Parish, AbbrWorkType, WorkType, Timing, TrafficManagement, Location, Duration, Start, Finish) VALUES '."\n";
if ($fh) {
// Create each set of values.
while (($csv_row = fgetcsv($fh, 2000, ',')) !== false) {
foreach ($csv_row as &$row) {
$row = strtr($row, array("'" => "\'", '"' => '\"'));
}
$insert_str .= '("'
// Implode the array and fix pesky apostrophes.
.implode('","', $csv_row)
.'"),'."\n";
}
// Remove the trailing comma.
$insert_str = rtrim($insert_str, ",\n");
// Insert all of the values at once.
$this->db->set($insert_str);
echo '<script type="text/javascript">
alert("Document successfully uploaded and saved to the database.");
location = "program/index";
</script>';
}
else {
echo '<script type="text/javascript">
alert("Sorry! Something went wrong please proceed to try again.");
location = "program/upload";
</script>';
}
}
}
When I run var_dump($fh); it shows: resource(89) of type (stream)
When I run var_dump($fpath) it shows: string(66) "/Applications/MAMP/htdocs/site/assets/uploads/CSV/wc_program.csv"
So it all uploads but what is wrong with it not updating the database?
I have tried all kinds of changing the fopen method but still no joy, I really need it to add to the database and the insert query and set query should do the trick but it doesn't.
Any help greatly appreciated!
You are not running any query on the database. You are mixing active record syntax with simple query syntax. The active record insert query will be executed by calling.
$this->db->insert('my_table');
db::set() does not actually query the database. It takes in a key/value pair that will be inserted or updated after db::insert() or db::update() is called. If you build the query yourself you need to use the db::query() function.
Review the active directory documentation.
You can use $this->db->query('put your query here'), but you lose the benefit of CodeIgniter's built in security. Review CodeIgniter's query functions.
I'll give you examples of just a few of the many ways you can insert into a database using CodeIgniter. The examples will generate the query from your comment. You will need to adjust your code accordingly.
EXAMPLE 1:
$result = $this->db
->set('JobRef', 911847)
->set('Area', 'Coastal')
->set('Parish', 'Yapton')
->set('AbbrWorkType', 'Micro')
->set('WorkType', 'Micro-Asphalt Surfacing')
->set('Timing', 'TBC')
->set('TrafficManagement', 'No Positive Traffic Management')
->set('Location', 'Canal Road (added PMI 16/07/12)')
->set('Duration', '2 days')
->set('Start', '0000-00-00')
->set('Finish', '0000-00-00')
->insert('wc_program');
echo $this->db->last_query() . "\n\n";
echo "RESULT: \n\n";
print_r($result);
EXAMPLE 2 (Using an associative array):
$row = array(
'JobRef' => 911847,
'Area' => 'Coastal',
'Parish' => 'Yapton',
'AbbrWorkType' => 'Micro',
'WorkType' => 'Micro-Asphalt Surfacing',
'Timing' => 'TBC',
'TrafficManagement' => 'No Positive Traffic Management',
'Location' => 'Canal Road (added PMI 16/07/12)',
'Duration' => '2 days',
'Start' => '0000-00-00',
'Finish' => '0000-00-00'
);
$this->db->insert('wc_program', $row);
// This will do the same thing
// $this->db->set($row);
// $this->db->insert('wc_program');
echo $this->db->last_query();
Example 1 and 2 are using the Active Record. The information is stored piece by piece and then the query is built when you make the final call. This has several advantages. It allows you to build queries dynamically without worrying about SQL syntax and order of the keywords. It also escapes your data.
EXAMPLE 3 (Simple Query):
$query = 'INSERT INTO
wc_program
(JobRef, Area, Parish, AbbrWorkType, WorkType, Timing, TrafficManagement, Location, Duration, Start, Finish)
VALUES
("911847","Coastal","Yapton","Micro","Micro-Asphalt Surfacing","TBC","No Positive Traffic Management","Canal Road (added PMI 16/07/12)","2 days","0000-00-00","0000-00-00")';
$result = $this->db->query($query);
echo $this->db->last_query() . "\n\n";
echo "RESULT: \n";
print_r($result);
This way leaves all the protection against injection up to you, can lead to more errors, and is harder to change/maintain.
If you are going to do it this way you should use the following syntax, which will protect against injection.
EXAMPLE 4:
$query = 'INSERT INTO
wc_program
(JobRef, Area, Parish, AbbrWorkType, WorkType, Timing, TrafficManagement, Location, Duration, Start, Finish)
VALUES
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);';
$row = array(
911847,
'Coastal',
'Yapton',
'Micro',
'Micro-Asphalt Surfacing',
'TBC',
'No Positive Traffic Management',
'Canal Road (added PMI 16/07/12)',
'2 days',
'0000-00-00',
'0000-00-00'
);
$result = $this->db->query($query, $row);
echo $this->db->last_query() . "\n\n";
echo "RESULT: \n";
print_r($result);
CodeIgniter will replace each "?" in the query with the corresponding value from the array after it is escaped. You can use this to run many queries that are of the same form, but have different data just by updating the $row array and benefit from CI's built in security.
As an example:
I load in the input from a .txt:
Benjamin,Schuvlein,Germany,1912,M,White
I do some code that I will not post here for brevity and get to the link:
https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ
I want to scrape multiple things from that page. In the code below, I only do 1.
I'd also like to make each item be separated by a , in the output .txt.
And, I'd like the output to be preceded by the input.
I'm using the following packages in the code:
use strict;
use warnings;
use WWW::Mechanize::Firefox;
use Data::Dumper;
use LWP::UserAgent;
use JSON;
use CGI qw/escape/;
use HTML::DOM;
Here's the relevant code:
my $ua = LWP::UserAgent->new;
open(my $o, '>', 'out2.txt') or die "Can't open output file: $!";
# Here is the url, although in practice, it is scraped itself using different code
my $url = 'https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ';
print "My URL is <$url>\n";
my $request = HTTP::Request->new(GET => $url);
$request->push_header('Content-Type' => 'application/json');
my $response = $ua->request($request);
die "Error ".$response->code if !$response->is_success;
my $dom_tree = new HTML::DOM;
$dom_tree->write($response->content);
$dom_tree->close;
my $str = $dom_tree->getElementsByTagName('table')->[0]->getElementsByTagName("td")->[10]->as_text();
print $str;
print $o $str;
Desired Output (from that link) is something like:
Benjamin,Schuvlein,Germany,1912,M,White,Queens,New York,Married,Same Place,Head, etc ....
(How much of that output section is scrapable?)
Any help on how to get the link within the link would be much appreciated!
This is fairly simply done using HTML::TreeBuilder::XPath to access the HTML. This program builds a hash of the data using the labels as keys, so any of the desired information can be extracted. I have enclosed in quotes any fields that contain commas or whitespace.
I don't know whether you have the permission of this web site to extract data this way, but I should draw your attention to this X-Copyright header in the HTTP responses. This approach clearly falls under the header of programmatic access.
X-Copyright: COPYRIGHT WARNING Data accessible through the FamilySearch API is protected by copyright. Any programmatic access, reformatting, or rerouting of this data, without permission, is prohibited. FamilySearch considers such unauthorized use a violation of its reproduction, derivation, and distribution rights. Contact devnet (at) familysearch.org for further information.
Am I to expect an email from you? I replied to your first mail but haven't heard since.
use strict;
use warnings;
use URI;
use LWP;
use HTML::TreeBuilder::XPath;
my $url = URI->new('https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ');
my $ua = LWP::UserAgent->new;
my $resp = $ua->get($url);
die $resp->status_line unless $resp->is_success;
my $tree = HTML::TreeBuilder::XPath->new_from_content($resp->decoded_content);
my #results = $tree->findnodes('//table[#class="result-data"]//tr[#class="result-item"]');
my %data;
for my $item (#results) {
my ($key, $val) = map $_->as_trimmed_text, $item->content_list;
$key =~ s/:$//;
$data{$key} = $val;
}
my $record = join ',', map { local $_ = $data{$_}; /[,\s]/ ? qq<"$_"> : $_ }
'name', 'birthplace', 'estimated birth year', 'gender', 'race (standardized)',
'event place', 'marital status', 'residence in 1935',
'relationship to head of household (standardized)';
print $record, "\n";
output
"Benjamin Schuvlein",Germany,1912,Male,White,"Assembly District 2, Queens, New York City, Queens, New York, United States",Married,"Same Place",Head
Try this
use LWP::Simple;
use LWP::UserAgent;
use HTML::TableExtract;
$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0;
$ua = LWP::UserAgent->new;
$ua->agent("Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.91 Safari/537.11");
$req = HTTP::Request->new(GET => "https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ");
$res = $ua->request($req);
$content = $res->content;
#$content = get("https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ") or die "Couldn't get it! $!";
$te = HTML::TableExtract->new( attribs => { 'class' => 'result-data' } );
# $te = HTML::TableExtract->new( );
$te->parse($content);
$table = $te->first_table_found;
# print $content; exit;
# $te->tables_dump(1);
#print Dumper($te);
#print Dumper($table);
print $table->cell(4,0) . ' = ' . $table->cell(4,1), "\n"; exit;
Which prints out
event place: = Assembly District 2, Queens, New York City, Queens, New York, United States
I also noticed this header:
X-Copyright:COPYRIGHT WARNING Data accessible through the FamilySearch API is protected by copyright. Any programmatic access, reformatting, or rerouting of this data, without permission, is prohibited. FamilySearch considers such unauthorized use a violation of its reproduction, derivation, and distribution rights. Contact devnet (at) familysearch.org for further information.
See also http://metacpan.org/pod/HTML::Element#SYNOPSIS
I thought I had answered your question.
The problem is that you are trying to fetch the webpage with LWP. Why are try to doing that if you already have WWW::Mechanize::Firefox?
Did you tried this?
It will retrieve and save each link for further analyses. A small change and you 'get' the DOM tree. Sorry, I do not have acccess to this page, so I just hope it will work.
my $i=1;
for my $link (#links) {
print Dumper $link->url;
print Dumper $link->text;
my $tempfile = './$i.html';$i++;
$mech->get( $link, ':content_file' => $tempfile, synchronize => 1 );
my $dom_tree = $mech->document();
my $str = $dom_tree->getElementsByTagName('table')->[0]->getElementsByTagName("td")->[9]->as_text();
}
EDIT:
Process the page content with regexp (Everyone: Please remember, there is always more than one way to do something wwith Perl!. It works, it is easy...)
it tried it out with this cmd:
wget -nd 'https://familysearch.org/pal:/MM9.1.1/K3BN-LLJ' -O 1.html|cat 1.html|1.pl
use Data::Dumper;
use strict;
use warnings;
local $/=undef;
my $html = <>;#read from file
#$html = $mech->content( format => 'html' );# read data from mech object
my $data = {};
my $current_label = "not_defined";
while ($html =~ s!(<td[^>]*>.*?</td>)!!is){ # process each TD
my $td = $1;
print "td: $td\n";
my $td_val = $td;
$td_val =~ s!<[^>]*>!!gis;
$td_val =~ s!\s+! !gs;
$td_val =~ s!(\A\s+|\s+\z)!!gs;
if ($td =~ m!result-label!){ #primitive state machine, store the current label
print "current_label: $current_label\n";
$current_label = $td_val;
} elsif ($td =~ m!result-value!){ #add each data to current label
push(#{$data->{$current_label}},$td_val);
} else {
warn "found something else: $td\n";
}
}
#process it using a white lists of known entries (son,race, etc).Delete from the result if you find it on white list, die if you find something new.
#multi type
foreach my $type (qw(son wife daughter head)){
process_multi($type,$data->{$type});
delete($data->{$type});
}
#simple type
foreach my $type (qw(birthplace age)){
process_simple($type,$data->{$type});
delete($data->{$type});
}
die "Unknown label!".Dumper($data) if scalar(keys %{$data})>0;
Output:
'line number:' => [
'28'
],
'estimated birth year:' => [
'1912'
],
'head' => [
'Benjamin Schuvlein',
'M',
'28',
'Germany'
],