How to upload large csv file into mysql fastly in codeigniter? - codeigniter

I am using codeigniter framework. I want to upload larg csv file into mysql database fastly. The file contains 87000 rows. I have added for uploading the data, but it is loading morethan 1 hour. i need to optimize this time.Please suggest me.
if(move_uploaded_file($_FILES['file_content']['tmp_name'],$path.$actual_file_name))
{
if (($handle = fopen($path.$actual_file_name, "r")) !== FALSE)
{
while (($data = fgetcsv($handle,100000, ",")) !== FALSE)
{
$insert_batch=array(
'unique_transactionid' => mb_check_encoding($data[0], 'UTF-8') ? $data[0] : utf8_encode($data[0]),
'data-year' => mb_check_encoding($data[2], 'UTF-8') ? $data[2] : utf8_encode($data[2]),
'supplier_group' => mb_check_encoding($data[4], 'UTF-8') ? $data[4] : utf8_encode($data[4]),
);
this->db->insert('table name', $insert_batch);
}
}
}
I need the data is uploading fastly. Currently our server max_excecution time is 10 mins. It excecutes more than 10 mins and the page is not redirect to my page and at last i got page not found error. Please help me to upload fastly.

take the insert() function out of the while loop and use batch_insert() instead. What you are doing right now, is creating the array and insert, for each of the rows.
$i=0;
while (($data = fgetcsv($handle,100000, ",")) !== FALSE)
{
$insert_batch[$i]=array(
'unique_transactionid' => mb_check_encoding($data[0], 'UTF-8') ? $data[0] : utf8_encode($data[0]),
'data-year' => mb_check_encoding($data[2], 'UTF-8') ? $data[2] : utf8_encode($data[2]),
'supplier_group' => mb_check_encoding($data[4], 'UTF-8') ? $data[4] : utf8_encode($data[4]),
);
$i++;
}
$this->db->insert_batch('table_name', $insert_batch);

Related

Laravel issue Credentials are required to create a Client

I need to test send SMS to mobile I get Credentials are required to create a Client error for My Code Here
.env
TWILIO_ACCOUNT_SID=AC15...................
TWILIO_AUTH_TOKEN=c3...................
TWILIO_NUMBER=+1111...
Config\App
'twilio' => [
'TWILIO_AUTH_TOKEN' => env('TWILIO_AUTH_TOKEN'),
'TWILIO_ACCOUNT_SID' => env('TWILIO_ACCOUNT_SID'),
'TWILIO_NUMBER' => env('TWILIO_NUMBER')
],
Controller
$accountSid = env('TWILIO_ACCOUNT_SID');
$authToken = env('TWILIO_AUTH_TOKEN');
$twilioNumber = env('TWILIO_NUMBER');
$client = new Client($accountSid, $authToken);
try {
$client->messages->create(
'0020109.....',
[
"body" => 'test',
"from" => $twilioNumber
// On US phone numbers, you could send an image as well!
// 'mediaUrl' => $imageUrl
]
);
Log::info('Message sent to ' . $twilioNumber);
} catch (TwilioException $e) {
Log::error(
'Could not send SMS notification.' .
' Twilio replied with: ' . $e
);
}
Twilio developer evangelist here.
A quick read over the environment config for Laravel suggests to me that you can use the env method within your config files, as you are doing, but it's not necessarily available in application code. Since you are committing your environment variables to the config object, I think you need to use the config method instead.
$accountSid = config('TWILIO_ACCOUNT_SID');
$authToken = config('TWILIO_AUTH_TOKEN');
$twilioNumber = config('TWILIO_NUMBER');
Let me know if that helps at all.

New Caller Insert to Database with Codeigniter using the Twilio API

Below is the function to receive all incoming calls in my Controller
public function call_incoming()
{
$blocklist = $this->call_log_model->get_blocklist($_REQUEST['From']);
$tenantNum = $this->call_log_model->get_called_tenant($_REQUEST['From']);
$tenantInfoByNumber = $this->account_model->getTenantInfoByNumber($tenantNum->to_tenant);
$officeStatus = $this->check_office_hours($tenantInfoByNumber->start_office_hours, $tenantInfoByNumber->end_office_hours);
$calldisposition = $this->calldisp_model->get_call_disposition($tenantInfoByNumber->user_id);
$response = new Services_Twilio_Twiml;
if($blocklist == 0)
{
if($officeStatus == "open")
{
if($_POST['Called'] != AGENTPOOL_NUM)
{
$data = array(
'caller'=>$_REQUEST['From'],
'to_tenant'=>$_POST['Called'],
'date_created'=>date('Y-m-d H:i:s')
);
$this->call_log_model->insert_caller_to_tenant($data);
$dial = $response->dial(NULL, array('callerId' => $_REQUEST['From']));
$dial->number(AGENTPOOL_NUM);
print $response;
}
else
{
$gather = $response->gather(array('numDigits' => 1, 'action'=>HTTP_BASE_URL.'agent/call_controls/call_incoming_pressed', 'timeout'=>'5' , 'method'=>'POST'));
$ctr = 1;
foreach($calldisposition as $val )
{
$gather->say('To go to '.$val->disposition_name.', press '.$ctr, array('voice' => 'alice'));
$gather->pause("");
$ctr++;
}
print $response;
}
}
else
{
$response->say('Thank you for calling. Please be advise that our office hours is from '.$tenantInfoByNumber->start_office_hours.' to '.$tenantInfoByNumber->end_office_hours);
$response->hangup();
print $response;
}
}
else
{
$response->say('This number is blocked. Goodbye!');
$response->hangup();
print $response;
}
}
Please advise if I need to post the model...
Here is whats happening everytime an unknown number calls in, the caller will hear an application error has occurred error message and when checking the Twilio console the error it is giving me is
A PHP Error was encountered
Severity: Notice
Message: Trying to get property of non-object
Filename: agent/Call_controls.php
Line Number: 357
Please be advised that this error only occurs when the caller is a number not in our database yet. When the call comes from a number already saved in our databse, this codes works...
Thank you for the help...
if($tenantNum) {
$tenantInfoByNumber = $this->account_model->getTenantInfoByNumber($tenantNum->to_t‌​enant);
} else {
$tenantInfoByNumber = ""; // fill this in with relevant fill data
}
This should fix your issue, as there is no TenantNum returned, there is no data, so make it yourself for unknown numbers.

How to set up two memcached server hosts in Joomla cache setting?

I have set up two instances of memcached running on two servers. How do I set up the hosts in Joomla cache setting?
You have to edit the configuration.php file at the site root folder and add the server details. Find the cache varibles $caching, $cache_handler, $memcache_server_host, $memcache_server_port.
Set them as follows
public $caching = '2';
public $cache_handler = 'memcache';
public $memcache_server_host = 'localhost';//Give your memcache server address
public $memcache_server_port = '11111'; //Memcache server port
In the libraries/joomla/cache/storage/memcache.php file I found this lines
/*
* This will be an array of loveliness
***#todo: multiple servers***
* $servers = (isset($params['servers'])) ? $params['servers'] : array();
*/
As you wanted to add two different memcache servers, you can see that it is still under development. But there is a way ---By hacking core files.
DONT TRY THE METHOD GIVEN BELOW IN PRODUCTION ENVIRONMENT
Though it is dangerous to play with core files but you can do at your own risk by following the method that I will suggest now. take backup first.
Take a backup of the files(configuration.php and libraries/joomla/cache/storage/memcache.php and libraries\vendor\joomla\session\Joomla\Session\Storage\Memcache.php). In the memcache file search for this code in the file
$server['host'] = $config->get('memcache_server_host', 'localhost');
$server['port'] = $config->get('memcache_server_port', 11211);
// Create the memcache connection
self::$_db = new Memcache;
self::$_db->addServer($server['host'], $server['port'], $this->_persistent);
$memcachetest = #self::$_db->connect($server['host'], $server['port']);
if ($memcachetest == false)
{
throw new RuntimeException('Could not connect to memcache server', 404);
}
Add this code below
//Give second server details like this
$server['host2'] = $config->get('memcache_server_host2', 'localhost');
$server['port2'] = $config->get('memcache_server_port2', 11211);
// Create the memcache connection
self::$_db = new Memcache;
self::$_db->addServer($server['host2'], $server['port2'], $this->_persistent);
$memcachetest = #self::$_db->connect($server['host2'], $server['port2']);
if ($memcachetest == false)
{
throw new RuntimeException('Could not connect to memcache server', 404);
}
In the configuration.php file add this lines
public $memcache_server_host2 = 'localhost';//Give your second memcache server address
public $memcache_server_port2 = '11111'; //Memcache second server port
Now for storing sessions you have to edit files that stores session cache
In the file libraries\vendor\joomla\session\Joomla\Session\Storage\Memcache.php you will find these line
$this->_servers = array(
array(
'host' => isset($options['memcache_server_host']) ? $options['memcache_server_host'] : 'localhost',
'port' => isset($options['memcache_server_port']) ? $options['memcache_server_port'] : 11211
)
);
Change this to
$this->_servers = array(
array(
'host' => isset($options['memcache_server_host']) ? $options['memcache_server_host'] : 'localhost',
'port' => isset($options['memcache_server_port']) ? $options['memcache_server_port'] : 11211
),
array(
'host2' => isset($options['memcache_server_host2']) ? $options['memcache_server_host2'] : 'localhost',
'port2' => isset($options['memcache_server_port2']) ? $options['memcache_server_port2'] : 11211
)
);

Filling data in Ext.net FormPanel

I am using Ext.net 2.0 and I am trying to load the first record of the store inside a form panel. I always get no records (getCount() = 0) in the store ? Am I missing something ?
#(Html.X().Store()
.ID("myStore")
.AutoSync(true)
.AutoDataBind(true)
.Proxy(proxy =>
proxy.Add(
Html.X().AjaxProxy().API(api =>
{
api.Create = "/Property/Save/";
api.Read = "/Property/GetById/";
})
.Reader(reader => reader.Add(Html.X().JsonReader().Root("data").IDProperty("P_ID")))
.Writer(writer => writer.Add(Html.X().JsonWriter().AllowSingle(true)))
))
.Listeners(c =>
{
c.DataChanged.Handler ="var store = Ext.getStore('myStore');" +
"alert(store.getCount());";
})
.AutoLoadParams(parameters =>
{
parameters.Add(Html.X().Parameter().Name("id").Value("document.location.href.split('/')[5]").Mode(ParameterMode.Raw));
})
.Model(model => model.Add(
Html.X().Model()
.Fields(fields =>
{
fields.Add(Html.X().ModelField().Name("ID").Type(ModelFieldType.Int));
fields.Add(Html.X().ModelField().Name("DispalyName").Type(ModelFieldType.String));
fields.Add(Html.X().ModelField().Name("Title").Type(ModelFieldType.String));
fields.Add(Html.X().ModelField().Name("ShortDescription").Type(ModelFieldType.String));
})
))
)
For the form panel
#(
Html.X().FormPanel()
.ID("myPanel")
.Layout(LayoutType.Form)
.Width(350)
.FieldDefaults(d => {
d.LabelWidth = 150;
})
.BodyPadding(10)
.Items(item =>
{
item.Add(Html.X().TextField().ID("Id").Name("ID").FieldLabel("Id").Hidden(true));
item.Add(Html.X().TextField().ID("DispalyName").Name("IdDispalyName").FieldLabel("Id Dispaly Name").MaxLength(400));
item.Add(Html.X().TextField().ID("Title").Name("Title").FieldLabel("Title").AllowBlank(false).MaxLength(200));
item.Add(Html.X().TextField().ID("ShortDescription").Name("ShortDescription").FieldLabel("Short Description").MaxLength(200));
}
) )
Thanks in advance.
More appropriate event is Load (it is fired when data is loaded to the store from a remote source)
See the following description
http://docs.sencha.com/ext-js/4-1/#!/api/Ext.data.Store-event-datachanged

use symfony form validator for Bulk-Insert

i receive an CSV File from the User and I want to insert it into the Database. I want to use the Form Validation:
$pf = new ProductForm();
$old_memory_usage = 0;
while($data = $csvreader->read())
{
$mem = memory_get_usage() / 1024;
echo "Memory-Usage: " . $mem . " KB; Mem-Diff: " . $mem - $old_memory_usage . "\n";
$old_memory_usage = $mem;
$p = new Product();
$p->fromArray($data);
$pf->bind($p->toArray(), array());
if($pf->isValid())
{
$p->save();
}
else
{
//display error message to the user
}
}
This works fine when isValid() returns true. The Memory Difference is 0 KB. But if there is an Error in the "form" the memory-difference is 6-7 KB. So I get an allowed_memory error when the CSV-File is very huge.
I tryed to free the form like:
unset($pf)
$pf = null;
$pf = new ProductForm();
no success. It is just worse!
ans ideas?
thx!
You can disable doctrine profiler in your databse yml:
dev:
doctrine:
class: sfDoctrineDatabase
param:
profiler: false

Resources