How to restore the database double encoded by mysqldump - utf-8

I use the mysqldump to make a backup of my database.
My database was destroyed by an accident and now I want to restore it.
But the SQL file is double encoded by bug#28969.
http://bugs.mysql.com/bug.php?id=28969
Is there any solution for my data to go back?
I only have the SQL file made by mysqldump.
Thanks.
I got my data back. Thanks everyone.
By this way,
1.import the messy data
2.use sqldump as 'mysqldump -h "$DB_HOST -u "$DB_USER" -p"$DB_PASSWORD" --opt --quote-names --skip-set-charset --default-character-set=latin1 "$DB_NAME" > /tmp/temp.sql'
Reference
http://pastebin.com/iSwVPk1w

I got my data back. Thanks everyone.
By this way,
1.import the messy data
2.use sqldump as mysqldump -h "$DB_HOST -u "$DB_USER" -p"$DB_PASSWORD" --opt --quote-names --skip-set-charset --default-character-set=latin1 "$DB_NAME" > /tmp/temp.sql
Reference
#!/bin/bash -e
DB_HOST="$1"
DB_USER="$2"
DB_PASSWORD="$3"
DB_NAME="$4"
mysqldump -h "$DB_HOST -u "$DB_USER" -p"$DB_PASSWORD" --opt --quote-names \
--skip-set-charset --default-character-set=latin1 "$DB_NAME" > /tmp/temp.sql
mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASSWORD" \
--default-character-set=utf8 "$DB_NAME" < /tmp/temp.sql

If it's just doubling the UTF-8 bytes or prepending something, I'd suggest putting together a quick sed/awk command to match and correct them.
http://www.osnews.com/story/21004/Awk_and_Sed_One-Liners_Explained
If you're not comfortable with this, any scripting language with regex support could be used to easily do the same thing, though it might take a few more minutes.

If Your DB contain correct Collation but full data in DB are Doubly Encoded then this will help you remember you only execute its once and take backup of your DB.
<?php
/**
* DoublyEncodeCorrection.php
*
* NOTE: Look for 'TODO's for things you may need to configure.
* PHP Version 5
*
*/
ini_set('display_errors','1');
//error_reporting(E_ALL ^ E_NOTICE ^ E_DEPRECATED);
// TODO: Pretend-mode -- if set to true, no SQL queries will be executed. Instead, they will only be echo'd
// to the console.
$pretend = true;
// TODO: Should SET and ENUM columns be processed?
$processEnums = false;
// TODO: The collation you want to convert the overall database to
$defaultCollation = 'utf8_general_ci';
// TODO Convert column collations and table defaults using this mapping
// latin1_swedish_ci is included since that's the MySQL default
$collationMap = array(
'latin1_bin' => 'utf8_bin',
'latin1_general_ci' => 'utf8_general_ci',
'latin1_swedish_ci' => 'utf8_general_ci'
);
$mapstring = '';
foreach ($collationMap as $s => $t) {
$mapstring .= "'$s',";
}
$mapstring = substr($mapstring, 0, -1); // Strip trailing comma
//echo $mapstring;
// TODO: Database information
$dbHost = 'localhost';
$dbName = 'tina';
$dbUser = 'root';
$dbPass = 'root';
// Open a connection to the information_schema database
$infoDB = mysql_connect($dbHost, $dbUser, $dbPass);
mysql_select_db('information_schema', $infoDB);
// Open a second connection to the target (to be converted) database
$targetDB = mysql_connect($dbHost, $dbUser, $dbPass, true);
mysql_select_db($dbName, $targetDB);
if (!is_resource($targetDB)) {
echo "Could not connect to db!: " . mysql_error();exit;
}
if (mysql_select_db($dbName, $targetDB) === FALSE) {
echo "Could not select database!: " . mysql_error();exit;
}
//
// TODO: FULLTEXT Indexes
//
// You may need to drop FULLTEXT indexes before the conversion -- execute the drop here.
// eg.
// sqlExec($targetDB, "ALTER TABLE MyTable DROP INDEX `my_index_name`", $pretend);
//
// If so, you should restore the FULLTEXT index after the conversion -- search for 'TODO'
// later in this script.
//
// Get all tables in the specified database
$tables = sqlObjs($infoDB,
"SELECT TABLE_NAME, TABLE_COLLATION
FROM TABLES
WHERE TABLE_SCHEMA = '$dbName'");
foreach ($tables as $table) {
$tableName = $table->TABLE_NAME;
$tableCollation = $table->TABLE_COLLATION;
// Find all columns that aren't of the destination collation
$cols = sqlObjs($infoDB,
"SELECT *
FROM COLUMNS
WHERE TABLE_SCHEMA = '$dbName'
AND TABLE_Name = '$tableName'
");
$intermediateChanges = array();
$finalChanges = array();
foreach ($cols as $col) {
// If this column doesn't use one of the collations we want to handle, skip it
if (in_array($col->COLLATION_NAME, $collationMap)) {
//echo "<pre>";print_r($col->COLUMN_NAME);exit;
sqlExec($targetDB,"UPDATE $dbName.$tableName SET $col->COLUMN_NAME = CONVERT(CAST(CONVERT($col->COLUMN_NAME USING latin1) AS BINARY) USING utf8)") ;
}
}
}
/**
* Executes the specified SQL
*
* #param object $db Target SQL connection
* #param string $sql SQL to execute
* #param boolean $pretend Pretend mode -- if set to true, don't execute query
*
* #return SQL result
*/
function sqlExec($db, $sql, $pretend = false)
{
echo "$sql;\n";
if ($pretend === false) {
$res = mysql_query($sql, $db);
//echo "<pre>";print_r($res);exit;
$error = mysql_error($db);
if ($error !== '') {
print "!!! ERROR: $error\n";
}
return $res;
}
return false;
}
/**
* Gets the SQL back as objects
*
* #param object $db Target SQL connection
* #param string $sql SQL to execute
*
* #return SQL objects
*/
function sqlObjs($db, $sql)
{
$res = sqlExec($db, $sql);
$a = array();
if ($res !== false) {
while ($obj = mysql_fetch_object($res)) {
$a[] = $obj;
}
}
return $a;
}
?>

Related

Codeigniter Call to undefined method CI_DB_mysqli_result::fetch_assoc()

hello i want to update the row by clicking on edit button, but it giving me this error, kindly help
function updateUser()
{
$data = stripslashes(file_get_contents("php://input"));
$mydata = json_decode($data, true);
$id = $mydata['sid'];
//retrieve specific info
$sql = "SELECT * FROM admin WHERE id = {$id}";
$result = $this->db->query($sql);
$row = $result->fetch_assoc();
echo json_encode($row); //returning json formate
}
Use $result->row_array(); instead of $result->fetch_assoc();
To prevent sql injection, you may also want to bind the $id separately instead of concatenating it into the query, like this:
$sql = "SELECT * FROM admin WHERE id = ?";
$result = $this->db->query($sql, array($id));
$row = $result->row_array();
See also: https://codeigniter.com/userguide3/database/queries.html#query-bindings

joomla tag form loads whole list from db

I noticed that joomla tag input field is quite stupid. It loads everything from db, in this case 9K tags. Obviously ui becomes so slow.
Any ideas how to fix it? it seems there is already an ajax functionality present, so why not rely on that completely? J ways are crazy.
1 idea is to modify getOption method, and load only the tags that are related to current article editor is editing.
But in this context I don't seem to have article id.
Any ideas how to solve situation? I'm sure some of you've run into this :S
/**
* Method to get a list of tags
*
* #return array The field option objects.
*
* #since 3.1
*/
protected function getOptions()
{
$published = $this->element['published']? $this->element['published'] : array(0,1);
$db = JFactory::getDbo();
$query = $db->getQuery(true)
->select('DISTINCT a.id AS value, a.path, a.title AS text, a.level, a.published, a.lft')
->from('#__tags AS a')
->join('LEFT', $db->qn('#__tags') . ' AS b ON a.lft > b.lft AND a.rgt < b.rgt');
// Filter language
if (!empty($this->element['language']))
{
$query->where('a.language = ' . $db->q($this->element['language']));
}
$query->where($db->qn('a.lft') . ' > 0');
// Filter on the published state
if (is_numeric($published))
{
$query->where('a.published = ' . (int) $published);
}
elseif (is_array($published))
{
JArrayHelper::toInteger($published);
$query->where('a.published IN (' . implode(',', $published) . ')');
}
$query->order('a.lft ASC');
// Get the options.
$db->setQuery($query);
try
{
$options = $db->loadObjectList();
}
catch (RuntimeException $e)
{
return false;
}
// Block the possibility to set a tag as it own parent
if ($this->form->getName() == 'com_tags.tag')
{
$id = (int) $this->form->getValue('id', 0);
foreach ($options as $option)
{
if ($option->value == $id)
{
$option->disable = true;
}
}
}
// Merge any additional options in the XML definition.
$options = array_merge(parent::getOptions(), $options);
// Prepare nested data
if ($this->isNested())
{
$this->prepareOptionsNested($options);
}
else
{
$options = JHelperTags::convertPathsToNames($options);
}
return $options;
}
So I've modified list that gets preloaded, only to load tags that are present in the article (saved as belonging to article). Autocomplete still works with ajax so no loss of functionality there.

Using JFactory::getDbo()->insertObject with on duplicate key update

How to use:
JFactory::getDbo()->insertObject('#__card_bonus', $object);
with on duplicate key update ?
You have a few options:
1) Check for an entity id. This is my preferred option, because it only uses a single query, is reusable for any object, and is database agnostic - meaning it will work on whichever DBMS you choose, whereas the other two options are exclusive to MySQL.
if (isset($object->id)) {
$db->updateObject('#__card_bonus', $object);
}
else {
$db->insertObject('#__card_bonus', $object, 'id');
}
I often create an abstract model with a save(stdClass $object) method that does this check so I don't have to duplicate it.
2) Write your own query using the MySQL ON DUPLICATE KEY UPDATE syntax, which is a proprietary extension to the SQL standard, that you have demonstrated understanding of.
3) Write your own query using MySQL's proprietary REPLACE INTO extension.
<?php
$jarticle = new stdClass();
$jarticle->id = 1544;
$jarticle->title = 'New article';
$jarticle->alias = JFilterOutput::stringURLSafe($jarticle->title);
$jarticle->introtext = '<p>re</p>';
$jarticle->state = 1;
$jarticle->catid = 13;
$jarticle->created_by = 111;
$jarticle->access = 1;
$jarticle->language = '*';
$db = JFactory::getDbo();
try {
$query = $db->getQuery(true);
$result = JFactory::getDbo()->insertObject('#__content', $jarticle);
}
catch (Exception $e){
$result = JFactory::getDbo()->updateObject('#__content', $jarticle, 'id');
}
I use this method - are not fully satisfied, but ...
or for not object method:
$query = $db->getQuery(true);
$columns = array('username', 'password');
$values = array($db->quote($username), $db->quote($password));
$query
->insert($db->quoteName('#__db_name'))
->columns($db->quoteName($columns))
->values(implode(',', $values));
$query .= ' ON DUPLICATE KEY UPDATE ' . $db->quoteName('password') . ' = ' . $db->quote($password);
$db->setQuery($query);
JFactory::getDbo()->insertObject('#__card_bonus', $object, $keyName);
The name of the primary key. If provided the object property is updated.
Joomla doc ...

How to add a custom field to the session table

I'm currently using Symfony 2.1.8 and the built-in PdoSessionHandler.
I want to add a user_id field in the session table to identify to which (logged-in) user the session belongs. The idea is that I can force the user to log back in destroying his session. In my case it will happen if the privileges of the user are updated.
I had a look to the build-in PdoSessionHandler that you can't extends because of those silly private variables.
So I've tried created a new one (copy/paste) and add my column user_id.
Now this column can be null if the user is not logged in (anonymous users).
So I want to write this user_id in the write method of the handler. The user is already stored in the $data so I was thinking that I could check if this user exists, grab its id and add it in the insert / update query.
The problem is that $data is encoded - I guess by session_encode() - so I'm not sure anymore this is the best place ever to handle my new field, but at the same time I can't see anywhere else I could do it as I need to update this MySQL query to insert the value of the new field.
So my question is: where is the best place to handle this additional field? And how to set this user_id value?
On another note, somehing really annoying is that Symfony is creating a new cookie each time I'm logging in or out. So the database ends up with lots of records for nothing (it's always the same user). Why Symfony is not using the same cookie value all the time?
You could extend PdoSessionHandler (solution for >=Symfony 2.1):
namespace Acme\DemoBundle\HttpFoundation\Session\Storage\Handler;
use Symfony\Component\HttpFoundation\Session\Storage\Handler\PdoSessionHandler;
use Symfony\Component\Security\Core\SecurityContext;
class UserIdPdoSessionHandler extends PdoSessionHandler
{
/**
* #var \PDO PDO instance.
*/
private $pdo;
/**
* #var array Database options.
*/
private $dbOptions;
/**
* #var SecurityContext
*/
private $context;
public function __construct(\PDO $pdo, array $dbOptions = array(), SecurityContext $context)
{
$this->pdo = $pdo;
$this->dbOptions = array_merge(
array('db_user_id_col' => 'user_id'),
$dbOptions
);
$this->context = $context;
parent::__construct($pdo, $dbOptions);
}
public function read($id)
{
// get table/columns
$dbTable = $this->dbOptions['db_table'];
$dbDataCol = $this->dbOptions['db_data_col'];
$dbIdCol = $this->dbOptions['db_id_col'];
try {
$sql = "SELECT $dbDataCol FROM $dbTable WHERE $dbIdCol = :id";
$stmt = $this->pdo->prepare($sql);
$stmt->bindParam(':id', $id, \PDO::PARAM_STR);
$stmt->execute();
// it is recommended to use fetchAll so that PDO can close the DB cursor
// we anyway expect either no rows, or one row with one column. fetchColumn, seems to be buggy #4777
$sessionRows = $stmt->fetchAll(\PDO::FETCH_NUM);
if (count($sessionRows) == 1) {
return base64_decode($sessionRows[0][0]);
}
// session does not exist, create it
$this->createNewSession($id);
return '';
} catch (\PDOException $e) {
throw new \RuntimeException(sprintf('PDOException was thrown when trying to read the session data: %s', $e->getMessage()), 0, $e);
}
}
/**
* {#inheritDoc}
*/
public function write($id, $data)
{
// get table/column
$dbTable = $this->dbOptions['db_table'];
$dbDataCol = $this->dbOptions['db_data_col'];
$dbIdCol = $this->dbOptions['db_id_col'];
$dbTimeCol = $this->dbOptions['db_time_col'];
$dbUserIdCol = $this->dbOptions['db_user_id_col'];
//session data can contain non binary safe characters so we need to encode it
$encoded = base64_encode($data);
$userId = $this->context->isGranted('IS_AUTHENTICATED_REMEMBERED') ?
$this->context->getToken()->getUser()->getId() :
null
;
try {
$driver = $this->pdo->getAttribute(\PDO::ATTR_DRIVER_NAME);
if ('mysql' === $driver) {
// MySQL would report $stmt->rowCount() = 0 on UPDATE when the data is left unchanged
// it could result in calling createNewSession() whereas the session already exists in
// the DB which would fail as the id is unique
$stmt = $this->pdo->prepare(
"INSERT INTO $dbTable ($dbIdCol, $dbDataCol, $dbTimeCol, $dbUserIdCol) VALUES (:id, :data, :time, :user_id) " .
"ON DUPLICATE KEY UPDATE $dbDataCol = VALUES($dbDataCol), $dbTimeCol = VALUES($dbTimeCol)"
);
$stmt->bindParam(':id', $id, \PDO::PARAM_STR);
$stmt->bindParam(':data', $encoded, \PDO::PARAM_STR);
$stmt->bindValue(':time', time(), \PDO::PARAM_INT);
$stmt->bindParam(':user_id', $userId, \PDO::PARAM_STR);
$stmt->execute();
} elseif ('oci' === $driver) {
$stmt = $this->pdo->prepare("MERGE INTO $dbTable USING DUAL ON($dbIdCol = :id) ".
"WHEN NOT MATCHED THEN INSERT ($dbIdCol, $dbDataCol, $dbTimeCol, $dbUserIdCol) VALUES (:id, :data, sysdate, :user_id) " .
"WHEN MATCHED THEN UPDATE SET $dbDataCol = :data WHERE $dbIdCol = :id");
$stmt->bindParam(':id', $id, \PDO::PARAM_STR);
$stmt->bindParam(':data', $encoded, \PDO::PARAM_STR);
$stmt->bindParam(':user_id', $userId, \PDO::PARAM_STR);
$stmt->execute();
} else {
$stmt = $this->pdo->prepare("UPDATE $dbTable SET $dbDataCol = :data, $dbTimeCol = :time WHERE $dbIdCol = :id");
$stmt->bindParam(':id', $id, \PDO::PARAM_STR);
$stmt->bindParam(':data', $encoded, \PDO::PARAM_STR);
$stmt->bindValue(':time', time(), \PDO::PARAM_INT);
$stmt->execute();
if (!$stmt->rowCount()) {
// No session exists in the database to update. This happens when we have called
// session_regenerate_id()
$this->createNewSession($id, $data);
}
}
} catch (\PDOException $e) {
throw new \RuntimeException(sprintf('PDOException was thrown when trying to write the session data: %s', $e->getMessage()), 0, $e);
}
return true;
}
private function createNewSession($id, $data = '')
{
// get table/column
$dbTable = $this->dbOptions['db_table'];
$dbDataCol = $this->dbOptions['db_data_col'];
$dbIdCol = $this->dbOptions['db_id_col'];
$dbTimeCol = $this->dbOptions['db_time_col'];
$dbUserIdCol = $this->dbOptions['db_user_id_col'];
$userId = $this->context->isGranted('IS_AUTHENTICATED_REMEMBERED') ?
$this->context->getToken()->getUser()->getId() :
null
;
$sql = "INSERT INTO $dbTable ($dbIdCol, $dbDataCol, $dbTimeCol, $dbUserIdCol) VALUES (:id, :data, :time, :user_id)";
//session data can contain non binary safe characters so we need to encode it
$encoded = base64_encode($data);
$stmt = $this->pdo->prepare($sql);
$stmt->bindParam(':id', $id, \PDO::PARAM_STR);
$stmt->bindParam(':data', $encoded, \PDO::PARAM_STR);
$stmt->bindValue(':time', time(), \PDO::PARAM_INT);
$stmt->bindParam(':user_id', $userId, \PDO::PARAM_STR);
$stmt->execute();
return true;
}
}
And configure session to use it:
# config.yml
framework:
session:
# ...
handler_id: session.storage.custom
parameters:
pdo.db_options:
db_table: session
db_id_col: session_id
db_data_col: session_value
db_time_col: session_time
db_user_id_col: session_user_id
services:
pdo:
class: PDO
arguments:
dsn: "mysql:host=%database_host%;dbname=%database_name%"
user: "%database_user%"
password: "%database_password%"
session.storage.custom:
class: Acme\DemoBundle\HttpFoundation\Session\Storage\Handler\UserIdPdoSessionHandler
arguments: [ #pdo, "%pdo.db_options%", #security.context ]
I'm not sure modifying session is a good idea, you can store session id(s) in user entity instead and delete those when needed. This way you can for example ensure only user can be logged in with only one session at a time.
The easiest way to accomplish it would be to use login listener.
Add sessionId field to user entity (or document or whatever persistance you use):
// Acme/UserBundle/Entity/User.php
namespace Acme\UserBundle\Entity;
use Doctrine\ORM\Mapping as ORM;
/**
* ORM\Entity
* #ORM\Table(name="fos_user")
*/
class User {
// ...
/**
* #ORM\Column(name="session_id", type="string")
*/
private $sessionId;
public function getSessionId() {
return $this->sessionId;
}
public function setSessionId($sessionId = null) {
$this->sessionId = $sessionId;
return $this;
}
}
And add a listener:
namespace Dbla\UserBundle\Listener;
use Symfony\Component\HttpFoundation\Session;
use Symfony\Component\Security\Http\Event\InteractiveLoginEvent;
class LoginListener
{
protected $doctrine;
protected $session;
public function __construct(Session $session, Registry $doctrine)
{
$this->doctrine = $doctrine;
$this->session = $session;
}
public function onLogin(InteractiveLoginEvent $event)
{
$user = $event->getAuthenticationToken()->getUser();
if ($user) {
$user->setSessionId($this->session->getId());
$em = $this->doctrine->getEntityManager();
$em->persist($user);
$em->flush();
}
}
}
And add it as a service:
services:
acme_user.listsner.login:
class: Acme\UserBundle\Listener\LoginListener
arguments: [ #session, #doctrine ]
tags:
- { name: kernel.event_listener, event: security.interactive_login, method: onLogin }
Then you can simply remove session for users:
$users = []; // ... get user list
$sessionIds = array_map(function($user) {
return $user->getId();
});
if (count(sessionIds) > 0) {
$sql = 'DELETE FROM session WHERE session_id IN (' . implode($sessionIds, ',') . ')';
$entityManager->getConnection()->exec($sql);
}
foreach ($users as $user) {
$user->setSessionId(null);
$entityManager->persist($user);
}
$entityManager->flush();

Generate password_hash which matches Magento Go API

I am trying to authenticate a user through the Magento Go SOAP API and having problems generating a matching hash. According to the docs the password_hash contains password:salt however when I md5 it's not matching the password_hash.
Example:
1) I changed my password through admin control panel to 'testtest'
2) Run the following code:
$client = new SoapClient('http://XXXX.gostorego.com/api/v2_soap/?wsdl');
$session = $client->login($api_user, $api_pass);
$params = array('filter'=>array(array('key'=>'email','value'=>'user#domain.com')));
$data = $client->customerCustomerList($session, $params);
echo '<pre>CUSTOMER: '.print_r($data, true).'</pre>';
if (count($data)) {
$hash = explode(':',$data[0]->password_hash);
$salt = $hash[1];
echo '<pre>HASH PARTS:'.print_r($hash, true).'</pre>';
echo '<br>' .md5($salt.$password);
}
3) password_hash is f35604820826428dd7633b91cd6078f4075c9bfa1a37db7bc70f563475ad8495:qK
4) MD5 is 0b04a656c770ba2f10b5918f94529cd8
I've never done this with with Magento Go (and I'm not sure it's supported/possible) but the hash string
f35604820826428dd7633b91cd6078f4075c9bfa1a37db7bc70f563475ad8495:qK
is too long to be a MD5 hash of a string. That's a 64 byte hash (plus the :, plus the salt qK). My guess is it's SHA256, but that's a guess based on character length.
On the backend, Both Md5 and SHA are being supported, with newer support leaning towards the SHA (in enterprise).
If your password were: 12341234
The DB Hash would infact be similar to: cdb757ce51af9749d2fabea4cf71dc72a1ec7b8721e5f8de83020f574ca3c5f1:TR
And is indeed SHA256.
However, the remote connection should be "https:" over SSL for the WSDL file and you should be entering your SOAP API key in normal/plain text.
ie:
$username = "myUsername"; //ie. yourApiUsername
$password = "myUserPass"; //ie. 12341234
If you want to replicate their hashing for your own internal purposes, you need to look at their methods: class Mage_Core_Model_Encryption
public function hash($data)
{
return md5($data);
}
/**
* Validate hash against hashing method (with or without salt)
*
* #param string $password
* #param string $hash
* #return bool
* #throws Exception
*/
public function validateHash($password, $hash)
{
$hashArr = explode(':', $hash);
switch (count($hashArr)) {
case 1:
return $this->hash($password) === $hash;
case 2:
return $this->hash($hashArr[1] . $password) === $hashArr[0];
}
Mage::throwException('Invalid hash.');
}
Enterprise:
public function hash($data, $version = self::HASH_VERSION_LATEST)
{
if (self::HASH_VERSION_MD5 === $version) {
return md5($data);
}
return hash('sha256', $data);
}
/**
* Validate hash by specified version
*
* #param string $password
* #param string $hash
* #param int $version
* #return bool
*/
public function validateHashByVersion($password, $hash, $version = self::HASH_VERSION_LATEST)
{
// look for salt
$hashArr = explode(':', $hash, 2);
if (1 === count($hashArr)) {
return $this->hash($password, $version) === $hash;
}
list($hash, $salt) = $hashArr;
return $this->hash($salt . $password, $version) === $hash;
}

Resources