I am running leofs 1.2.22 on a single node. Everything was working ok but when I started leofs today I am unable to list the contents of any buckets. I am getting errors stating the node is down.
A leofs-adm status shows
$ leofs-adm status
[System Confiuration]
-----------------------------------+----------
Item | Value
-----------------------------------+----------
Basic/Consistency level
-----------------------------------+----------
system version | 1.2.22
cluster Id | leofs_1
DC Id | dc_1
Total replicas | 1
number of successes of R | 1
number of successes of W | 1
number of successes of D | 1
number of rack-awareness replicas | 0
ring size | 2^128
-----------------------------------+----------
Multi DC replication settings
-----------------------------------+----------
max number of joinable DCs | 2
number of replicas a DC | 1
-----------------------------------+----------
Manager RING hash
-----------------------------------+----------
current ring-hash | 433fe365
previous ring-hash | 433fe365
-----------------------------------+----------
[State of Node(s)]
-------+--------------------------+--------------+----------------+----------- -----+----------------------------
type | node | state | current ring | prev ring | updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
S | storage_0#127.0.0.1 | running | 433fe365 | 433fe365 | 2017-06-27 01:00:50 -0400
G | gateway_0#127.0.0.1 | running | 433fe365 | 433fe365 | 2017-10-09 06:49:48 -0400
-------+--------------------------+--------------+----------------+----------------+----------------------------
This shows the storage node is running. However if checking the details of the storage node it returns
$ leofs-adm du storage_0#127.0.0.1
[ERROR] nodedown
I tried to recover the node but this also fails
$ leofs-adm recover-node storage_0#127.0.0.1
[ERROR] Could not connect
I am able to list the buckets
$ leofs-adm get-buckets
cluster id | bucket | owner | permissions | created at
-------------+-------------------------+--------+----------------------------------------+---------------------------
leofs_1 | workflow | simon | Me(full_control) | 2017-06-28 20:47:08 -0400
leofs_1 | weather | simon | Me(full_control) | 2017-06-26 08:27:26 -0400
leofs_1 | workers | simon | Me(full_control), Everyone(read,write) | 2017-06-26 08:30:30 -0400
But I am unable to list the contents of any bucket
$ s3cmd ls s3://weather/
WARNING: Retrying failed request: /?delimiter=/
WARNING: 500 (InternalError): We encountered an internal error. Please try again.
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?delimiter=/
WARNING: 500 (InternalError): We encountered an internal error. Please try again.
I am not sure how to recover the node and have not found anything online to help. Updating the version of leofs is not an option as I was unable to get Python boto2 to communicate with a later version of LeoFS.
Regards
Simon
Related
I'm totally bamboozled, since this code was working previously.
I'm trying to create a Mixed Edge index on an edge property in my graph (trxn_dt_int).
It shows, from the output of mgmt.printSchema(), that it was created successfully (see, Status: Enabled, I've highlighted the relevant rows).
mgmt.printSchema() output:
==>------------------------------------------------------------------------------------------------
Vertex Label Name | Partitioned | Static |
---------------------------------------------------------------------------------------------------
entity | false | false |
---------------------------------------------------------------------------------------------------
Edge Label Name | Directed | Unidirected | Multiplicity |
---------------------------------------------------------------------------------------------------
transacts_with | true | false | MULTI |
---------------------------------------------------------------------------------------------------
Property Key Name | Cardinality | Data Type |
---------------------------------------------------------------------------------------------------
benef_nm | SINGLE | class java.lang.String |
orig_nm | SINGLE | class java.lang.String |
trxn_dt_int | SINGLE | class java.lang.Float |
---------------------------------------------------------------------------------------------------
Vertex Index Name | Type | Unique | Backing | Key: Status |
---------------------------------------------------------------------------------------------------
***************************************************************************************************
Edge Index (VCI) Name | Type | Unique | Backing | Key: Status |
---------------------------------------------------------------------------------------------------
byDate | Mixed | false | search | trxn_dt_int: ENABLED|
***************************************************************************************************
---------------------------------------------------------------------------------------------------
Relation Index | Type | Direction | Sort Key | Order | Status |
Here's my groovy script to create the index:
graph.tx().rollback()
mgmt = graph.openManagement()
mgmt.printSchema()
// get index count
index_count = mgmt.getGraphIndexes(Edge.class).size()
if (index_count==0) {
// create edge schema for graph (only for properties to be indexed)
if (!mgmt.getEdgeLabel('transacts_with') & !mgmt.getPropertyKey('trxn_dt_int')) {
transacts_with = mgmt.makeEdgeLabel('transacts_with').make()
trxn_dt = mgmt.makePropertyKey('trxn_dt_int').dataType(Float.class).make()
mgmt.addProperties(transacts_with, trxn_dt)
}
mgmt.commit()
mgmt = graph.openManagement()
if (!mgmt.containsGraphIndex("byDate")) {
trxn_dt = mgmt.getPropertyKey('trxn_dt_int')
mgmt.buildIndex('byDate', Edge.class).addKey(trxn_dt).buildMixedIndex('search')
mgmt.commit()
ManagementSystem.awaitGraphIndexStatus(graph, 'byDate').call()
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex("byDate"), SchemaAction.REINDEX).get()
mgmt.commit()
}
}
When I query the elasticsearch server, it shows it's there:
curl --silent 'http://127.0.0.1:9200/_cat/indices' | cut -d\ -f3
> janusgraph_bydate
When I query for the number of documents, it returns 0.
curl --silent 'http://127.0.0.1:9200/janusgraph_bydate/_count?q=*'
> {"count":0,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
My question is, why aren't any documents being added?
I initially thought I had to reindex, but when I tried, it still didn't work. No errors at all.
mgmt = graph.openManagement()
index = mgmt.getGraphIndex('byDate')
mgmt.updateIndex(index, SchemaAction.REINDEX).get()
mgmt.commit()
Setup:
Janusgraph 0.5.1
Elasticsearch 6.0.1
Well, this is embarrassing; it wasn't updating the index with new data because my machine lacked sufficient memory, but also didn't offer warnings to suggest this. After freeing up memory elsewhere it now works again.
I am creating a shell script to Backup openstack cinder volume to glance image as like below.
test1 is volume name in the script.
#!/bin/bash
DATE=`date +%Y-%m-%d`
BACKUPDIR=/mnt/osbk/
declare -a VMS=(
test1
)
source /root/admin-openrc
echo $DATE `date` Starting Backup process of images
for vmname in "${VMS[#]}"
do
echo Backing up $vmname
echo cinder upload-to-image ${vmname} ${vmname}-vol-bkp --disk-format qcow2 --container-format bare --force True
cinder upload-to-image ${vmname} ${vmname}-vol-bkp --disk-format qcow2 --container-format bare --force True
echo glance image-download ${vmname}-vol-bkp --file $BACKUPDIR/${vmname}-vol-bkp-${DATE}.qcow2
glance --os-image-api-version 1 image-download ${vmname}-vol-bkp --file $BACKUPDIR/${vmname}-vol-bkp-${DATE}.qcow2
done
Output looks like this:
2018-12-29 Sat Dec 29 16:37:45 IST 2018 Starting Backup process of images
Backing up test1
cinder upload-to-image test1 test1-vol-bkp --disk-format qcow2 --container-format bare --force True
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| container_format | bare |
| disk_format | qcow2 |
| display_description | |
| id | 26c90209-8151-4136-b5de-f2ad7419b100 |
| image_id | 01e88175-a3fa-4354-8c0f-e4fafd9c9fc3 |
| image_name | test1-vol-bkp |
| is_public | False |
| protected | False |
| size | 2 |
| status | uploading |
| updated_at | 2018-12-29T11:07:00.000000 |
| volume_type | None |
+---------------------+--------------------------------------+
glance image-download test1-vol-bkp --file /mnt/osbk//test1-vol-bkp-2018-12-29.qcow2
404 Not Found
The resource could not be found.
Image 01e88175-a3fa-4354-8c0f-e4fafd9c9fc3 is not active (HTTP 404)
From above output the status is uploading...
I need to hold my script to wait or check the status of volume change to Active, then only the glance image download command has to run.
What am I doing wrong?
I have a massive chunk of files in an extremely fast SAN disk that I like to do Hive query on them.
An obvious option is to copy all files into HDFS by using a command like this:
hadoop dfs -copyFromLocal /path/to/file/on/filesystem /path/to/input/on/hdfs
However, I don't want to create a second copy of my files, just to be to Hive query in them.
Is there any way to point an HDFS folder into a local folder, such that Hadoop sees it as an actual HDFS folder? The files keep adding to the SAN disk, so Hadoop needs to see the new files as they are being added.
This is similar to Azure's HDInsight approach that you copy your files into a blob storage and HDInsight's Hadoop sees them through HDFS.
For playing around with small files using the local file system might be fine, but I wouldn't do it for any other purpose.
Putting a file in an HDFS means that it is being split to blocks which are replicated and distributed.
This gives you later on both performance and availability.
Locations of [external] tables can be directed to the local file system using file:///.
Whether it works smoothly or you'll start getting all kinds of error, that's to be seen.
Please note that for the demo I'm doing here a little trick to direct the location to a specific file, but your basic use will probably be for directories.
Demo
create external table etc_passwd
(
Username string
,Password string
,User_ID int
,Group_ID int
,User_ID_Info string
,Home_directory string
,shell_command string
)
row format delimited
fields terminated by ':'
stored as textfile
location 'file:///etc'
;
alter table etc_passwd set location 'file:///etc/passwd'
;
select * from etc_passwd limit 10
;
+----------+----------+---------+----------+--------------+-----------------+----------------+
| username | password | user_id | group_id | user_id_info | home_directory | shell_command |
+----------+----------+---------+----------+--------------+-----------------+----------------+
| root | x | 0 | 0 | root | /root | /bin/bash |
| bin | x | 1 | 1 | bin | /bin | /sbin/nologin |
| daemon | x | 2 | 2 | daemon | /sbin | /sbin/nologin |
| adm | x | 3 | 4 | adm | /var/adm | /sbin/nologin |
| lp | x | 4 | 7 | lp | /var/spool/lpd | /sbin/nologin |
| sync | x | 5 | 0 | sync | /sbin | /bin/sync |
| shutdown | x | 6 | 0 | shutdown | /sbin | /sbin/shutdown |
| halt | x | 7 | 0 | halt | /sbin | /sbin/halt |
| mail | x | 8 | 12 | mail | /var/spool/mail | /sbin/nologin |
| uucp | x | 10 | 14 | uucp | /var/spool/uucp | /sbin/nologin |
+----------+----------+---------+----------+--------------+-----------------+----------------+
You can mount your hdfs path into local folder, for example with hdfs mount
Please follow this for more info
But if you want speed, it isn't an option
I am in the process of moving and testing a development site on the actual domain name now and I just wanted to check if I was missing anything and also get some advice.
It is a Magento 1.8.1 install from Turnkey Linux running on an m1.medium instance.
What I have done (so far) is, create an image of the development instance, made a new account and copied it over to there. I then made an elastic IP and associated it with the new instance. Next I pointed the A name record of the production domain to the elastic IP.
Now, if I go to the production domain I get redirected to the development domain. Is there a reason for this?
Ideally I would like to have two instances, one dev one that is off unless needed and of course the production on which is going to be live 24/7. However if I turn the development domain off it stops the other too.
I have a feeling it's just because I need to change instances of the dev domain in the Magento database / back-end however I wanted to get a more knowledgable answer as I don't want to break either of the instance.
Also, I should probably mention that the development domain is a subdomain i.e. shop.mysite.com and the live one is just normal i.e. mysite.com. Not entirely sure this is relevant but thought it worth a mention.
Thanks in advance for any help.
The reason your URL on your new instance is getting redirected to the old URL is because in the core_config_data table of your magento database the web/unsecure/base_url and web/secure/base_url paths point to your old URL.
So if you are using mysql you can query your database as follows:
mysql> use magento;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from core_config_data;
+-----------+---------+----------+-------------------------------+-------------------------------------+
| config_id | scope | scope_id | path | value |
+-----------+---------+----------+-------------------------------+-------------------------------------+
| 1 | default | 0 | web/seo/use_rewrites | 1 |
| 2 | default | 0 | admin/dashboard/enable_charts | 0 |
| 3 | default | 0 | web/unsecure/base_url | http://magento.myolddomain.com/ |
| 4 | default | 0 | web/secure/use_in_frontend | 1 |
| 5 | default | 0 | web/secure/base_url | https://magento.myolddomain.com/ |
| 6 | default | 0 | web/secure/use_in_adminhtml | 1 |
| 7 | default | 0 | general/locale/code | en_US |
| 8 | default | 0 | general/locale/timezone | Europe/London |
| 9 | default | 0 | currency/options/base | USD |
| 10 | default | 0 | currency/options/default | USD |
| 11 | default | 0 | currency/options/allow | USD |
| 12 | default | 0 | general/region/display_all | 1 |
| 13 | default | 0 | general/region/state_required | AT,CA,CH,DE,EE,ES,FI,FR,LT,LV,RO,US |
| 14 | default | 0 | catalog/category/root_id | 2 |
+-----------+---------+----------+-------------------------------+-------------------------------------+
14 rows in set (0.00 sec)
and you can change it as follows:
mysql> update core_config_data set value='http://magento.mynewdomain.com' where path='web/unsecure/base_url';
mysql> update core_config_data set value='https://magento.mynewdomain.com' where path='web/secure/base_url';
I have a problem with index management inside my Magento 1.6.2.0 store. Basically I can't get them to update. The status says Processing but it says like that for over a 3 weeks now.
And when I try to reindex I am getting this message Stock Status Index process is working now. Please try run this process later but later is 3 weeks now? So it looks like the process is frozen but I don't know how to restart.
Any ideas?
cheers
Whenever you start an indexing process, Magento writes out a lock file to the var/locks folder.
$ cd /path/to/magento
$ ls var/locks
index_process_1.lock index_process_4.lock index_process_7.lock
index_process_2.lock index_process_5.lock index_process_8.lock
index_process_3.lock index_process_6.lock index_process_9.lock
The lock file prevents another user from starting an indexing process. However, if the indexing request times out or fails before it can complete, the lock file will be left in a lock state. That's probably what happened to you. I'd recommend you check the last modified dates on the lock files to make sure someone else isn't running the re-indexer right now, and then remove the lock files. This will clear up your
Stock Status Index process is working now. Please try run this process later
error. After that, run the indexers one at a time to make sure each one completes.
Hello Did you call script manually if not then create one file in your root folder and write this code in it
require_once 'app/Mage.php';
umask( 0 );
Mage :: app( "default" );
$process = Mage::getSingleton('index/indexer')->getProcessByCode('catalog_product_flat');
$process->reindexAll();
this code do indexing of your magento manually some times it's happen that if your magento store contain large number of products then it will required lot's of time to reindexing of products so when you can go to your index management from admin it will show some indexing in processing stage so may be this code will help you to remove processing stage to ready stage of your indexes.
or you can also do indexing with SSH if you have rights of it. it's faster too for indexing
For newer versions of magento , ie 2.1.3 I had to use this solution:
http://www.elevateweb.co.uk/magento-ecommerce/magento-error-sqlstatehy000-general-error-1205-lock-wait-timeout-exceeded
This might happen if you are running a lot of custom scripts and killing the scripts before the database connection gets chance to close
If you login to MySQL from CLI and run the command
SHOW PROCESSLIST;
you will get the following output
+———+—————–+——————-+—————–+———+——+——-+——————+———–+—————+———–+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined | Rows_read |
+———+—————–+——————-+—————–+———+——+——-+——————+———–+—————+———–+
| | 6794372 | db_user| 111.11.0.65:21532 | db_name| Sleep | 3800 | | NULL | 0 | 0 | 0
|
| 6794475 | db_user| 111.11.0.65:27488 | db_name| Sleep | 3757 | | NULL | 0 | 0 | 0
|
| 6794550 | db_user| 111.11.0.65:32670 | db_name| Sleep | 3731 | | NULL | 0 | 0 | 0
|
| 6794797 | db_user| 111.11.0.65:47424 | db_name | Sleep | 3639 | | NULL | 0 | 0 | 0
|
| 6794909 | db_user| 111.11.0.65:56029 | db_name| Sleep | 3591 | | NULL | 0 | 0 | 0
|
| 6794981 | db_user| 111.11.0.65:59201 | db_name| Sleep | 3567 | | NULL | 0 | 0 | 0
|
| 6795096 | db_user| 111.11.0.65:2390 | db_name| Sleep | 3529 | | NULL | 0 | 0 | 0
|
| 6795270 | db_user| 111.11.0.65:10125 | db_name | Sleep | 3473 | | NULL | 0 | 0 | 0
|
| 6795402 | db_user| 111.11.0.65:18407 | db_name| Sleep | 3424 | | NULL | 0 | 0 | 0
|
| 6795701 | db_user| 111.11.0.65:35679 | db_name| Sleep | 3330 | | NULL | 0 | 0 | 0
|
| 6800436 | db_user| 111.11.0.65:57815 | db_name| Sleep | 1860 | | NULL | 0 | 0 | 0
|
| 6806227 | db_user| 111.11.0.67:20650 | db_name| Sleep | 188 | | NULL | 1 | 0 | 0
+———+—————–+——————-+—————–+———+——+——-+——————+———–+—————+———–+
15 rows in set (0.00 sec)
You can see as an example
6794372 the command is sleep and time is 3800. This is preventing other operations
These processes should be killed 1 by 1 using the command.
KILL 6794372;
Once you have killed all the sleeping connections, things should start working as normal again
You need to do two steps:
give 777 permition to var/locks folders
delete all file of var/locks folder
Whenever you start an indexing process, Magento writes out a lock file to the var/locks folder. So uou need to do two steps:
Give 777 permission to var/locks folders
Delete all file of var/locks folder.
Now refresh the index management page in admin panel.
Enjoy!!