I was trying to import the data from one hbase(v0.98.4) to another hbase(v0.98.13).
I have exported the data using the below command -
hbase org.apache.hadoop.hbase.mapreduce.Driver export 'tblname' /path/
But I am not able to import it using the below command -
hbase org.apache.hadoop.hbase.mapreduce.Driver import 'tblname' /hdfs/path/
I get the below deprecation messages as well as an Exception thrown -
Is it becoz of version conflicts between source db and destination db?
I happen to solve it. All I had to do was create an empty table with same metadata and then import it. :)
Try using the commands here for Hbase versions above 0.94. May be you are using generalized Map reduce class and giving export and import as arguments, when the actual classes Export and Import are present. Hope it helps. Happy coding
Related
I have one ElasticNode 7.3.2 in my cluster which has crashed and since then isn't able to restart, but still contains important data (last replicas are on this node).
Caused by: java.io.IOException: failed to find metadata for existing index indexname-2019.06.23 [location: ORVU14kLSf6kIv8ULliijA, generation: 178]
i dont care about this special index, can i just remove this one from the state file? I already tried to remove it via an HexEditor, but then he complains about are not valid hash-checksum ;)
I already tried to decode it via SMILE, but it seems that *.st files dont follow the exact specifications only partially.
Does someone has an idea? Or a good tool to edit it?
Thanks
I had a similar case in which I corrupted my index settings with invalid similarity settings. Unfortunately, the settings were written to file by ES without validity check and afterwards I could not open the index again. So I got the index meta data from data/nodes/0/indices/<index UID>/_state/state-xx.st and figured out how to load it, change it and store it back again. I used ES 5.4 so the code I give here will be slightly different in ES 7.x. It goes like this:
import org.elasticsearch.cluster.metadata.IndexMetaData;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.xcontent.NamedXContentRegistry;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.file.Paths;
public class RepairIndexSettings {
public static void main(String args[]) throws IOException {
// Load the existing, possibly corrupt index file
IndexMetaData d = IndexMetaData.FORMAT.read(NamedXContentRegistry.EMPTY, Paths.get( "indexstaterepair","_state", "state-11.st"));
// Create new IndexMetaData by copying all the valid meta data from the original and removing or fixing
// the corrupt settings.
Settings.Builder sb = Settings.builder();
for (String key : d.getSettings().keySet()) {
if (!key.contains("similarity"))
sb.put(key, d.getSettings().get(key));
}
IndexMetaData newd = IndexMetaData.builder(d).settings(sb).build();
// Write the new index state to file
IndexMetaData.FORMAT.write(newd, Paths.get("indexstaterepair"));
}
}
I replaced the original state file with the new one and could open and use the index normally afterwards.
You would have to use another MetaData subclass but I think it should be quite similar otherwise.
No, there's no way to edit any files inside the data directory by hand. If you're getting an exception like failed to find metadata for existing index then there's something quite wrong with your storage, or something other than Elasticsearch has modified its contents. In either case, this has corrupted this node. The best path forward is to wipe the data path and start the node afresh so that Elasticsearch can rebuild any lost shards from elsewhere in the cluster or else restore them from a recent snapshot.
I've been trying to import com.parse.ParseImageView so i can display image queried from parse. But I get this error in my xml file
The following classes could not be found:
- com.parse.ParseImageView
Is ParseImageView no longer supported ? What is the alternative way to display image from parse database? Thanks
how can I import the exported _Installation-data? When I try to set the classname to _Installation, I just get the error "cannot import into this special class"
Workaround so far: Iterating over the exported _Installation.json and recreating the objects via POST against https://api.parse.com/1/classes/_Installation .
This is no real backup restore as this creates new objectIds, createdAts and updatedAts but better than nothing.
Meanwhile, this feature has been added.
Are there any way to read data from a file and put them into Hbase table without using any java? I tried to store data from pig script by using
sample = LOAD '/mapr/user/username/sample.txt' AS (all:chararray);
STORE deneme INTO 'hbase://sampledata' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('mysampletable:intdata');
but this gave this error message:
ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filterWritableByteArrayComparable
ERROR org.apache.pig.tools.grunt.Grunt java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/WritableByteArrayComparable
Pig seems like a good idea to import data into HBase. Check what Armon suggested about setting the $PIG_CLASSPATH.
Another possibility to bulk loading data into HBase is to use featured tools like ImportTsv (Tab Separated Values) and CompleteBulkLoad.
http://hbase.apache.org/book/ops_mgt.html#importtsv
Well, there's the Stargate REST interface, which is usable from any language. It's not perfect, but it's worth a look.
You just need to make sure that $PIG_CLASSPATH also points at hbase.jar
I'm trying to set the OutputFormat of my job to MapFileOutputFormat using:
jobConf.setOutputFormat(MapFileOutputFormat.class);
I get this error: mapred.output.format.class is incompatible with new reduce API mode
I suppose I should use the set setOutputFormatClass() of the new Job class but the problem is that when I try to do this:
job.setOutputFormatClass(MapFileOutputFormat.class);
it expects me to use this class: org.apache.hadoop.mapreduce.lib.output.MapFileOutputFormat.
In hadoop 1.0.X there is no such class. It only exists in earlier versions (e.g 0.x)
How can I solve this problem ?
Thank you!
This problem has no decently easily implementable solution.
I gave up and used Sequence files which fit my requirements too.
Have you tried the following?
import org.apache.hadoop.mapreduce.lib.output;
...
LazyOutputFormat.setOutputFormatClass(job, MapFileOutputFormat.class);