How to convert EBCDIC with chinese chars to UTF-8 format - utf-8

Question

I have a requirement to convert a file with EBCDIC encoding which is encoded using the IBM937 code page to UTF-8 format for loading the file into a multi-byte enabled DB2 database.
I have tried unix recode and iconv. None of them has the ability to convert IBM 937 to UTF8. I'm looking for any utility (java, perl, unix ) in this world which can do that on a unix based system. Can someone help me here?
SL

Answer 1

Take a look at ICU (International Components for Unicode): http://site.icu-project.org/
It has a converter for IBM-937: http://demo.icu-project.org/icu-bin/convexp?conv=ibm-937_P110-1999&s=ALL
CU is a mature, widely used set of
C/C++ and Java libraries providing
Unicode and Globalization support for
software applications. ICU is widely
portable and gives applications the
same results on all platforms and
between C/C++ and Java software. ICU
is released under a nonrestrictive
open source license that is suitable
for use with both commercial software
and with other open source or free
software.
Here are a few highlights of the
services provided by ICU:
Code Page Conversion: Convert text
data to or from Unicode and nearly any
other character set or encoding. ICU's
conversion tables are based on charset
data collected by IBM over the course
of many decades, and is the most
complete available anywhere.
Collation: Compare strings according
to the conventions and standards of a
particular language, region or
country. ICU's collation is based on
the Unicode Collation Algorithm plus
locale-specific comparison rules from
the Common Locale Data Repository, a
comprehensive source for this type of
data.
Formatting: Format numbers, dates,
times and currency amounts according
the conventions of a chosen locale.
This includes translating month and
day names into the selected language,
choosing appropriate abbreviations,
ordering fields correctly, etc. This
data also comes from the Common Locale
Data Repository.
Time Calculations: Multiple types of
calendars are provided beyond the
traditional Gregorian calendar. A
thorough set of timezone calculation
APIs are provided.
Unicode Support: ICU closely tracks
the Unicode standard, providing easy
access to all of the many Unicode
character properties, Unicode
Normalization, Case Folding and other
fundamental operations as specified by
the Unicode Standard.
Regular Expression: ICU's regular
expressions fully support Unicode
while providing very competitive
performance.
Bidi: support for handling text
containing a mixture of left to right
(English) and right to left (Arabic or
Hebrew) data.
Text Boundaries: Locate the positions
of words, sentences, paragraphs within
a range of text, or identify locations
that would be suitable for line
wrapping when displaying the text.
And much more. Refer to the ICU User Guide for details.

Answer 2

It appears that Java can convert the IBM937 code page to UTF-8.
You would specify the input format as "cp937".
Here are two methods from the Oracle page on Character and Byte Streams:
static String readInput() {
StringBuffer buffer = new StringBuffer();
try {
FileInputStream fis = new FileInputStream("test.txt");
InputStreamReader isr = new InputStreamReader(fis,
"cp937");
Reader in = new BufferedReader(isr);
int ch;
while ((ch = in.read()) > -1) {
buffer.append((char)ch);
}
in.close();
return buffer.toString();
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
and
static void writeOutput(String str) {
try {
FileOutputStream fos = new FileOutputStream("test.txt");
Writer out = new OutputStreamWriter(fos, "UTF8");
out.write(str);
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}