I am using pdf417gen to generate an AAMVA barcode. The generated barcode is perfectly readable and easy to read. But if I enter the original data and compare my barcode with the original, visually, they are different, but decoding gives the same result.
My Code
from pdf417gen import encode, render_image
# Some data to encode
text = '#\n\x1E\rANSI 636015080001DL00310290DLDCAC\nDDAF\nDDB10102016\nDCBA\nDCDNONE\nDBA04062025\nDCSPERELLA\nDDEN\nDACCODY\nDDFN\nDADGLENN\nDDGN\nDBD04192019\nDBB04061995\nDBC1\nDAYGRN\nDAZBLN\nDAU064 IN\nDAW125\nDCLW\nDAG6615 STICHTER AVE \nDAIDALLAS\nDAJTX\nDAK752300000 \nDCK34509203 20190419\nDAQ34509203\nDCF00619980045119245470\nDCGUSA\n\r'
codes = encode(text, columns=15, security_level=3)
# Generate barcode as image
image = render_image(codes, scale=2, ratio=2, padding=0) # Pillow Image object
image.save('barcode.jpg')
My barcode
enter image description here
Original barcode
enter image description here
Barcodes differ in the number of lines. They also have different left and right columns, which are responsible for the barcode data.
If you change the protection level and add spaces to the end of the data, you can get approximately the same result. But it's still not a complete repeat.
I think the original uses a fixed value of rows and columns, if there are not enough characters to fill, they are replaced with blanks or service characters, but there is no such information in documents and on the Internet.
I don't know what I'm missing, for a complete repetition of the original. I would be very grateful for any help
Everything is stored as 0 1 in digital , binary format file or other format file
So when we are trying to open a Executive file with hex editor it could show all the random ASCII character that will be produced based on 7, 0 1 bits that randomly created that ( anything ) file because it is 0 1 at the storage, in memory.
So why it shows strange character that are not ASCII char?
A hex reader doesn't just parse 7 bit per 7 bit or 8 bit? it read some meta data in file and then read based on that?
Your hex editor is choosing to decode the bytes not as ASCII but as some other character encoding.
You are right the ASCII character set has 128 codepoints and the ASCII character encoding encodes them in single bytes in the range 0 to 127. Since bytes in an arbitrary file could range from 0 to 255 and the ASCII character set isn't used much for text files, decoding as ASCII wouldn't reveal as much information about potential text as a more likely character encoding and would reveal information about only half of the values of binary files.
A hex editor's job is to display and allow you to edit bytes. Additional presentations and editing capability are extra features. Many do present text, some allow search and replace of text. Some even work with other data formats such as decimal integers, multibyte integers, floating point, etc.
There is no text but encoded text. So, to support text, a hex editor—and any other program (including compilers)—must choose or allow you to choose a character encoding. For byte values that can't be decoded using that encoding, a dot or question mark is often substituted in the text display of a hex editor.
If you don't find which character encoding is used by your hex editor, you could test it by creating a file with byte values 0 to 255, see what it displays and match it against the many, many possibilities. It might be one that your operating system uses for your "default". In Windows cmd, go chcp; In Linux terminal, go locale.
I have created layout based on cobol copybook.
Layout snap-shot:
I tried to load data also selecting same layout, it gives me wrong result for some columns. I try using all binary numeric type.
CLASS-ORDER-EDGE
DIV-NO-EDG
OFFICE-NO-EDG
REG-AREA-NO-EDG
CITY-NO-EDG
COUNTY-NO-EDG
BILS-COUNT-EDG
REV-AMOUNT-EDG
USAGE-QTY-EDG
GAS-CCF-EDG
result snapshot
Input file can be find below attachment
enter link description here
or
https://drive.google.com/open?id=0B-whK3DXBRIGa0I0aE5SUHdMTDg
Expected output:
Related thread
Unpacking COMP-3 digit using Java
First Problem you have done an EBCDIC --> ascii conversion on the file !!!!
The EBCDIC --> ascii conversion will also try and convert binary fields as well as text.
For example:
Comp-3 value hex hex after Ascii conversion
400 x'400c' x'200c' x'40' is the ebcdic space character
it gets converted to the ascii
space character x'20'
You need to do binary transfer, keeping the file as ebcdic:
Check the file on the Mainframe if it has a RECFM=FB you can do a transfer
If the file is RECFM=VB make sure you transfer the RDW (Record Descriptor word) (or copy the VB file to a FB file on the mainframe).
Other points:
You will have to update RecordEditor/JRecord
The font will need to be ebcdic (cp037 for US ebcdic; for other lookup)
The FileStructure/FileOrganisation needs to change (Fixed length / VB)
Finally
BILS-Count-EDG is either 9 characters long or starts in column 85 (and is 8 bytes long).
You should include Xml in as text not copy a picture in.
In the RecordEditor if you Right click >>> Edit Record; it will show the fields as Value, Raw Text and Hex. That is useful for seeing what is going on
You do not seem to accept many answers; it is not relevant whether the answer solves your problem; it is whether the answer is correct answer for the question.
I have some binary data that I want to convert to something more easily readable and copy/pastable.
The binary data shows up like this
?Q?O?,???W%ʐ):?g????????
Which is pretty ugly. I can convert it to hex with:
value.unpack("H*").first
But since hexadecimal only has 16 characters, it isn't very compressed. I end up with a string that is hundreds of chars long.
I'd prefer a format that uses letters (capitalized and lowercase), numbers, and basic symbols, to make best use of the possible values. What can I use?
I'd also prefer something that comes built-in to Ruby, that doesn't require another library. Unfortunately I can't require another library unless it's really well known and popular, or ideally built-in to Ruby.
I tried the stuff from http://apidock.com/ruby/String/unpack and couldn't find anything.
A simple method uses Base64 encoding to encode the value. It's very similar to Hex encoding (which is Base16), but uses a longer dictionary.
Base64 strings, when properly prepared, contain only printable characters. This is a benefit for copy/paste and for sharing.
The secondary benefit is that it has a 3:4 encoding ratio, which means that it's reasonably efficient. A 3:4 encoding ration means that for each 3 bytes in the input, 4 bytes are used to encode (75% efficient); Hex encoding is a less efficient 1:2 encoding ratio, or for each 1 byte of input, 2 bytes are used to encode (50% efficient).
You can use the Ruby standard library Base64 implementation to encode and decode, like so:
require "base64"
encoded = Base64.encode64("Taste the thunder!") # <== "VGFzdGUgdGhlIHRodW5kZXIh\n"
decoded = Base64.decode64(encoded) # <== "Taste the thunder!"
Note that there is a (mostly) URL-safe version, as well, so that you can include an encoded value anywhere in a URL without requiring any additional URL encoding. This would allow you to pass information in a URL in an obscured way, and especially information that normally wouldn't be easily passed in that manner.
Try this to encode your data:
encoded_url_param = Base64.urlsafe_encode64("cake+pie=yummy!") # <== "Y2FrZStwaWU9eXVtbXkh"
decoded_url_param = Base64.urlsafe_decode64(encoded_url_param) # <== "cake+pie=yummy!"
Using Base64 in a URL, while actually not "security", will help keep prying eyes from your data and intent. The only potential downside to using Base64 values in a URL is that the URL must remain case-sensitive, and some applications don't honor that requirement. See the Should URL be case sensitive SO question for more information.
Sounds to me like you want base 64. It is part of the standard library:
require 'base64'
Base64.encode64(some_data)
Or using pack,
[some_data].pack("m")
The resulting data is about 4/3 the size of the input.
Base36 string encoding is a reasonable alternative to both Base64 and Hex encoding, as well. In this encoding method, only 36 characters are used, typically the ASCII lowercase letters and the ASCII numbers.
There's not a Ruby API that specifically does this, however this SO answer Base36 Encode a String shows how to do this efficiently in Ruby:
Encoding to Base36:
encoded = data.unpack('H*')[0].to_i(16).to_s(36)
Decoding from Base36:
decoded = [encoded.to_i(36).to_s(16)].pack 'H*'
Base36 encoding will work well when used in URLs, similarly to Base64, however it is not sensitive to the case sensitivity issues that Base64 is.
Note that Base36 string encoding should not be confused with base 36 radix integer encoding, which simply converts an integer value to the corresponding base 36 encoding. The integer technique uses String#to_i(36) and Fixnum#to_s(36) to accomplish its goals.
Wikipedia says
Base64 encoding schemes are commonly used when there is a need to encode binary data that needs be stored and transferred over media that are designed to deal with textual data. This is to ensure that the data remains intact without modification during transport.
But is it not that data is always stored/transmitted in binary because the memory that our machines have store binary and it just depends how you interpret it? So, whether you encode the bit pattern 010011010110000101101110 as Man in ASCII or as TWFu in Base64, you are eventually going to store the same bit pattern.
If the ultimate encoding is in terms of zeros and ones and every machine and media can deal with them, how does it matter if the data is represented as ASCII or Base64?
What does it mean "media that are designed to deal with textual data"? They can deal with binary => they can deal with anything.
Thanks everyone, I think I understand now.
When we send over data, we cannot be sure that the data would be interpreted in the same format as we intended it to be. So, we send over data coded in some format (like Base64) that both parties understand. That way even if sender and receiver interpret same things differently, but because they agree on the coded format, the data will not get interpreted wrongly.
From Mark Byers example
If I want to send
Hello
world!
One way is to send it in ASCII like
72 101 108 108 111 10 119 111 114 108 100 33
But byte 10 might not be interpreted correctly as a newline at the other end. So, we use a subset of ASCII to encode it like this
83 71 86 115 98 71 56 115 67 110 100 118 99 109 120 107 73 61 61
which at the cost of more data transferred for the same amount of information ensures that the receiver can decode the data in the intended way, even if the receiver happens to have different interpretations for the rest of the character set.
Your first mistake is thinking that ASCII encoding and Base64 encoding are interchangeable. They are not. They are used for different purposes.
When you encode text in ASCII, you start with a text string and convert it to a sequence of bytes.
When you encode data in Base64, you start with a sequence of bytes and convert it to a text string.
To understand why Base64 was necessary in the first place we need a little history of computing.
Computers communicate in binary - 0s and 1s - but people typically want to communicate with more rich forms data such as text or images. In order to transfer this data between computers it first has to be encoded into 0s and 1s, sent, then decoded again. To take text as an example - there are many different ways to perform this encoding. It would be much simpler if we could all agree on a single encoding, but sadly this is not the case.
Originally a lot of different encodings were created (e.g. Baudot code) which used a different number of bits per character until eventually ASCII became a standard with 7 bits per character. However most computers store binary data in bytes consisting of 8 bits each so ASCII is unsuitable for tranferring this type of data. Some systems would even wipe the most significant bit. Furthermore the difference in line ending encodings across systems mean that the ASCII character 10 and 13 were also sometimes modified.
To solve these problems Base64 encoding was introduced. This allows you to encode arbitrary bytes to bytes which are known to be safe to send without getting corrupted (ASCII alphanumeric characters and a couple of symbols). The disadvantage is that encoding the message using Base64 increases its length - every 3 bytes of data is encoded to 4 ASCII characters.
To send text reliably you can first encode to bytes using a text encoding of your choice (for example UTF-8) and then afterwards Base64 encode the resulting binary data into a text string that is safe to send encoded as ASCII. The receiver will have to reverse this process to recover the original message. This of course requires that the receiver knows which encodings were used, and this information often needs to be sent separately.
Historically it has been used to encode binary data in email messages where the email server might modify line-endings. A more modern example is the use of Base64 encoding to embed image data directly in HTML source code. Here it is necessary to encode the data to avoid characters like '<' and '>' being interpreted as tags.
Here is a working example:
I wish to send a text message with two lines:
Hello
world!
If I send it as ASCII (or UTF-8) it will look like this:
72 101 108 108 111 10 119 111 114 108 100 33
The byte 10 is corrupted in some systems so we can base 64 encode these bytes as a Base64 string:
SGVsbG8Kd29ybGQh
Which when encoded using ASCII looks like this:
83 71 86 115 98 71 56 75 100 50 57 121 98 71 81 104
All the bytes here are known safe bytes, so there is very little chance that any system will corrupt this message. I can send this instead of my original message and let the receiver reverse the process to recover the original message.
Encoding binary data in XML
Suppose you want to embed a couple images within an XML document. The images are binary data, while the XML document is text. But XML cannot handle embedded binary data. So how do you do it?
One option is to encode the images in base64, turning the binary data into text that XML can handle.
Instead of:
<images>
<image name="Sally">{binary gibberish that breaks XML parsers}</image>
<image name="Bobby">{binary gibberish that breaks XML parsers}</image>
</images>
you do:
<images>
<image name="Sally" encoding="base64">j23894uaiAJSD3234kljasjkSD...</image>
<image name="Bobby" encoding="base64">Ja3k23JKasil3452AsdfjlksKsasKD...</image>
</images>
And the XML parser will be able to parse the XML document correctly and extract the image data.
Why not look to the RFC that currently defines Base64?
Base encoding of data is used in
many situations to store or transfer
data in environments that, perhaps for
legacy reasons, are restricted to
US-ASCII [1] data.Base encoding can
also be used in new applications
that do not have legacy restrictions,
simply because it makes it possible
to manipulate objects with text
editors.
In the past, different applications
have had different requirements and
thus sometimes implemented base
encodings in slightly different
ways. Today, protocol specifications
sometimes use base encodings in
general, and "base64" in particular,
without a precise description or
reference. Multipurpose Internet Mail
Extensions (MIME) [4] is often used
as a reference for base64 without
considering the consequences for
line-wrapping or non-alphabet
characters. The purpose of this
specification is to establish common
alphabet and encoding
considerations. This will hopefully
reduce ambiguity in other
documents, leading to better
interoperability.
Base64 was originally devised as a way to allow binary data to be attached to emails as a part of the Multipurpose Internet Mail Extensions.
Media that is designed for textual data is of course eventually binary as well, but textual media often use certain binary values for control characters. Also, textual media may reject certain binary values as non-text.
Base64 encoding encodes binary data as values that can only be interpreted as text in textual media, and is free of any special characters and/or control characters, so that the data will be preserved across textual media as well.
It is more that the media validates the string encoding, so we want to ensure that the data is acceptable by a handling application (and doesn't contain a binary sequence representing EOL for example)
Imagine you want to send binary data in an email with encoding UTF-8 -- The email may not display correctly if the stream of ones and zeros creates a sequence which isn't valid Unicode in UTF-8 encoding.
The same type of thing happens in URLs when we want to encode characters not valid for a URL in the URL itself:
http://www.foo.com/hello my friend -> http://www.foo.com/hello%20my%20friend
This is because we want to send a space over a system that will think the space is smelly.
All we are doing is ensuring there is a 1-to-1 mapping between a known good, acceptable and non-detrimental sequence of bits to another literal sequence of bits, and that the handling application doesn't distinguish the encoding.
In your example, man may be valid ASCII in first form; but often you may want to transmit values that are random binary (ie sending an image in an email):
MIME-Version: 1.0
Content-Description: "Base64 encode of a.gif"
Content-Type: image/gif; name="a.gif"
Content-Transfer-Encoding: Base64
Content-Disposition: attachment; filename="a.gif"
Here we see that a GIF image is encoded in base64 as a chunk of an email. The email client reads the headers and decodes it. Because of the encoding, we can be sure the GIF doesn't contain anything that may be interpreted as protocol and we avoid inserting data that SMTP or POP may find significant.
Here is a summary of my understanding after reading what others have posted:
Important!
Base64 encoding is not meant to provide security
Base64 encoding is not meant to compress data
Why do we use Base64
Base64 is a text representation of data that consists of only 64 characters which are the alphanumeric characters (lowercase and uppercase), +, / and =.
These 64 characters are considered ‘safe’, that is, they can not be misinterpreted by legacy computers and programs unlike characters such as <, > \n and many others.
When is Base64 useful
I've found base64 very useful when transfering files as text. You get the file's bytes and encode them to base64, transmit the base64 string and from the receiving side you do the reverse.
This is the same procedure that is used when sending attachments over SMTP during emailing.
How to perform base64 encoding/decoding
Conversion from base64 text to bytes is called decoding.
Conversion from bytes to base64 text is called encoding. This is a bit different from how other encodings/decodings are named.
Dotnet and Powershell
Microsoft's Dotnet framework has support for encoding and decoding bytes to base64. Look for the Convert namespace in the mscorlib library.
Below are powershell commands you can use:
// Base64 encode PowerShell
// See: https://adsecurity.org/?p=478
$Text='This is my nice cool text'
$Bytes = [System.Text.Encoding]::Unicode.GetBytes($Text)
$EncodedText = [Convert]::ToBase64String($Bytes)
$EncodedText
// Convert from base64 to plain text
[System.Text.Encoding]::Unicode.GetString([Convert]::FromBase64String('VABoAGkAcwAgAGkAcwAgAG0AeQAgAG4AaQBjAGUAIABjAG8AbwBsACAAdABlAHgAdAA='))
Output>This is my nice cool text
Bash has a built-in command for base64 encoding/decoding. You can use it like this:
To encode to base64:
echo 'hello' | base64
To decode base64-encoded text to normal text:
echo 'aGVsbG8K' | base64 -d
Node.js also has support for base64. Here is a class that you can use:
/**
* Attachment class.
* Converts base64 string to file and file to base64 string
* Converting a Buffer to a string is known as decoding.
* Converting a string to a Buffer is known as encoding.
* See: https://nodejs.org/api/buffer.html
*
* For binary to text, the naming convention is reversed.
* Converting Buffer to string is encoding.
* Converting string to Buffer is decoding.
*
*/
class Attachment {
constructor(){
}
/**
*
* #param {string} base64Str
* #returns {Buffer} file buffer
*/
static base64ToBuffer(base64Str) {
const fileBuffer = Buffer.from(base64Str, 'base64');
// console.log(fileBuffer)
return fileBuffer;
}
/**
*
* #param {Buffer} fileBuffer
* #returns { string } base64 encoded content
*/
static bufferToBase64(fileBuffer) {
const base64Encoded = fileBuffer.toString('base64')
// console.log(base64Encoded)
return base64Encoded
}
}
You get the file buffer like so:
const fileBuffer = fs.readFileSync(path);
Or like so:
const buf = Buffer.from('hey there');
You can also use an API to do for you the encoding and encoding, here is one:
To encode, you pass in the plain text as the body.
POST https://mk34rgwhnf.execute-api.ap-south-1.amazonaws.com/base64-encode
To decode, pass in the base64 string as the body.
POST https://mk34rgwhnf.execute-api.ap-south-1.amazonaws.com/base64-decode
Fantasy example of when you might need base64
Here is a far fetched scenario of when you might need to use base64.
Suppose you are a spy and you're on a mission to copy and take back a picture of great value back to your country's intelligence.
This picture is on a computer that has no access to internet and no printer. All you have in your hands is a pen and a single sheet of paper. No flash disk, no CD etc. What do you do?
Your first option would be to convert the picture into binary 1s and 0s , copy those 1s and 0s to the paper one by one and then run for it.
However, this can be a challenge because representing a picture using only 1s and 0s as your alphabet will result in very many 1s and 0s. Your paper is small and you dont have time. Plus, the more 1s and 0s the more chances of error.
Your second option is to use hexadecimal instead of binary. Hexadecimal allows for 16 instead of 2 possible characters so you have a wider alphabet hence less paper and time required.
Still a better option is to convert the picture into base64 and take advantage of yet another larger character set to represent the data. Less paper and less time to complete. There you go!
Base64 instead of escaping special characters
I'll give you a very different but real example: I write javascript code to be run in a browser. HTML tags have ID values, but there are constraints on what characters are valid in an ID.
But I want my ID to losslessly refer to files in my file system. Files in reality can have all manner of weird and wonderful characters in them from exclamation marks, accented characters, tilde, even emoji! I cannot do this:
<div id="/path/to/my_strangely_named_file!#().jpg">
<img src="http://myserver.com/path/to/my_strangely_named_file!#().jpg">
Here's a pic I took in Moscow.
</div>
Suppose I want to run some code like this:
# ERROR
document.getElementById("/path/to/my_strangely_named_file!#().jpg");
I think this code will fail when executed.
With Base64 I can refer to something complicated without worrying about which language allows what special characters and which need escaping:
document.getElementById("18GerPD8fY4iTbNpC9hHNXNHyrDMampPLA");
Unlike using an MD5 or some other hashing function, you can reverse the encoding to find out what exactly the data was that actually useful.
I wish I knew about Base64 years ago. I would have avoided tearing my hair out with ‘encodeURIComponent’ and str.replace(‘\n’,’\\n’)
SSH transfer of text:
If you're trying to pass complex data over ssh (e.g. a dotfile so you can get your shell personalizations), good luck doing it without Base 64. This is how you would do it with base 64 (I know you can use SCP, but that would take multiple commands - which complicates key bindings for sshing into a server):
https://superuser.com/a/1376076/114723
One example of when I found it convenient was when trying to embed binary data in XML. Some of the binary data was being misinterpreted by the SAX parser because that data could be literally anything, including XML special characters. Base64 encoding the data on the transmitting end and decoding it on the receiving end fixed that problem.
Most computers store data in 8-bit binary format, but this is not a requirement. Some machines and transmission media can only handle 7 bits (or maybe even lesser) at a time. Such a medium would interpret the stream in multiples of 7 bits, so if you were to send 8-bit data, you won't receive what you expect on the other side. Base-64 is just one way to solve this problem: you encode the input into a 6-bit format, send it over your medium and decode it back to 8-bit format at the receiving end.
In addition to the other (somewhat lengthy) answers: even ignoring old systems that support only 7-bit ASCII, basic problems with supplying binary data in text-mode are:
Newlines are typically transformed in text-mode.
One must be careful not to treat a NUL byte as the end of a text string, which is all too easy to do in any program with C lineage.
What does it mean "media that are
designed to deal with textual data"?
That those protocols were designed to handle text (often, only English text) instead of binary data (like .png and .jpg images).
They can deal with binary => they can
deal with anything.
But the converse is not true. A protocol designed to represent text may improperly treat binary data that happens to contain:
The bytes 0x0A and 0x0D, used for line endings, which differ by platform.
Other control characters like 0x00 (NULL = C string terminator), 0x03 (END OF TEXT), 0x04 (END OF TRANSMISSION), or 0x1A (DOS end-of-file) which may prematurely signal the end of data.
Bytes above 0x7F (if the protocol that was designed for ASCII).
Byte sequences that are invalid UTF-8.
So you can't just send binary data over a text-based protocol. You're limited to the bytes that represent the non-space non-control ASCII characters, of which there are 94. The reason Base 64 was chosen was that it's faster to work with powers of two, and 64 is the largest one that works.
One question though. How is that
systems still don't agree on a common
encoding technique like the so common
UTF-8?
On the Web, at least, they mostly have. A majority of sites use UTF-8.
The problem in the West is that there is a lot of old software that ass-u-me-s that 1 byte = 1 character and can't work with UTF-8.
The problem in the East is their attachment to encodings like GB2312 and Shift_JIS.
And the fact that Microsoft seems to have still not gotten over having picked the wrong UTF encoding. If you want to use the Windows API or the Microsoft C runtime library, you're limited to UTF-16 or the locale's "ANSI" encoding. This makes it painful to use UTF-8 because you have to convert all the time.
Why/ How do we use Base64 encoding?
Base64 is one of the binary-to-text encoding scheme having 75% efficiency. It is used so that typical binary data (such as images) may be safely sent over legacy "not 8-bit clean" channels.
In earlier email networks (till early 1990s), most email messages were plain text in the 7-bit US-ASCII character set. So many early comm protocol standards were designed to work over "7-bit" comm links "not 8-bit clean".
Scheme efficiency is the ratio between number of bits in the input and the number of bits in the encoded output.
Hexadecimal (Base16) is also one of the binary-to-text encoding scheme with 50% efficiency.
Base64 Encoding Steps (Simplified):
Binary data is arranged in continuous chunks of 24 bits (3 bytes) each.
Each 24 bits chunk is grouped in to four parts of 6 bit each.
Each 6 bit group is converted into their corresponding Base64 character values, i.e. Base64 encoding converts three octets into four encoded characters. The ratio of output bytes to input bytes is 4:3 (33% overhead).
Interestingly, the same characters will be encoded differently depending on their position within the three-octet group which is encoded to produce the four characters.
The receiver will have to reverse this process to recover the original message.
What does it mean "media that are designed to deal with textual data"?
Back in the day when ASCII ruled the world dealing with non-ASCII values was a headache. People jumped through all sorts of hoops to get these transferred over the wire without losing out information.