For reasons probably best left unanswered I need to generate infinite RSA public/private keys. Note this is not being used for anything highly secure, so please don't tell me not to do this, yes I know its not ideal. By "infinite" I mean I need a unknown number of them think billions to trillions, and creating them before being used is not possible.
Since this would consume infinite space and take infinite time to generate I need to do it at runtime.
However I also need to have for a given input the same key pair. This means I need to deterministically recreate the RSA key given the input.
I am using Go and normally you create keys using the following,
k, err := rsa.GenerateKey(rand.Reader, 2048)
Of course the catch is that rand.Reader is supplied by crypto/rand and as such there is no way to seed this.
I thought that it would be possible to provide my own reader implementation to achieve my goal. I looked through the source of GenerateKey and noted that it is looking for prime numbers, so I implemented my own reader, such that I could control the "random" primes returned, allowing me to generate the same key when required,
type Reader struct {
data []byte
sum int
primes []int
}
func NewReader(toRead string) *Reader {
primes := sieveOfEratosthenes(10_000_000)
return &Reader{[]byte(toRead), 0, primes}
}
func (r *Reader) Read(p []byte) (n int, err error) {
r.sum = r.sum + 1
if r.sum >= 100_000 {
return r.primes[rand.Intn(len(r.primes))], io.EOF
}
return r.primes[rand.Intn(len(r.primes))], nil
}
func sieveOfEratosthenes(N int) (primes []int) {
b := make([]bool, N)
for i := 2; i < N; i++ {
if b[i] == true {
continue
}
primes = append(primes, i)
for k := i * i; k < N; k += i {
b[k] = true
}
}
return
}
I can then call into generate key like so
k, err := rsa.GenerateKey(NewReader(""), 2048)
Which compiles, but crashes at runtime due to nil pointers. I am fairly comfortable with Go, but the implementation of RSA for this is beyond my understanding. Looking for either a better way to achieve this, or perhaps what I need to do to get my reader working.
Note, the only hard requirement's I have here are being able to generate the same key for a given input, and use rsa.GenerateKey or a drop in compatible replacement. The input can be anything really, so long as I get the same key as the output.
Here is a Go playground link demonstrating where I am currently https://go.dev/play/p/jd1nAoPR5aD
The Read method is not doing what is expected. It does not fill the input p byte slice with random bytes. If you look at the implementation for Unix of crypto/rand.Read method, it passes the input byte slice into another reader. So basically you what you need to fill the byte slice with random numbers. For example:
func (r *Reader) Read(p []byte) (n int, err error) {
i := 0
b := p
for i < len(b) {
if len(b) < 4 {
b[0] = 7
b = b[1:]
} else {
binary.LittleEndian.PutUint32(b, uint32(rand.Intn(len(r.primes))))
b = b[4:]
}
}
return len(p), nil
}
Here is the link to playground.
UPD
As mentioned in the answer by Erwin, there is a function called MaybeReadRand that with 50% chance read 1 byte from rand reader to make this function nondeterministic. But you can get around by adding if statement in Read method: if length of the input slice is 1, then ignore everything and return. Otherwise, feed the input slice with prime numbers:
func (r *Reader) Read(p []byte) (n int, err error) {
i := 0
b := p
if len(p) == 1 {
println("maybeReadRand")
return 1, nil
}
for i < len(b) {
if len(b) < 4 {
b[0] = 7
b = b[1:]
} else {
binary.LittleEndian.PutUint32(b, uint32(r.primes[r.i]))
r.i++
b = b[4:]
}
}
return len(p), nil
}
In this snippet I am creating 2 keys, and both of them are equal.
Someone posted an answer here, and I was about to award it to them as it got me the final piece I needed to resolve the question.
In short, I didn't read how the read was being called properly and I needed to populate the input slice. Changing the read to the below,
func (r *Reader) Read(p []byte) (n int, err error) {
i := 0
b := p
for i < len(b) {
if len(b) < 4 {
b[0] = 7
b = b[1:]
} else {
binary.LittleEndian.PutUint32(b, uint32(rand.Intn(len(r.primes))))
b = b[4:]
}
}
return len(p), nil
}
resolves the crashes, and since it is now using math/rand I can set the seed for the list of primes allowing me to ensure it is deterministic. Everything solved without needing anything from 3rd parties. Excellent.
To whoever posted that, if you post again I will award you the accepted answer.
You have made a strange assumption. rand.Reader returns completely random data. It doesn't return prime numbers. rsa.GenerateKey uses the data internally to come up with -very probable- prime numbers. But it doesn't depend on any particular pattern in the reader. In fact, it prefers there to be no pattern (and "being a prime number" is definitely a pattern)
Sure, you're free to feed rsa.GenerateKeym, or excerpts from Shakespeare's works, but rsa.GenerateKey is not going to make use of the fact that it is.
So running a sieve of Eratosthenes is a waste of time (literally).
If rsa.GenerateKey allowed you to deterministically generate keys, then you could directly generate bytes using a pseudo-random number generator that uses a seed, directly.
But, that's not the case. Look at the source code, line 287 of rsa.go:
func GenerateMultiPrimeKey(random io.Reader, nprimes int, bits int) (*PrivateKey, error) {
randutil.MaybeReadByte(random) // <------- this line here is the important one
The function GenerateMultiPrimeKey is directly invoked by GenerateKey.
Now what does randutil.MaybeReadByte do? The documentation is clear:
MaybeReadByte reads a single byte from r with ~50% probability. This
is used to ensure that callers do not depend on non-guaranteed
behaviour, e.g. assuming that rsa.GenerateKey is deterministic w.r.t.
a given random stream.
In short, it ensures that it is not possible to deterministically generate a particular RSA key.
If you still needed to do that, you would need to copy the source code of GenerateMultiPrimeKey to your own library, and remove the line randutil.MaybeReadByte. Or you'd need to rethink what your requirement really is, a level higher than you're currently thinking, and see if there is a different way to achieve your requirement. (Have a look at the "XY problem")
If you need a deterministic-random io.Reader, here's how to do it with the standard library
import (
"io"
"math/rand"
)
func main() {
const seed = 1
var myRand = rand.New(rand.NewSource(seed))
}
rand.NewSource creates a deterministic seeded random source, and rand.New wraps it to create a more featureful *Rand object, which includes an implementation of io.Reader.
You can also reset the *Rand by calling myRand.Seed(seed).
Related
I trying for personal skills improvement to solve the hacker rank challenge:
There is a string, s, of lowercase English letters that is repeated infinitely many times. Given an integer, n, find and print the number of letter a's in the first n letters of the infinite string.
1<=s<=100 && 1<=n<=10^12
Very naively I though this code will be fine:
fs := strings.Repeat(s, int(n)) // full string
ss := fs[:n] // sub string
fmt.Println(strings.Count(ss, "a"))
Obviously I explode the memory and got an: "out of memory".
I never faced this kind of issue, and I'm clueless on how to handle it.
How can I manipulate very long string to avoid out of memory ?
I hope this helps, you don't have to actually count by running through the string.
That is the naive approach. You need to use some basic arithmetic to get the answer without running out of memory, I hope the comments help.
var answer int64
// 1st figure out how many a's are present in s.
aCount := int64(strings.Count(s, "a"))
// How many times will s repeat in its entirety if it had to be of length n
repeats := n / int64(len(s))
remainder := n % int64(len(s))
// If n/len(s) is not perfectly divisible, it means there has to be a remainder, check if that's the case.
// If s is of length 5 and the value of n = 22, then the first 2 characters of s would repeat an extra time.
if remainder > 0{
aCountInRemainder := strings.Count(s[:remainder], "a")
answer = int64((aCount * repeats) + int64(aCountInRemainder))
} else{
answer = int64((aCount * repeats))
}
return answer
There might be other methods but this is what came to my mind.
As you found out, if you actually generate the string you will end up having that huge memory block in RAM.
One common way to represent a "big sequence of incoming bytes" is to implement it as an io.Reader (which you can view as a stream of bytes), and have your code run a r.Read(buff) loop.
Given the specifics of the exercise you mention (a fixed string repeated n times), the number of occurrence of a specific letter can also be computed straight from the number of occurences of that letter in s, plus something more (I'll let you figure out what multiplications and counting should be done).
How to implement a Reader that repeats the string without allocating 10^12 times the string ?
Note that, when implementing the .Read() method, the caller has already allocated his buffer. You don't need to repeat your string in memory, you just need to fill the buffer with the correct values -- for example by copying byte by byte your data into the buffer.
Here is one way to do it :
type RepeatReader struct {
str string
count int
}
func (r *RepeatReader) Read(p []byte) (int, error) {
if r.count == 0 {
return 0, io.EOF
}
// at each iteration, pos will hold the number of bytes copied so far
var pos = 0
for r.count > 0 && pos < len(p) {
// to copy slices over, you can use the built-in 'copy' method
// at each iteration, you need to write bytes *after* the ones you have already copied,
// hence the "p[pos:]"
n := copy(p[pos:], r.str)
// update the amount of copied bytes
pos += n
// bad computation for this first example :
// I decrement one complete count, even if str was only partially copied
r.count--
}
return pos, nil
}
https://go.dev/play/p/QyFQ-3NzUDV
To have a complete, correct implementation, you also need to keep track of the offset you need to start from next time .Read() is called :
type RepeatReader struct {
str string
count int
offset int
}
func (r *RepeatReader) Read(p []byte) (int, error) {
if r.count == 0 {
return 0, io.EOF
}
var pos = 0
for r.count > 0 && pos < len(p) {
// when copying over to p, you should start at r.offset :
n := copy(p[pos:], r.str[r.offset:])
pos += n
// update r.offset :
r.offset += n
// if one full copy of str has been issued, decrement 'count' and reset 'offset' to 0
if r.offset == len(r.str) {
r.count--
r.offset = 0
}
}
return pos, nil
}
https://go.dev/play/p/YapRuioQcOz
You can now count the as while iterating through this Reader.
What is the fastest and simplest way to generate fixed length random numbers in Go?
Say to generate 8-digits long numbers, the problem with rand.Intn(100000000) is that the result might be far less than 8-digits, and padding it with leading zeros doesn't look like a good answer to me.
I.e., I care about the the quality of the randomness more in the sense of its length. So I'm thinking, for this specific problem, would the following be the fastest and simplest way to do it?
99999999 - rand.Int63n(90000000)
I.e., I guess Int63n might be better for my case than Intn. Is it ture, or it is only a wishful thinking? Regarding randomness of the full 8-digits, would the two be the same, or there is really one better than the other?
Finally, any better way than above?
UPDATE:
Please do not provide low + rand(hi-low) as the answer, as everyone knows that. It is equivalent of what I'm doing now, and it doesn't answer my real question, "Regarding randomness of the full 8-digits, would the two be the same, or there is really one better than the other? "
If nobody can answer that, I'll plot a 2-D scatter plot between the two and find out myself...
Thanks
This is a general purpose function for generating numbers within a range
func rangeIn(low, hi int) int {
return low + rand.Intn(hi-low)
}
See it on the Playground
In your specific case, trying to generate 8 digit numbers, the range would be (10000000, 99999999)
It depend on value range you want to use.
If you allow value range [0-99999999] and padding zero ip number of char < 8, then use fmt like fmt.Sprintf("%08d",rand.Intn(100000000)).
If you dont want padding, which value in range [10000000, 99999999], then give it a base like ranNumber := 10000000 + rand.Intn(90000000)`
See it on Playground
crypto/rand package is used to generate number.
func generateRandomNumber(numberOfDigits int) (int, error) {
maxLimit := int64(int(math.Pow10(numberOfDigits)) - 1)
lowLimit := int(math.Pow10(numberOfDigits - 1))
randomNumber, err := rand.Int(rand.Reader, big.NewInt(maxLimit))
if err != nil {
return 0, err
}
randomNumberInt := int(randomNumber.Int64())
// Handling integers between 0, 10^(n-1) .. for n=4, handling cases between (0, 999)
if randomNumberInt <= lowLimit {
randomNumberInt += lowLimit
}
// Never likely to occur, kust for safe side.
if randomNumberInt > int(maxLimit) {
randomNumberInt = int(maxLimit)
}
return randomNumberInt, nil
}
I recently needed to do something like this, but with a certain byte length (rather than number of digits) and with numbers larger than max int64 (so using math/big.Int). Here was my general solution:
See on the Playground (with added code comments)
func generateRandomBigInt(numBytes int) (*big.Int, error) {
value := make([]byte, numBytes)
_, err := rand.Reader.Read(value)
if err != nil {
return nil, err
}
for true {
if value[0] != 0 {
break
}
firstByte := value[:1]
_, err := rand.Reader.Read(firstByte)
if err != nil {
return nil, err
}
}
return (&big.Int{}).SetBytes(value), nil
}
I'm playing around with Golang tour and I wonder why using naked return give me the correct result but the normal one doesn't. This is the exercise that I have this problem https://tour.golang.org/methods/12.
The objective is to create a reader that can decipher rot13. and the rot13 function is already tested.
func (r rot13Reader) Read(b []byte) (n int, err error) {
n, err = r.r.Read(b)
for i, v := range b {
b[i] = rot13(v)
}
return
}
The code above give me the correct result.
func (r rot13Reader) Read(b []byte) (int, error) {
for i, v := range b {
b[i] = rot13(v)
}
return r.r.Read(b)
}
And this doesn't change anything from the input stream.
Could anybody explain why? Thank you in advance.
It's not a problem with returns, but in the first case you're reading the data in before transforming it and in the second case you're transforming junk in a buffer and only then reading in the data (and simply passing what has been read from the underlying reader).
While this is not required for correctness, I'd suggest you don't transform the whole buffer every time, but only the portion that has been read, i.e. changing your first example from for i, v := range b to for i, v := range b[:n]. That's because io.Read call is not able to modify length of slice b, but just its content.
Take a look at the documentation of io.Reader, it should give you some more idea on how this interface is expected to work.
The Read() operation mutates the input array b. In the second example the rot13() operations are overwritten by the Read() operation. Furthermore, the rot13() operation is performed before any data has been read into the array, so you're probably doing rot13() on garbage data.
If you wanted the second example to work you'd need to write something like this:
func (r rot13Reader) Read(b []byte) (int, error) {
n, err := r.r.Read(b)
for i, v := range b {
b[i] = rot13(v)
}
return n, err
}
I'm trying to create an RSA Public Key from a Modulus and Exponent stored in a byte array. After some experimentation I've got the following:
func bytes_to_int(b []byte) (acc uint64) {
length := len(b)
if length % 4 != 0 {
extra := (4 - length % 4)
b = append([]byte(strings.Repeat("\000", extra)), b...)
length += extra
}
var block uint32
for i := 0; i < length; i += 4 {
block = binary.BigEndian.Uint32(b[i:i+4])
acc = (acc << 32) + uint64(block)
}
return
}
func main() {
fmt.Println(bytes_to_int(data[:128]))
fmt.Println(bytes_to_int(data[128:]))
}
This appears to work (although I'm not convinced there isn't a better way). My next step was to convert it to use math/big in order to handle larger numbers. I can see an Lsh function to do the << but can't figure out how to recursively add the Uint32(block) to the big.Int.
For reference, the Public Key I'm attempting to import is a Mixmaster Key stored in a keyring (pubring.mix):
http://www.mixmin.net/draft-sassaman-mixmaster-XX.html#key-format
http://pinger.mixmin.net/pubring.mix
You want Int.SetBytes to make a big.int from a slice of []byte.
func (z *Int) SetBytes(buf []byte) *Int
SetBytes interprets buf as the bytes of a big-endian unsigned integer, sets z to that value, and returns z.
This should be quite straightforward to use in your application since your keys are in big-endian format according to the doc you linked.
import "math/big"
z := new(big.Int)
z.SetBytes(byteSliceHere)
Like Nick mentioned, you could use SetBytes, keep in mind the input is in base64 so you have to decode that first.
Example:
func Base64ToInt(s string) (*big.Int, error) {
data, err := base64.StdEncoding.DecodeString(s)
if err != nil {
return nil, err
}
i := new(big.Int)
i.SetBytes(data)
return i, nil
}
This is homework
I'm working on a project, and a very small (very small, once I get this working...it's basically the pre-req for the rest of the project) part of it is to generate some combinations using a Go routine.
The code I have is thusly:
package bruteforce
// GenerateCombinations is an iterator function. Given an alphabet and a
// length, it will generate every possible combination of the letters in
// alphabet of the specified length.
//
// It is meant to be consumed by using the range clause:
//
// for combination := range GenerateCombinations(alphabet, length) {
// process(combination)
// }
//
func GenerateCombinations(alphabet string, length int) <-chan string {
GenerateCombinations(alphabet, length):
if length == 0:
yield ""
else:
for i := range alphabet{
for j := range GenerateCombinations(alphabet, length-1){
i + j
}
}
return nil
}
I seriously do not get this at all. As you can see, there is some instructor-provided pseudo code there, but the implementation of it is frying my brain.
Example I/O would be something like this:
If the alphabet is {0, 1} and the password was length 2, then it would need to generate {0, 1, 00, 01, 10, 11}.
I appreciate all suggestions, but please recognize that the term "beginner" doesn't begin to describe my competency with go. Saying things like "use channels" doesn't help me at all. Explanations are my friend... something that I haven't had a lot of luck getting out of my professor aside from "use channels."
Your teacher has already hinted that you should use a channel instead of returning a big array. By solving it like that, you will not have to store this big chunk of data containing all combination, but rather feed your function with different iterations and process them one at a time.
We can see your teachers hint in that the GenerateCombinations returns a chan string and not a []string:
func GenerateCombinations(alphabet string, length int) <-chan string
This also means that the function must 1) create a channel to return, and 2) start a goroutine that feeds iterations to the channel. This function would look something like this:
func GenerateCombinations(alphabet string, length int) <-chan string {
c := make(chan string)
// Starting a separate goroutine that will create all the combinations,
// feeding them to the channel c
go func(c chan string) {
defer close(c) // Once the iteration function is finished, we close the channel
// This is where the iteration will take place
// Your teacher's pseudo code uses recursion
// which mean you might want to create a separate function
// that can call itself.
}(c)
return c // Return the channel to the calling function
}
While I will leave the actual iteration to you, every loop should result in you putting a combination string into the channel. Because it is not a buffered channel, the iterating function will wait til the main function has read the value before continue to process the next iteration.
A playground version including the main function: http://play.golang.org/p/CBkSjpmQ0t
A full working solution using recursion: http://play.golang.org/p/0bWDCibSUJ
This is a tricky problem if you are entirely unfamiliar with Go and/or how to generate permutations. Below is a full implementation of the solution. I suggest you only look at this if you really get stuck, because doing so will obviously remove the learning experience.
You can run it on the go playground to see it work.
This approach is not recursive as your instructor's example suggests, but it gets the job done quite nicely.
package main
import "fmt"
import "sync"
func main() {
// Make sure the alphabet is sorted.
const alphabet = "abcde"
for str := range generate(alphabet) {
fmt.Println(str)
}
}
func generate(alphabet string) <-chan string {
c := make(chan string, len(alphabet))
go func() {
defer close(c)
if len(alphabet) == 0 {
return
}
// Use a sync.WaitGroup to spawn permutation
// goroutines and allow us to wait for them all
// to finish.
var wg sync.WaitGroup
wg.Add(len(alphabet))
for i := 1; i <= len(alphabet); i++ {
go func(i int) {
// Perform permutation on a subset of
// the alphabet.
Word(alphabet[:i]).Permute(c)
// Signal Waitgroup that we are done.
wg.Done()
}(i)
}
// Wait for all routines to finish.
wg.Wait()
}()
return c
}
type Word []rune
// Permute generates all possible combinations of
// the current word. This assumes the runes are sorted.
func (w Word) Permute(out chan<- string) {
if len(w) <= 1 {
out <- string(w)
return
}
// Write first result manually.
out <- string(w)
// Find and print all remaining permutations.
for w.next() {
out <- string(w)
}
}
// next performs a single permutation by shuffling characters around.
// Returns false if there are no more new permutations.
func (w Word) next() bool {
var left, right int
left = len(w) - 2
for w[left] >= w[left+1] && left >= 1 {
left--
}
if left == 0 && w[left] >= w[left+1] {
return false
}
right = len(w) - 1
for w[left] >= w[right] {
right--
}
w[left], w[right] = w[right], w[left]
left++
right = len(w) - 1
for left < right {
w[left], w[right] = w[right], w[left]
left++
right--
}
return true
}