I've been trying to write a program which takes in integer input from the user and performs some calculation. What's happening is that every alternate time, the program ends prematurely without having taken in any input. Both Scanf and Scanln follow the same behavior.
The relevant code:
func main() {
var N int
var output []int
fmt.Println("Enter test cases")
//This bottom line executes only every alternate time
fmt.Scanf("%d", &N)
testCases(N, N, output)
}
It prints the line "Enter test cases" and the program terminates. But when I run the program once more, it follows through. This pattern then repeats every time I try to run the program.
Better use bufio package, it implements buffered I/O.
scanf/scanln are unbuffered.
scanner := bufio.NewScanner(os.Stdin)
scanner.Scan()
input := scanner.Text()
Related
I tried to get the input (many number with space) and converted it to slice.
number of numbers is up to 300,000
I got an error and I googled it. and there's some problem with buffer size.
so I wrote the code as below.
func ChangeToInt(input string) []int {
var nums []int
for _, word := range strings.Fields(input) {
num, _ := strconv.Atoi(word)
nums = append(nums, num)
}
return nums
}
scanner := bufio.NewScanner(os.Stdin)
maxCapacity := 4*300000
buf := make([]byte, maxCapacity)
scanner.Buffer(buf, maxCapacity)
scanner.Scan()
input := scanner.Text()
nums := ChangeToInt(input)
but still not working. what's the problem?
You are using bufio.Scanner to read your input. By default bufio.Scanner reads lines, and it uses an internal buffer to store the line. By default the line may have a max length of bufio.MaxScanTokenSize which is 64 KB. If your lines are longer than this, you'll get an error.
The internal buffer size may be changed / increased using the Scanner.Buffer() method, but if your input is a space separated list of numbers, I'd advise to change the split function of the Scanner.
As mentioned earlier, by default the scanner splits input by lines. Instead change it to split the input by words. The bufio package has a "ready" split function for that: bufio.Scanwords. Use it like this:
scanner := bufio.NewScanner(os.Stdin)
scanner.Split(bufio.ScanWords)
Now the scanner.Text() will return the words (numbers in your case) instead of complete lines, so the default 64 KB limit now applies to words, not lines. Your numbers should be less than 64 KB.
Also do check if scanning succeeds by calling scanner.Err().
You can use bufio.NewReader and its ReadString('\n') method. It will be good for large input data.
Forgive me I am starting out in Go and I am learning about the bufio package but every time I use the Scanner type the command line is stuck on the input and does not continue with normal program flow. I have tried pressing Enter but it just keeps going to a new line.
Here is my code.
/*
Dup 1 prints the text of each line that appears more than
once in the standard input, proceeded by its count.
*/
package main
import(
"bufio"
"fmt"
"os"
)
func main(){
counts := make(map[string]int)
fmt.Println("Type Some Text")
input := bufio.NewScanner(os.Stdin)
for input.Scan(){
counts[input.Text()]++
}
//NOTE: Ignoring potential Errors from input.Err()
for line,n := range counts{
if n > 1{
fmt.Printf("%d \t %s \n",n,line)
}
}
}
You have a for loop which reads lines from the standard input. This loop will run as long as os.Stdin doesn't report io.EOF (that's one case when Scanner.Scan() would return false). Normally this won't happen.
If you want to "simulate" the end of input, press Ctrl+Z on Windows, or Ctrl+D on Linux / unix systems.
So enter some lines (each "closed" by Enter), and when you're finished, press the above mentioned key.
Example output:
Type Some Text
a
a
bb
bb
bbb <-- CTRL+D pressed here
2 a
2 bb
Another option would be to use a "special" word for termination, such as "exit". It could look like this:
for input.Scan() {
line := input.Text()
if line == "exit" {
break
}
counts[line]++
}
Testing it:
Type Some Text
a
a
bb
bb
bbb
exit
2 a
2 bb
I am trying to do functional testing of a cli app similar to this way.
As the command asks a few input on command prompt, I am putting them in a file and setting it as os.Stdin.
cmd := exec.Command(path.Join(dir, binaryName), "myArg")
tmpfile := setStdin("TheMasterPassword\nSecondAnswer\n12121212\n")
cmd.Stdin = tmpfile
output, err := cmd.CombinedOutput()
The setStdin just creates a tmpFile, write the string in file and returns the *os.File.
Now, I am expecting TheMasterPassword to be first input, and it's working. But for the second input always getting Critical Error: EOF.
The function I am using for asking and getting user input this :
func Ask(question string, minLen int) string {
reader := bufio.NewReader(os.Stdin)
for {
fmt.Printf("%s: ", question)
response, err := reader.ReadString('\n')
ExitIfError(err)
if len(response) >= minLen {
return strings.TrimSpace(response)
} else {
fmt.Printf("Provide at least %d character.\n", minLen)
}
}
}
Can you please help me to find out what's going wrong?
Thanks a lot!
Adding setStdin as requested
func setStdin(userInput string) *os.File {
tmpfile, err := ioutil.TempFile("", "test_stdin_")
util.ExitIfError(err)
_, err = tmpfile.Write([]byte(userInput))
util.ExitIfError(err)
_, err = tmpfile.Seek(0, 0)
util.ExitIfError(err)
return tmpfile
}
It pretty much looks like in your app your call Ask() whenever you want a single input line.
Inside Ask() you create a bufio.Reader to read from os.Stdin. Know that bufio.Reader–as its name suggests–uses buffered reading, meaning it may read more data from its source than what is returned by its methods (Reader.ReadString() in this case). Which means if you just use it to read one (or some) lines and you throw away the reader, you will throw away buffered, unread data.
So next time you call Ask() again, attempting to read from os.Stdin, you will not continue from where you left off...
To fix this issue, only create a single bufio.Reader from os.Stdin, store it in a global variable for example, and inside Ask(), always use this single reader. So buffered and unread data will not be lost between Ask() calls. Of course this solution will not be valid to call from multiple goroutines, but reading from a single os.Stdin isn't either.
For example:
var reader = bufio.NewReader(os.Stdin)
func Ask(question string, minLen int) string {
// use the global reader here...
}
Also note that using bufio.Scanner would be easier in your case. But again, bufio.Scanner may also read more data from its source than needed, so you have to use a shared bufio.Scanner here too. Also note that Reader.ReadString() returns you a string containing the delimeter (a line ending with \n in your case) which you probably have to trim, while Scanner.Text() (with the default line splitting function) will strip that first before returning the line. That's also a simplification you can take advantage of.
I am trying GO as a complete newbie to programming. I have a doubt in sequence inside the following code. This code scans for user input.
func main() {
fmt.Print("Enter a number: \n")
var input float64
fmt.Scanf("%f", &input)
output := input * 2
fmt.Println(output)
But, after the string display, at the new line when I enter any number, it should just go into a buffer of some sort or become junk value. I say so, because the allocation of scanned input starts after the first line. Had it been the first or prior step, it would perfectly make sense.
The value you enter is allocated at the input memory space (using the &). The input variable is created before the scanf (line 2) so there is no problem at all with the order of your instruction. Maybe you can clarify ?
I am using encoding/csv to read and parse a very large .csv file.
I need to randomly select lines and pass them through some test.
My current solution is to read the whole file like
reader := csv.NewReader(file)
lines, err := reader.ReadAll()
then randomly select lines from lines
The obvious problem is it takes a long time to read the whole thing and I need lots of memory.
Question:
my question is, encoding/csv gives me an io/reader is there a way to use that to read random lines instead of loading the whole thing at once?
This is more of a curiosity to learn more about io/reader than a practical question, since it is very likely that in the end it is more efficient to read it once and access it in memory, that to keep seeking random lines off on the disk.
Apokalyptik's answer is the closest to what you want. Readers are streamers so you can't just hop to a random place (per-se).
Naively choosing a probability against which you keep any given line as you read it in can lead to problems: you may get to the end of the file without holding enough lines of input, or you may be too quick to hold lines and not get a good sample. Either is much more likely than guessing correctly, since you don't know beforehand how many lines are in the file (unless you first iterate it once to count them).
What you really need is reservoir sampling.
Basically, read the file line-by-line. Each line, you choose whether to hold it like so: The first line you read, you have a 1/1 chance of holding it. After you read the second line, you have 1/2 chance of replacing what you're holding with this one. After the third line, you have a 1/2 * 2/3 = 1/3 chance of holding onto that one instead. Thus you have a 1/N chance of holding onto any given line, where N is the number of lines you've read in. Here's a more detailed look at the algorithm (don't try to implement it just from what I've told you in this paragraph alone).
The simplest solution would be to make a decision as you read each line whether to test it or throw it away... make your decision random so that you don't have the requirement of keeping the entire thing in RAM... then pass through the file once running your tests... you can also do this same style with non-random distribution tests (e.g. after X bytes, or x lines, etc)
My suggestion would be to randomize the input file in advance, e.g. using shuf
http://en.wikipedia.org/wiki/Shuf
Then you can simply read the first n lines as needed.
This doesn't help you learning more about io/readers, but might solve your problem nevertheless.
I had a similar need: to randomly read (specific) lines from a massive text file. I wrote a package that I call ramcsv to do this.
It first reads through the entire file once and marks the byte offset of each line (it stores this information in memory, but does not store the full line).
When you request a line number, it will transparently seek to the correct offset and give you the csv-parsed line.
(Note that the csv.Reader parameter that is passed as the second argument to ramcsv.New is used only to copy the settings into a new reader.) This could no doubt be made more efficient, but it was sufficient for my needs and spared me from reading a ~20GB text file into memory.
encoding/csv does not give you an io.Reader it gives you a csv.Reader (note the lack of package qualification on the definition of csv.NewReader [1] indicating that the Reader it returns belongs to the same package.
A csv.Reader implements only the methods you see there, so it looks like there is no way to do what you want short of writing your own CSV parser.
[1] http://golang.org/pkg/encoding/csv/#NewReader
Per this SO answer, there's a relatively memory efficient way to read a single random line from a large file.
package main
import (
"bufio"
"bytes"
"fmt"
"io"
"math/rand"
"strconv"
"time"
)
var words []byte
func main() {
prepareWordsVar()
var r = rand.New(rand.NewSource(time.Now().Unix()))
var line string
for len(line) == 0 {
line = getRandomLine(r)
}
fmt.Println(line)
}
func prepareWordsVar() {
base := []string{"some", "really", "file", "with", "many", "manyy", "manyyy", "manyyyy", "manyyyyy", "lines."}
words = make([]byte, 200*len(base))
for i := 0; i < 200; i++ {
for _, s := range base {
words = append(words, []byte(s+strconv.Itoa(i)+"\n")...)
}
}
}
func getRandomLine(r *rand.Rand) string {
wordsLen := int64(len(words))
offset := r.Int63n(wordsLen)
rd := bytes.NewReader(words)
scanner := bufio.NewScanner(rd)
_, _ = rd.Seek(offset, io.SeekStart)
// discard - bound to be partial line
if !scanner.Scan() {
return ""
}
scanner.Scan()
if err := scanner.Err(); err != nil {
fmt.Printf("err: %s\n", err)
return ""
}
// now we have a random line.
return scanner.Text()
}
Go Playground
Couple of caveats:
You should use crypto/rand if you need it to be cryptographically secure.
Note the bufio.Scanner's default MaxScanTokenSize, and adjust code accordingly.
As per original SO answer, this does introduce bias based on the length of the line.