does vim read the whole file into memory - performance

When I open a file, does vim read it all into memory? I experienced significant slowdowns when I open large files. Or is it busy computing something (e.g., line number)?

Disabling features like syntax highlighting, cursorline, line numbers and so on will greatly reduce the load and make Vim snappier in these cases.
There's even a plugin to handle that for you and a Vim tip for some background info.

Yes, Vim loads the whole file into memory.
If you run htop in another pane you can watch it happen in real time.
If you don't have enough memory available, it will start hitting the swap which makes it take even longer.
You can disable plugins and heavy features (like syntax highlighting) to get improved performance (the -u effectively tells vim not to load your ~/.vimrc):
vim -u NONE mysqldump.sql
However, unless you really need to edit the file, I prefer to just use a different tool. I typically use less. I mostly search the files in vim with / and less supports that just fine.
less mysqldump.sql

Related

How to set less to clear the screen on exit only if the output fills more than a single page?

I'm trying to figure out a way to pipe the output of a command (ag, in this case) to less -F (i.e. --quit-if-one-screen), but if the output is less than one page, the screen just flashes the content before it disappears. I've read that I can use -X (--no-init) to disable clearing the screen upon exiting less, but in that case long output doesn't get cleared either, which kinda defeats the purpose of a pager.
Is there a way to make less -X work with -F? I.e., to clear the output upon exiting less, except if the output fits in a single page?
It's 2018 now and Less is available in version 530. One of the key changes is the behavior of less -F with content of less than one full screen.
The solution is easy: Install Less 530 from your package repository, or download from Free Software Foundation and compile it yourself. Then you can have less -F leaving the content on screen if it doesn't fill up one full screen.
This very question has been answered in Unix.SE. The top-voted answer there has actually been expanded into a full-fledged command-line tool that can act as a replacement for less: https://github.com/stefanheule/smartless.
I've been using it myself with great results (plus the author is very responsive to bug reports and feature requests on Github), so I highly recommend it to anyone facing this issue.

(Mac)Vim quite slow when syntax is set to Ruby

I'm on MacVim 7.4 (I use the command line version), installed via Homebrew.
Vim is slow when syntax highlighting Ruby code. htop shows a 80%-100% CPU usage when moving inside a Ruby file in vim.
I found these here on SO:
Vim slow with ruby syntax highlighting
Syntax highlighting causes terrible lag in Vim
and tried the proposed solutions. What I did:
set regexpengine=1: nothing changed. Still very high CPU usage and slow performances.
set lazyredraw: things are better, but the tradeoff is very noticeable (cursor disappearing while moving)
I examined the autocmd statements in my .vimrc and found nothing slow in particular. I tried removing all the plugins but the problem is still there.
I tried turning the syntax off and, well, it solves the problem. Also, starting vim with vim -u NONE and then turning syntax on solves the problem, so it must be something in my .vimrc I guess?
Here's a link to my vimrc.
Edit
I may have found the guilty settings. It seems there are two settings that are noticeably slowing down movement in vim:
set relativenumber
set cursorline
Note that both of these settings trigger this behavior even alone.
These settings force vim to redraw quite a lot of stuff on the screen when scrolling holding j or k.
I doubt there's a solution here, but I'm very open to anything to speed this up.
Edit #2
Note that relativenumber and cursorline trigger this behavior only when used in Ruby files. Every other filetype I tried (with relativenumber and cursorline on) scrolls smoothly, no matter how long.
As avivr said, Vim is sometimes slow to (especially for ins-completion) in large files due to foldmethod=syntax
From :help todo:
Slow combination of folding and PHP syntax highlighting. Script to
reproduce it. Caused by "syntax sync fromstart" in combination with patch
7.2.274. (Christian Brabandt, 2010 May 27) Generally, folding with
'foldmethod' set to "syntax" is slow. Do profiling to find out why.
The FastFold plugin makes it so folds are only recalculated on save (so you're always using foldmethod=manual -- but the folds are calculated with foldmethod=syntax or whatever you had set before).
This solved the problem for me. Now I can use compl-whole-line completion in my 5000 line C++ file and it's instant and snappy instead of taking minutes and unresponsive.
Ruby syntax file has been known to be slow, it's better to disable 'cursorline', 'cursorcolumn' since they will cause the most effect.
However you should also have a look at Vim slow with ruby syntax highlighting. Something that could potentially also help improve things.
I had this issue, also tried set regexpengine=1 and various other things.
To me it seemed like the slowness was more or less severe depending on the size/complexity of file being edited, but it took a while to pinpoint the exact reason.
In my case the culprit was the following setting:
autocmd Filetype ruby setlocal foldmethod=syntax
This setting tells Vim to create folds automatically based on syntax elements (classes, functions, conditionals).
I don't think it matters that it was set with an autocmd in this case.
I think the problem was that for fairly complex files, the folds were updated during the editing process, causing dramatic slowness for complex files.
Hope this helps someone.

How to check Matplotlib's speed in Xcode and increase performance?

I'm running into some considerable speed bottlenecks with a Python-Matplotlib-Xcode combination. I know some immediate responses will probably ask "Why are you doing python stuff in Xcode, just man up and use vim" --> I like the organizing ability and the built in version control, it makes elements of my work easier to deal with.
Getting python to run in xcode in the first place was a bit more tricky than I had hoped, but its possible. Now I have the following scenario:
A master file, 'main.py' does all the import stuff for me and sets up some universal formatting to make all the figures (for eventual inclusion in my PhD thesis) nice and uniform. Afterwards it runs a series of execfile commands to generate whichever graphics I need. Two things I can think of right off the bat:
1) at the very beginning of main.py after I import all the normal python stuff you tend to need, I call a system script which checks whether a certain filesystem is mounted. I keep all my climate model data on there since my local hard drive is too small to deal with all of it at once. Python pauses itself and waits for the system to do its thing, but once the filesystem has been found, it keeps going. Usually this only needs to happen once in the morning when I get to work, or if the VPN server kicked me off for whatever reason. (Side question, it'd be cool to know if theres a trick to automate an VPN login to reconnect as soon as it notices its not connected)
2) I'm not sure how much xcode is using on its own. running the same program from terminal is (somewhat) faster. I've tried to be memory conscience and turn off stuff I don't need while running the python/xcode combination.
Also, python launches a little window whenever I call plt.show(), this in itself takes time, I've considered just saving them as quick png files and opening them with some other viewer, although I guess that would also have to somehow take time to open up. Given how often these graphics change as I add model runs or think of nicer ways of displaying the data, it'd be nice to not waste something on the order of 15 to 30 minutes (possibly more) out of the entire day twiddling my thumbs and waiting for a window to pop up.
Benchmark it!
import datetime
start = datetime.datetime.now()
# your plotting code
td = datetime.datetime.now() - start
print td.total_seconds() # requires python version >= 2.7
Run it in xcode and from the command line, see what the difference is.

Emacs: some programs only work in ansi-term, some programs only work in shell

Relative Emacs newbie here, just trying to adapt my programming workflow to fit with emacs. So far I've discovered shell-pop and I'm quite enjoying on-demand terminals that pop up when needed for banging out the odd commands.
What I understand so far about Emacs is that shell is a "dumb" terminal that doesn't support any ansi control codes, and that makes it incompatible with things like ncurses that attempt to draw complex UI's on a terminal emulator. This is why you can't use less or top or similar in shell-mode.
However, I seem to be having trouble with ansi-term, it's not the be-all, end-all that it's cracked up to be. Sure, it has no problems running less or git log or even nano, but there are a few things that can't quite seem to display properly when they're running in an ansi-term, such as apt-get and nosetests. I'm not sure quite what the name is for it, but apt-get's output is characterised by live-updating what is displayed on the very last line, and then having unchanging lines of text scroll out above that line. It seems to be halfway between something like less and something dumber, like cat. Somehow ansi-term doesn't like this at all, and I get very garbled output, where it seems to output everything on one line only or just generally lose it's place and output things all over, randomly. In the case of nosetests, it starts off ok, but if any libraries spew out any STDERR, the output all goes to hell in a similar way.
With some fiddling it seems possible to fix this by mashing C-l and RET, but it's not always reliable.
Does anybody know what's going on here? Is there some way to fix ansi-term so that it can display everything properly? Or is there perhaps some other mode that I don't know about that is way better? Ideally I'd like something that "just works" as effortlessly as, eg, Gnome Terminal, which can run all of the above mentioned programs without a single hiccup.
Thanks!
I resolved this issue by commenting out my entire .emacs.el and then uncommenting and restarting emacs for every single line in the file. I discovered that the following line alone was responsible for the issue:
'(fringe-mode 0 nil (fringe))
(this line disables the fringes from inside custom-set-variables).
I guess this is a bug in Emacs, that disabling the fringe causes term-mode to garble it's output really badly whenever any output line exceeds $COLUMN columns.
Anyway, I don't really like the fringes much at all, and it seems I was able to at least disable the left fringe without triggering this issue:
(set-fringe-mode (cons 0 8))
Maybe apt-get does different things based on the $TERM environment variable. What happens if you set TERM=dumb? If that makes things work, then you can experiment with different values until you find one that supports enough features but still works.
Note that git 2.0.1 (June 25th, 2014) now better detects dumb terminal when displaying verbose messages.
That might help Emacs better display some of the messages received from git, but the fringe-mode bug reported above is certainly the main cause.
See commit 38de156 by Michael Naumov (mnaoumov)
sideband.c: do not use ANSI control sequence on non-terminal
Diagnostic messages received on the sideband #2 from the server side are sent to the standard error with ANSI terminal control sequence "\033[K" that erases to the end of line appended at the end of each line.
However, some programs (e.g. GitExtensions for Windows) read and interpret and/or show the message without understanding the terminal control sequences, resulting them to be shown to their end users.
To help these programs, squelch the control sequence when the standard error stream is not being sent to a tty.

Diff-ing windows in vim

I am working on a script that has become fairly convoluted. I suspect there are several sections that have nearly identical code. Can I (and how can I) open the file in vim, with two (or more) windows on the buffer, and diff the contents of the windows on the same file? vimdiff seems to work only on two files. If I make a copy of the file and try to vimdiff the two versions, the diff origin remains locked on the beginning of the file. Although I can unscroll-lock the windows, and move the windows to the parts of the file I want to compare the diffs do not show up. Any hints or tips? I could cut and paste the sections I want to compare to different files and then apply vimdiff but then I risk getting lost in what section came from where when I try to patch the separate files together, and I feel sure there must be a more straightforward, easier way.
What I usually do is diff to a copy
:%w %.alt
:vert diffsplit %.alt
And then happiliy rearrange the 'alt' version so that the pseudo-matching bits get aligned.
Note that (presumably) git contains spiffy merge/diff cow-powers that should be able to detect sub-file moved block changes.
Although I haven't (yet) actually put this into practice, I have a hunch that the very nice git plugin fugitive for vim might be able to leverage some of this horsepower to make this easier. Note: fully expect this to require scriptinh before being usable, but I still thought it would be nice to share this idea (perhaps you can share a script if you get to it first!)
As an alternative solution that I've been using occasionally and which works very nicely in my opinion is linediff.vim.
It allows you to use visual mode to select two bodies of text from arbitrary buffers (or the same for that matter) and run vimdiff on them. The beauty of it, is that when you edit and save the temporary diff buffers, you update the original buffers with the changes, without saving.
One of my use-cases is when I'm resolving merge issues related to script refactoring and reordering, where a function has been moved and perhaps also modified. In order to make sure you do not lose any of the modifications coming in from either ancestor, you diff the two versions of the function alone by visually selecting them and running the linediff command.

Resources