Better CSVs
I'd like to share a little Neovim plugin I've been writing in my spare time called better-csvs.nvim. It's a plugin designed to improve the native CSV file editing experience. If you're interested, go try it out!
Comma Separated Values ¶
A CSV file is a plaintext file format that represents a table of data. Each line represents a row and each column is separated by a delimiter (usually a comma but it can also be a tab or a semi-colon).
For example, this CSV file:
Name,Price,Qty,Total
Bananas,$1.50,3,$4.50
Mandarin oranges,$2.00,2,$4.00
Translates into this table:
| Name | Price | Qty | Total |
|---|---|---|---|
| Bananas | $1.50 | 3 | $4.50 |
| Mandarin oranges | $2.00 | 2 | $4.00 |
Except... the file itself doesn't look very pretty. It only gets worse when you work with bigger CSV files.
Motivation ¶
I decided to make a Neovim plugin for editing CSVs because editing these files by hand is painful. The columns rarely ever line up and it's terribly easy to forget a comma or miss a column because of the plain formatting.
To be clear, I'm aware other programs exist for editing these kinds of files. I've used Excel and Google Sheets but I wanted something with Vim keybindings that didn't rely on network connectivity. I actually used sc-im for a very long time for this reason, but it wasn't as configurable as I wanted it to be. Plus, there's no official build for Alpine Linux yet.
When I learned that Neovim 0.10+ supports inline virtual text I decided to look back into this and see if I could use that to my advantage.
Features ¶
What does this plugin do? First and foremost, it consistently spaces all columns, giving you the look and feel of a real data table. It makes this:
Date,Name,Unit,Qty,Total,Notes
2023-10-09,ALL PURP GREEN LID PAIL,,,5.87,Home Depot
2023-10-09,NON SCRATCH SCRUB SPONGE,,,3.98
2023-10-09,AA 6-PACK,,,7.87
2023-10-09,BLOCK SANDING SPONGE 3PK,,,8.48
2023-10-09,DRYWALL SANDING RESP N95,,,7.47
2023-10-09,KT KLEIN-KURVE STRPR CT,,,20.97
2023-10-09,8IN WRENCH,,,13.97
2023-10-09,4IN HMMERKNF,,,11.98
2023-10-09,SALES TAX,,,6.04
Display like this:
Date |Name |Unit |Qty |Total |Notes
2023-10-09 |ALL PURP GREEN LID PAIL | | |5.87 |Home Depot
2023-10-09 |NON SCRATCH SCRUB SPONGE | | |3.98
2023-10-09 |AA 6-PACK | | |7.87
2023-10-09 |BLOCK SANDING SPONGE 3PK | | |8.48
2023-10-09 |DRYWALL SANDING RESP N95 | | |7.47
2023-10-09 |KT KLEIN-KURVE STRPR CT | | |20.97
2023-10-09 |8IN WRENCH | | |13.97
2023-10-09 |4IN HMMERKNF | | |11.98
2023-10-09 |SALES TAX | | |6.04
It doesn't modify the file in any way. The spaces are purely visual (your cursor will skip over the spaces).
I also added support for arithmetic and basic SUM formulas similar to Excel. It makes this:
Date,Name,Unit,Qty,Total
2024-08-07,ETN 15A 120/277V SP DECO,,,2.36
2024-08-07,ETN 15A 120/277V 3W DEC L,13.28,3.00,=C3*D3
2024-08-07,TAX,,,0.18
2024-08-07,ETN 15A 120/277V DECO LIT,=9.98-0.5,2.00,=C5*D5
2024-08-07,0.5-CU FT PEA GRAVEL,=5.58-0.28,3.00,=C6*D6
2024-08-07,6-FT T-POST GREEN 1.25-LB,=6.98-0.35,4.00,=C7*D7
,,,YR 1 TOTAL,=SUM(E2:E7)
Display like this:
Date |Name |Unit |Qty |Total
2024-08-07 |ETN 15A 120/277V SP DECO | | |2.36
2024-08-07 |ETN 15A 120/277V 3W DEC L |13.28 |3.00 |=C3*D3
=39.84
2024-08-07 |TAX | | |0.18
2024-08-07 |ETN 15A 120/277V DECO LIT |=9.98-0.5 |2.00 |=C5*D5
=9.48 =18.96
2024-08-07 |0.5-CU FT PEA GRAVEL |=5.58-0.28 |3.00 |=C6*D6
=5.3 =15.9
2024-08-07 |6-FT T-POST GREEN 1.25-LB |=6.98-0.35 |4.00 |=C7*D7
=6.63 =26.52
| | |YR 1 TOTAL |=SUM(E2:E7)
=103.76
Once again, the file is not modified. The calculations are inserted into "virtual" lines that only affect the layout of the buffer. The sums only work in one direction (horizontal/vertical) and must be specified with =SUM(lowest:highest). What's great about this is that Google Sheets and Excel support importing text formulas as real formulas, so you can write up an invoice or budget plan in a CSV format and import it into Google Sheets at any point in the future.
How It Works ¶
The plugins works by using a Neovim feature called extmarks. Extmarks are annotations you can insert in a buffer that only exist visually but do not affect the contents of that buffer. A lot of plugins actually use this feature to display all sorts of things in the buffer's margins such as language server messages, notes to self, or git blame information[1]. This plugin parses each line, calculates the greatest width each column can be, then inserts inline extmarks to add padding to each columns to make them align. I also use conceallevel to hide the CSV separators and display them as | symbols. If your cursor is on that line, they reveal the actual separators typed in the file.
And then there's the formula calculations. I maintain a few tables in lua to store information about each cell in the data table so it's relatively straightforward to perform calculations referencing other cells. If a cell starts with = then we evaluate the value of that cell using load() and display it using another extmark. The hardest part is sanitizing the input because a user could type:
Date,Name
2024,=print(1)
I allowlist certain arithmetic characters (0-9, asterisks, parentheses, etc) using the expression [^%d-.*/+()^]+ then search for =SUM() expressions and evaluate those manually. I also check to prevent infinite recursion in cases like the following:
Date,Name
2024,=B2
All of this is calculated and refreshed every time the buffer changes (on InputLeave, TextChanged, BufRead, or BufNewFile) because I wanted the experience to be snappy.
This probably doesn't scale well with large CSVs in the range of millions of cells, but it works fine for my purposes.
Conclusion ¶
I had a lot of fun writing this and I'm glad to be able to finally read formatted CSVs with the power of Vim keybindings. I hope you like my plugin! If you have any code suggestions, send me an email or a patch and I'll take a look.
Shamless plug for git-blame.nvim. It's such a great plugin that performs exceptionally well compared to other git plugins. ↩︎