csvquote

smart and simple CSV processing on the command line

Are you looking for a way to process CSV data with standard UNIX shell commands?

Are you running into problems with embedded commas and newlines that mess
everything up?

Do you wish there was some way to add some CSV intelligence to these UNIX tools?

  • awk, sed
  • cut, join
  • head, tail
  • sort, uniq
  • wc, split

This program can be used at the start and end of a text processing pipeline
so that regular unix command line tools can properly handle CSV data that
contain commas and newlines inside quoted data fields.

Without this program, embedded special characters would be incorrectly
interpretated as separators when they are inside quoted data fields.

By using csvquote, you temporarily replace the special characters inside quoted
fields with harmless nonprinting characters that can be processed as data by
regular text tools. At the end of processing the text, these nonprinting
characters are restored to their previous values.

In short, csvquote wraps the pipeline of UNIX commands to let them work on
clean data that is consistently separated, with no ambiguous special
characters present inside the data fields.

By default, the program expects to use these as special characters:

" quote character  
, field delimiter  
\n record separator  

It is possible to specify different characters for the field and record
separators, such as tabs or pipe symbols.

Note that the quote character can be contained inside a quoted field
by repeating it twice, eg.

field1,"field2, has a comma in it","field 3 has a ""Quoted String"" in it"

Typical usage of csvquote is as part of a command line pipe, to permit
the regular unix text-manipulating commands to avoid misinterpreting
special characters found inside fields. eg.

csvquote foobar.csv | cut -d ',' -f 5 | sort | uniq -c | csvquote -u

or taking input from stdin,

cat foobar.csv | csvquote | cut -d ',' -f 7,4,2 | csvquote -u

other examples:

csvquote -t foobar.tsv | wc -l

csvquote -q "'" foobar.csv | sort -t, -k3 | csvquote -u

csvquote foobar.csv | awk -F, '{sum+=$3} END {print sum}'

Installation

To install using go, clone the repo and then build it:

> go build -o csvquote cmd/cvsquote/main.go
> cp ./csvquote /usr/local/bin

To install using Earthly (linux):

earthly github.com/adamgordonbell/csvquote+build
cp ./csvquote /usr/local/bin

Install on MacOS (X86)

earthly github.com/adamgordonbell/csvquote+for-darwin-amd64
cp ./csvquote /usr/local/bin

Install on Windows

earthly github.com/adamgordonbell/csvquote+for-windows-amd64
# Then add to your path

History

This is a fork of original version in C by Dan Brown found here:
https://github.com/dbro/csvquote

More specifically, this is a fork of a fork of the C version. It is based on a GoLang version Dan wrote at some point. Go makes cross-compilation easier.

GitHub

View Github