RFC 4180 compliant CSV parsing and encoding for Elixir.
Add
{:csv, "~> 3.0"}
to your deps in mix.exs
like so:
defp deps do
[
{:csv, "~> 3.0"}
]
end
CSV is a notoriously unstable format, with many implementations interpreting the standard differently. This often leads to situations where large datasets can not be parsed completely because of a few or even just one line deviating from the standard.
CSV
has two decoding modes, normal CSV.decode
and strict CSV.decode!
.
Normal mode will return tuples, ok: ["field1", "field2"]
for correctly formatted rows, and err: "Message"
for incorrectly formatted ones. This makes this mode well suited for extracting all correctly formatted data
from a file while avoiding having to manipulate the file.
If you want the decoder to fail on the first error, use strict mode instead. It will raise the first error it sees as an exception.
Parallelism has been replaced by a binary matching parser in version 3.x. This library is able to parse about half a million rows of a moderately complex CSV file per second in a single process, ensuring that parsing will unlikely be the bottleneck of any operation.
TBD
- Elixir
1.5.0
is required for all versions above2.5.0
. - Elixir
1.1.0
is required for all versions above1.1.5
.
There are two interesting things you want to do regarding CSV - encoding end decoding.
Do this to decode:
File.stream!("data.csv") |> CSV.decode
And you'll get a stream of row tuples:
[ok: ["a", "b"], ok: ["c", "d"]]
And, potentially error tuples:
[error: "", ok: ["c", "d"]]
Use the bang to decode! into a two-dimensional list, raising errors as they occur:
File.stream!("data.csv") |> CSV.decode!
Be sure to read more about decode
and its angry sibling decode!
Do this to encode a table (two-dimensional list):
table_data |> CSV.encode
And you'll get a stream of lines ready to be written to an IO. So, this is writing to a file:
file = File.open!("test.csv", [:write, :utf8])
table_data |> CSV.encode |> Enum.each(&IO.write(file, &1))
Pass in another separator to the decoder:
File.stream!("data.csv") |> CSV.decode(separator: ?\t)
If you want to take revenge on whoever did this to you, encode with semicolons like this:
your_data |> CSV.encode(separator: ?;)
You can also specify headers when encoding, which will encode map values into the right place:
[%{"a" => "value!"}] |> CSV.encode(headers: ["z", "a"])
# ["z,a\\r\\n", ",value!\\r\\n"]
You can also specify a keyword list, the keys of the list will be used as the keys for the rows, but the values will be the value used for the header row name in CSV output
[%{a: "value!"}] |> CSV.encode(headers: [a: "x", b: "y"])
# ["x,y\\r\\n", "value!,\\r\\n"]
You'll surely appreciate some more info on encode
.
Make sure your data gets encoded the way you want - implement the CSV.Encode
protocol for whatever strange you wish to encode:
defimpl CSV.Encode, for: MyData do
def encode(%MyData{has: fun}, env \\ []) do
"so much #{fun}" |> CSV.Encode.encode(env)
end
end
Or similar.
The encoding protocol implements a fallback to Any for types where a simple call
o to_string
will provide unambiguous results. Protocol dispatch for the
fallback to Any is very slow when protocols are not consolidated, so make sure
you have consolidate_protocols: true
in your mix.exs
or you consolidate protocols manually for production in order
to get good performance.
There is more to know about everything ™️ - Check the doc
Please make sure to add tests. I will not look at PRs that are either failing or lowering coverage. Also, solve one problem at a time.
Copyright (c) 2022 Beat Richartz
CSV source code is licensed under the MIT License.