How to take data from a CSV and insert into Postgres in Elixir

Question

I am using a CSV module to import a CSV file, this stores the data in a map, that all works, giving me the usual data mapping result. I then need to take that mapped data and send it to ecto.repo. This is using the standard attrs argument that is then unpacked and sent to the changeset to be sent to the database. How do I iterate over each key and return the value and then take each sub value and pass it to ecto.repo.create_object?

%{{:ok, ["key", "key", "key"]} => {:ok, ["value", "value", "value"}}..}

All the keys and values are mapped exactly the same, so each key is the same value as the value. The other thing is, I don't know the name of the key so I can't call each key. Furthermore each set of key, value pairs is a separate line in the CSV file, and I need to pass each line into ecto.repo.create_object to be entered into the database. Basically creating a new record for each line of the CSV file. What is the best way to achieve this?

Aleksei Matiushkin · Accepted Answer · 2018-03-20 06:06:04Z

3

I’d use Kernel.SpecialForms.for/1 comprehension here:

data = %{{:ok, ["k1", "k2", "k3"]} => {:ok, ["v1", "v2", "v3"]},
         {:ok, ["k21", "k22", "k23"]} => {:ok, ["v21", "v22", "v23"]}}

for {{:ok, keys}, {:ok, values}} <- data, do: Enum.zip(keys, values)
#⇒ [
#   [{"k1", "v1"}, {"k2", "v2"}, {"k3", "v3"}],
#   [{"k21", "v21"}, {"k22", "v22"}, {"k23", "v23"}]
# ]

Optionally one might construct a list of Keywords or Maps

 for {{:ok, keys}, {:ok, values}} <- data do
   keys
   # NB! If and only keys do not come from user input
   #     Please check the comment below for details
   # |> Enum.map(&String.to_atom/1)
   |> Enum.zip(values)   # keywords
   |> Enum.into(%{})     # maps
 end
 #⇒ [%{k1: "v1", k2: "v2", k3: "v3"},
 #   %{k21: "v21", k22: "v22", k23: "v23"}]

edited Mar 20, 2018 at 6:06

answered Mar 19, 2018 at 7:49

Aleksei Matiushkin

121k12 gold badges109 silver badges174 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Patrick Oscity Over a year ago

Be careful with String.to_atom/1 if the CSV is user input and/or the keys are not always the same. The OP said "I don't know the name of the key", so this seems to be the case. Atoms are not garbage collected and you'll eventually run out of RAM and crash the Erlang VM. I'd advise to keep the keys as strings to be safe.

Aleksei Matiushkin Over a year ago

@PatrickOscity yes, thanks, I have updated the answer; I am always under an impression that Atom DOS is well-known.

Collectives™ on Stack Overflow

How to take data from a CSV and insert into Postgres in Elixir

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related