0

I'm trying to figure out the best flow to insert/update a document into elasticsearch (or any nosql database really) after creating a new row in a SQL database, and where to put the logic in Elixir Phoenix.

Imagine my SQL tables are posts has_many comments, and my elasticsearch structure has a posts index where comments is a nested document, so I could search/retrieve the post and comment content at the same time. Meaning if a user creates or updates their comment's message, the related post document in elasticsearch should update.

What would be the best approach?

1 Answer 1

1

The exact implementation probably depends on a couple things: how fast are the operations and does the user need to know the outcome?

If both operations are reasonably fast (i.e. the user can wait), then you could use a simple with statement to ensure both successfully complete, e.g.

with {:ok, _result1} <- update_postgres(data),
  {:ok, _result2} <- update_elasticsearch(data) do
   {:ok, "Everything updated!"}
end

If you need a bit more formality around "binding" 2 operations together, then an Ecto transaction does the job. The docs usually assume that the operations are both database operations or that they use the same Repo, but you can do arbitrary tasks in your transaction, e.g. updating Elasticache:

  def two_things(data) do
    My.Repo.transaction fn ->
      with {:ok, _result1} <- update_postgres(data),
           {:ok, _result2} <- update_elasticache(data)
      do
        {:ok, "Everything updated!"}
      else
        {:error, e} -> 
           My.Repo.rollback(e)
           {:error, "Something failed!"}
    
      end
    end
  end

Using a transaction only really makes sense when you need to perform a rollback on failure.

The other option here assumes that the operations might be slow enough that you wouldn't want to keep the user waiting around for them to complete. Usually the "primary" operation of critical importance is updating the database. Sometimes the other related updates (e.g. in cache layers) are best left to an asynchronous side-effect. For example:

result = update_postgres(data)
Task.async(fn -> update_elasticache(data) end)
result

Or perhaps trigger the side-effect only if the database operation is successful:

case update_postgres(data) do
  {:ok, result} -> 
    Task.async(fn -> update_elasticache(data) end)
    {:ok, result}
  {:error, error} -> {:error, error}
end

Or a bit cleaner syntax:

with {:ok, result} <- update_postgres(data) do
  Task.async(fn -> update_elasticache(data) end)
    {:ok, result}
end

There are many syntaxes/helpers that accomplish this, but the idea is that any secondary operations are non-blocking and can complete after returning a result to the user.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.