(get-file-buffer filename) returns the buffer visiting filename, or nil if there is none.
https://www.gnu.org/software/emacs/manual/html_node/elisp/Buffer-File-Name.html
Edit: justburjustbur proposes using (find-buffer-visiting file) instead, as it also works when the buffer has a different name than the file.
So one solution would be sth in the lines of
(defun batch (file)
(let ((keep (find-buffer-visiting file)))
(find-file file)
(end-of-buffer)
(insert "nix")
(save-buffer)
(unless keep (kill-buffer)))
but there may be more elegant solutions.