Prelude

It seems that several of your questions lately have revolved around trying to prevent React component rerenders. This is fine and well, but don't spend too much time prematurely optimizing. React runs quite well out-of-the-box.

Regarding memo HOC and optimizing performance, even the docs state outright:

This method only exists as a performance optimization. Do not rely on it to “prevent” a render, as this can lead to bugs.

This means React may still rerender a component if it needs to. I believe mapping the messages array is one of these cases. When the messages state updates it's a new array, and so it must be rerendered. React's reconciliation needs to rerender the array and the each element of the array, but it may not need to go any deeper.

You could test this by adding a memoized child component to ChatLine and watch as even though ChatLine is wrapped in memo HOC that it is still rerendered while the memoized child is not.

const Child = memo(({ id }) => {
  useEffect(() => {
    console.log('Child rendered', id); // <-- doesn't log when messages updates
  })
  return <>Child: {id}</>;
});

...

const ChatLine = (props) => {
  ...

  useEffect(() => {
    console.log("Chatline rendered", line.id); // <-- logs when messages updates
  });

  return (
    <div>
      ...
          <Child id={line.id} />
      ...
    </div>
  );
};

export default memo(ChatLine);

The takeaway here should be that you shouldn't prematurely optimize. Tools like memoization and virtualization should only be looked at if you find an actual performance issue and have properly benchmarked/audited performance.

You shouldn't also "over-optimize" either. The React app I develop for a client I work with we did this early on thinking we were saving ourselves time but eventually over time (and as we've gained familiarity with React hooks) we've ripped out most or nearly all of our "optimizations" as they ended up not really saving us much and add more complexity. We eventually found our performance bottlenecks that had more to do with our architecture and component composition than it did with the # of components rendered in lists.

Suggested Solution

So you were using the useChatService custom hook in several components but as written each hook was its own instance and providing its own copy of the messages state and other various callbacks. This is why you had to pass the messages state as a prop from ChatSection to Chat. Here I suggest to move the messages state and callbacks into a React context so each useChatService hook "instance" can provide the same context value.

useChatService

(could probably be renamed since more than just a hook now)

Create a context:

export const ChatServiceContext = createContext({
  messages: [],
  updateMessage: () => {},
  addMessage: () => {},
  deleteMessage: () => {}
});

Create a context provider:

getChatAndUnsub wasn't awaiting anything so there was no reason to declare it async. Memoize all the callbacks to add, update, and delete messages.

const ChatServiceProvider = ({ children }) => {
  const [messages, setMessages] = useState([]);

  const getChatAndUnsub = () => {
    const q = query(collection(db, "messages"), orderBy("createdAt"));

    const unsubscribe = onSnapshot(q, (snapshot) => {
      const data = snapshot.docs.map((doc, index) => {
        const entry = doc.data();

        return { .... };
      });

      setMessages(data);
    });

    return unsubscribe;
  };

  useEffect(() => {
    const unsubscribe = getChatAndUnsub();

    return () => {
      unsubscribe();
    };
  }, []);

  const updateMessage = useCallback(async (message, id) => {
    try {
      await updateDoc(doc(db, "messages", id), {
        message,
        updatedAt: serverTimestamp()
      });
    } catch (err) {
      console.log(err);
    }
  }, []);

  const addMessage = useCallback(async (message) => {
    if (!message) {
      return;
    }

    const messageData = { .... };

    try {
      await addDoc(collection(db, "messages"), messageData);
    } catch (e) {
      console.log(e);
    }
  }, []);

  const deleteMessage = useCallback(async (idToDelete) => {
    if (!idToDelete) {
      return;
    }
    try {
      await deleteDoc(doc(db, "messages", idToDelete));
    } catch (err) {
      console.log(err);
    }
  }, []);

  const keyGen = () => { .... };

  return (
    <ChatServiceContext.Provider
      value={{
        messages,
        updateMessage,
        addMessage,
        deleteMessage
      }}
    >
      {children}
    </ChatServiceContext.Provider>
  );
};

export default ChatServiceProvider;

Create the useChatService hook:

export const useChatService = () => useContext(ChatServiceContext);

Provide the Chat service to the app

index.js

import ChatServiceProvider from "./hooks/useChatService";

ReactDOM.render(
  <React.StrictMode>
    <ChatServiceProvider>
      <App />
    </ChatServiceProvider>
  </React.StrictMode>,
  document.getElementById("root")
);

ChatSection

Use the useChatService hook to consume the messages state.

const ChatSection = () => {
  const { messages } = useChatService();

  return (
    <div>
      {messages.length ? <Chat /> : <NoChat />}
      <p>ADD A MESSAGE</p>
      <ChatInput />
    </div>
  );
};

export default ChatSection;

Chat

Remove the editing state and setters (more on this later). Use the useChatService hook to consume the messages state.

const Chat = () => {
  const { messages } = useChatService();

  return (
    <div>
      <p>MESSAGES :</p>
      {messages.map((line) => (
        <ChatLine key={line.id} line={line} />
      ))}
    </div>
  );
};

export default Chat;

ChatLine

Move the editing state here. Instead of an editingId state use a boolean toggle for an edit mode. Encapsulate the edit id in the updateMessage callback from the context. Manage all the editing state here locally, don't pass the state values and setter as callbacks for another component to call. Note that the EditMessage component API was updated.

const ChatLine = ({ line }) => {
  const [editValue, setEditValue] = useState("");
  const [isEditing, setIsEditing] = useState(false);

  const { updateMessage, deleteMessage } = useChatService();

  return (
    <div>
      {!isEditing ? (
        <>
          <span style={{ marginRight: "20px" }}>{line.id}: </span>
          <span style={{ marginRight: "20px" }}>[{line.displayName}]</span>
          <span style={{ marginRight: "20px" }}>{line.message}</span>
          <button
            onClick={() => {
              setIsEditing(true);
              setEditValue(line.message);
            }}
          >
            EDIT
          </button>
          <button
            onClick={() => {
              deleteMessage(line.id);
            }}
          >
            DELETE
          </button>
        </>
      ) : (
        <EditMessage
          value={editValue}
          onChange={setEditValue}
          onSave={() => {
            // updating message in DB
            updateMessage(editValue, line.id);
            setEditValue("");
            setIsEditing(false);
          }}
          onCancel={() => setIsEditing(false)}
        />
      )}
    </div>
  );
};

Here you can use the memo HOC. You can further hint to React that maybe this component shouldn't rerender if the line id remains equal, but recall that this doesn't completely prevent the component from being rerendered. It's only a hint that maybe React can bail on rerenders.

export default memo(ChatLine, (prev, next) => {
  return prev.line.id === next.line.id;
});

EditMessage

Just proxy the props to their respective props of the textarea and button. In other words, let ChatLine maintain the state it needs.

const EditMessage = ({ value, onChange, onSave, onCancel }) => {
  return (
    <div>
      <textarea
        onKeyPress={(e) => {
          if (e.key === "Enter") {
            // prevent textarea default behaviour (line break on Enter)
            e.preventDefault();
            onSave();
          }
        }}
        onChange={(e) => onChange(e.target.value)}
        value={value}
        autoFocus
      />
      <button type="button" onClick={onCancel}>
        CANCEL
      </button>
    </div>
  );
};

export default EditMessage;

ChatInput

Consume addMessage from the useChatService hook. I don't think much changed here but including anyway for completeness' sake.

const ChatInput = () => {
  const [inputValue, setInputValue] = useState("");
  const { addMessage } = useChatService();

  return (
    <textarea
      onKeyPress={(e) => {
        if (e.key === "Enter") {
          e.preventDefault();
          addMessage(inputValue);
          setInputValue("");
        }
      }}
      placeholder="new message..."
      onChange={(e) => {
        setInputValue(e.target.value);
      }}
      value={inputValue}
      autoFocus
    />
  );
};

export default ChatInput;