Grouping repeated XML elements into JSON arrays

With FOR XML PATH

As FOR XML PATH won't accept a GROUP BY, you'll have to add layers of subqueries, alternating subqueries that FOR XML PATH and subqueries that GROUP BY (this would of course be simplified if you used STRING_AGG() instead, but you probably have your reasons).

• After some pain with subqueries, I rewrote your query from subqueries to CTE, which (in my opinion) give a more intelligible way of reading the query with its sequenced steps… but adds a lot of verticality
• as pointed by @Charlieface, using the dedicated string_escape() will handle more cases than replace(…,'\','\\') (already a good start)
• a CROSS APPLY allows to remove one subquery / CTE, by computing your x and b tables in one pass
  (there probably is a way to simplify even more, by accessing /*/* or doing only 1 level deep in each call to Func_XmlToJson, but this was hard enough that I'm content to at least have a result, so optimizing could do another question on SO)
• thus element outputs (non-grouped) rows for each XML direct subelement
• then elgr will do the grouping by localname, concatenating all subelements with the same name, with [ and ] added only for a count of 2 or more (to get ["abc123","abc456"], but only "Book1" without brackets)
• we finally concat those possibly array subelements to construct the final JSON object
• Note that the GROUP BY will reorder the elements by alphabetical order, but hey, we're in JSON so this is semantically correct.

CREATE FUNCTION [dbo].[Func_XmlToJson](@XmlData xml)  
RETURNS nvarchar(max)  
AS  
BEGIN  
    DECLARE @m nvarchar(max);
    WITH
        element AS
        (
            SELECT
                coalesce(b.c.value('local-name(.)', 'NVARCHAR(255)'),'') localname,
                case when b.c.value('count(*)','int')=0   
                then  '"'+isnull(string_escape(b.c.value('text()[1]','NVARCHAR(MAX)'),'json'),'')+'"'  
                else dbo.Func_XmlToJson(b.c.query('.'))  
                end contents
            FROM @XmlData.nodes('/*') x(a)
            CROSS APPLY a.nodes('*') b(c)
        ),
        elgr AS
        (
            SELECT
                localname,
                CASE
                    WHEN count(*) = 1 THEN MAX(contents)
                    ELSE
                    '['
                    +
                    STUFF
                    (
                        (
                            SELECT ','+contents
                            FROM element eall
                            WHERE eall.localname = element.localname
                            FOR XML PATH(''), TYPE
                        ).value('.', 'NVARCHAR(MAX)'),
                        1,1,''
                    )
                    +
                    ']'
                    END
                contents
            FROM element
            GROUP BY localname
        )
        SELECT @m='{' + STUFF((SELECT ',"'+localname+'":'+  
                  contents FROM elgr
             FOR XML PATH(''), TYPE).value('(./text())[1]','NVARCHAR(MAX)')  
           ,1,1,'')+'}'
      ;
   RETURN @m;
END;

Which on your example returns:

{
    "Book": "Book1",
    "Edition": [
        {
            "Color": "Red",
            "Name": "Ed1",
            "Price": "100"
        },
        {
            "Color": "Blue",
            "Name": "Ed2",
            "Price": "200"
        }
    ],
    "Publisher": "Amazon",
    "PublisherId": "1",
    "TransactionId": [
        "abc123",
        "abc456"
    ],
    "UserId": "1234"
}

(as seen on this SQL Server 2014 fiddle)

With STRING_AGG

You didn't give your reasons for using FOR XML PATH as a way to STRING_AGG, so I'm supposing that you are limited to an old (< 2017) version of SQL Server.
So, for your information, here is what you could do in SQL Server 2017: same structure, but way simpler (and obvious) concatenating over the GROUP BY:

-- With STRING_AGG instead of the 2 STUFF(…) occurrences:
CREATE FUNCTION [dbo].[Func_XmlToJson](@XmlData xml)  
RETURNS nvarchar(max)  
AS  
BEGIN  
    DECLARE @m nvarchar(max);
    WITH
        element AS
        (
            SELECT
                coalesce(b.c.value('local-name(.)', 'NVARCHAR(255)'),'') localname,
                case when b.c.value('count(*)','int')=0   
                then  '"'+isnull(string_escape(b.c.value('text()[1]','NVARCHAR(MAX)'),'json'),'')+'"'  
                else dbo.Func_XmlToJson(b.c.query('.'))  
                end contents
            FROM @XmlData.nodes('/*') x(a)
            CROSS APPLY a.nodes('*') b(c)
        ),
        elgr AS
        (
            SELECT
                localname,
                CASE
                    WHEN count(*) = 1 THEN MAX(contents)
                    ELSE '['+STRING_AGG(contents, ',')+']'
                END contents
            FROM element
            GROUP BY localname
        )
        SELECT @m='{'+STRING_AGG('"'+localname+'":'+contents, ',')+'}'
        FROM elgr
   ;
   RETURN @m;
END;

(last function definition in this fiddle running SQL Server 2017)