Changing symbol with different pattern in a text file

Python 3 script solution

The script below is an explicit more "manual" solution that doesn't use regular expressions, but does the job with a few extra considerations. The key to its operation is that the script opens for reading whatever file we've provided on command-line, iterates over each character in each line, seeking to find brackets. If we see brackets, we record whatever is inside, and after throwing away commas we decide if that's a numerical string or not. If it is a numerical string - the recorded items go into list of words, which is later rebuild into a line using .join() function with ", " as separator. Fairly straight-forward.

 #!/usr/bin/env python3
import sys

with open(sys.argv[1]) as fd:
    for line in fd:
        # we're going to store everything into list of words
        # and record individual characters into 'item' string
        # and rebuild everything as we go.
        words = []
        item_found = False
        item = ""
        counter = 0
        for char in line:
            # if we see ( or [ we start recording chars
            # difference is that [ means item already been edited
            # so no need to do anything - just put it into words list
            # as is
            if char == "(" or char == "[":
                item_found = True
                counter = counter + 1
                continue

            if char == ")":
                item_found = False
                if item.replace(",","").isdigit():
                   words.append("[" + item + "]")
                else:
                   words.append("("+item+")")
                item = ""

            if char == "]":
              item_found = False
              item = item + char
              words.append("[" + item) 
              item = ""

            if item_found:
                item = item + char

        # if we didn't see any open brackets or no closing brackets
        # just print the line as is - otherwise give us the altered one
        if counter == 0 or item_found:
            print(line.strip())
        else:
            print(", ".join(words))

Test run:

I took liberty with OP's input to include extra 2 lines that include couple different test cases.

$ # original input file
$ cat input.txt
(1), (3), (1,2,3), (1,2,3,4,5,6,7,8,9), (Fig1) (Fig1,Fig2), (Table-1, Table-2)
(table-25),[1,2,3],(figure-35)
(figure-1),(figure-2)
$ # script output 
$ ./change_brackets.py input.txt                                                          
[1], [3], [1,2,3], [1,2,3,4,5,6,7,8,9], (Fig1), (Fig1,Fig2), (Table-1, Table-2)
(table-25), [1,2,3], (figure-35)
(figure-1), (figure-2)

With 40,000 lines of text it performs fairly quick:

$ wc -l big_input.txt                                                                     
40000 big_input.txt
$ time ./change_brackets.py big_input.txt  > /dev/null                                    
    0m01.64s real     0m01.60s user     0m00.01s system

Possible suggestion for improvement ( and to address one of the things that Stephane has mentioned ) is to change if item.replace(",","").isdigit() line into if item.replace(",","").replace(".","").isdigit(). This will allow us to deal with floating point numbers (such as 3.1415) as well.

Lengthy ? Yes. Explicit ? Yes. Works ? Well, yes.

sed -E 's/[(](([0-9]+,?)+)[)]/[\1]/g'

sed -e 's/(\(\([0-9]\{1,\},\{0,\}\)\{1,\}\))/[\1]/g'

We are looking for the regex [0-9]+,? occurring at least once and at the same time bounded by ( and ) brackets. The outer brackets are changed to [ ] on success.

The second sed statement is the POSIXly version of the above.