EDIT: if you are trying to replace it, you need to use preg_replace_callback() instead, like so:
$c = preg_replace_callback('/\[gallery=([0-9]+)\]/', 'gallery', $m);
function gallery($m)
{
$id = (int) $m[1];
$result = mysql_query("select * from `table` where `id` = '$id'");
// Fetch $result and return the appropriate gallery code here
}
Old answer
You should use preg_match() to find the match because you're not trying to replace it with an SQL query, simply obtaining the value from the string to use in the SQL query. Try this:
$m = array();
preg_match('/\[gallery=([0-9]+)\]/', $content, $m);
$id = (int) $m[1]; // The value of the backreference $1
gallery($id);
Also I believe your gallery() function should return mysql_query() so you can analyze the result set of the query:
function gallery($id)
{
return mysql_query("select * from `table` where `id` = '$id'");
}