How do I open a file in its default program - Linux

You need to run gnome-open, kde-open, or exo-open, depending on which desktop you are using.

I believe there is a project called xdg-utils that attempts to provide a unified interface to the local desktop.

So, something like:

snprintf(s, sizeof s, "%s %s", "xdg-open", the_file);
system(s);

Beware of code injection. It's safer to bypass scripting layers with user input, so consider something like:

pid = fork();
if (pid == 0) {
  execl("/usr/bin/xdg-open", "xdg-open", the_file, (char *)0);
  exit(1);
}
// parent will usually wait for child here

Ubuntu 10.10 is based on GNOME. So, it would be good idea to use g_app_info_launch_default_for_uri().

Something like this should work.

#include <stdio.h>
#include <gio/gio.h>

int main(int argc, char *argv[])
{
        gboolean ret;
        GError *error = NULL;

        g_type_init();

        ret = g_app_info_launch_default_for_uri("file:///etc/passwd",
                                                NULL,
                                                &error);
        if (ret)
                g_message("worked");
        else
                g_message("nop: %s", error->message);

        return 0;
}

BTW, xdg-open, a shell script, tries to determin your desktop environment and call a known helper like gvfs-open for GNOME, kde-open for KDE, or something else. gvfs-open ends up calling g_app_info_launch_default_for_uri().

A simple solution with less coding:

I've tested this program on my Ubuntu and it is working fine, and if I am not wrong you are looking for something like this

#include <stdio.h>
#include <stdlib.h>

int main()
{
    system("firefox file:///dox/song.mp3");
    return 0;
}