How to create custom JQuery function and how to use it?

By "custom function" I am assuming you mean "plugin". If that's the case, there is a good tutorial on the jQuery site. The basic idea is this:

(function($) {
    $.fn.myPlugin = function() {
        return this.each(function() {
            //Do stuff
        });
    };
}(jQuery));

Basically, the code above does a few things. Firstly, it captures the value of jQuery and passes it into an anonymous function where it can then be referred to as $ (this is so that users of your plugin who happen to be using the $ identifier for something else can still use it.)

It then declares a method on $.fn, which is just an alias for $.prototype. Inside that method, this refers to the matched set of elements on which the plugin has been called. Since thats a jQuery object, and may contain multiple elements, you need to iterate over that set (that's why the each is there).

The return statement is used to maintain chainability of the plugin with other jQuery methods. Since each returns an instance of jQuery, the plugin itself returns an instance of jQuery, and other jQuery methods can obviously be called on an instance of jQuery.

As gesutery said, use extend(). You can add your own properties and functions as values:

        $(function(){
            $.extend({
                propAsString: '',
                propAsNumber: 12345,
                propAsObject: {},
                propAsFunction: function() {
                    //your function code
                }
            });

        $.propAsFunction();     //call your function
        });

you can use it like this

  $(document).ready(function() {
        $('#button').click(function(){
            $(this).myFunction();
         });
         $.fn.myFunction = function() { 
            alert('test'); 
         }
    });

(function($){
    $.fn.extend({ 
        //plugin name - animatemenu
        animateMenu: function(options) {

            //Settings list and the default values
            var defaults = {
                animatePadding: 60,
                defaultPadding: 10,
                evenColor: '#ccc',
                oddColor: '#eee'
            };

            var options = $.extend(defaults, options);

            return this.each(function() {
                var o =options;

                //Assign current element to variable, in this case is UL element
                var obj = $(this);              

                //Get all LI in the UL
                var items = $("li", obj);

                //Change the color according to odd and even rows
                $("li:even", obj).css('background-color', o.evenColor);             
                $("li:odd", obj).css('background-color', o.oddColor);                     

                //Attach mouseover and mouseout event to the LI  
                items.mouseover(function() {
                    $(this).animate({paddingLeft: o.animatePadding}, 300);

                }).mouseout(function() {
                    $(this).animate({paddingLeft: o.defaultPadding}, 300);
                });

            });
        }
    });
})(jQuery);

For those who are looking for a "custom function" as per the title, it is as simple as:

if(window.$)
    window.$.customMethod = function() { * your code here * };

This will then work like $.ajax() does

I wanted my custom functions to be invoked like $.myfunction(). I defined those functions in an external js file somewhat like this

$.myfunction = function(){

//Your code here    
};

Let's say you got the code below:

jQuery(document).ready(function($){
    function getFormData($form){
        var unindexed_array = $form.serializeArray();
        var indexed_array = {};

        $.map(unindexed_array, function(n, i){
            indexed_array[n['name']] = n['value'];
        });

        return indexed_array;
    }

    $("#form_name").submit(function (event) {
        var formData = getFormData($(this));
        console.log(formData);
        
        event.preventDefault();
    });
});

And it works just fine. But if you want to turn that function into a jQuery plugin, you can rewrite it as:

jQuery(document).ready(function($){

    $.fn.getFormData = function() {
        var unindexed_array = this.serializeArray();
        var indexed_array = {};

        $.map(unindexed_array, function(n, i){
            indexed_array[n['name']] = n['value'];
        });

        return indexed_array;
    };

    $("#form_name").submit(function (event) {
        var formData = $(this).getFormData();
        console.log(formData);

        event.preventDefault();
    });
});

That's it. You can find more information in the official JQuery documentation.