Your inner literal tags are backwards, and you are attempting to nest additional literals inside a {literal} tag (which obviously will not work because {literal} tells Smarty to leave the contents unchanged.) It should probably look like this instead:
{literal}
<script type="text/javascript">
jQuery(document).ready(function($){
$("#lbgallery").click(function() {
{/literal} {if $photos} $.lightbox(["{"\", \""|implode:$photos}"]); {/if}{literal}
return false;
});
});
</script>
{/literal}
Notice the two literal tags inside your outer literal tags have been reversed. You need to first end the literal, then re-start it again, otherwise the contents of the inner {literal} tags will also be treated as literal, and not evaluated - and that just doesn't make sense for what you are trying to do :)