How to improve performance of SQL query with parameters?

Well, the explanation seems simple enough. For your current condition, since @Param1=0 is false (you set the parameter to 1 previously), it needs to evaluate your second condition, wich has a subquery and might take a long time. If you change your first filter to @Param1=1, then you are saying that it is true and there is no need to evaluate your second filter, hence making your query faster.

You seem to be confusing the optimiser with your OR statement. If you remove it, you should find it generates two different execution plans for the SELECT statement - one with a filter, and the other without:

DECLARE @Param1 BIT
SET @Param1 = 1

if @Param1=0
begin
    SELECT 
        t1.Col1,
        t1.Col2
    FROM
        Table1 t1
end
else
begin
    SELECT 
        t1.Col1,
        t1.Col2
    FROM
        Table1 t1
    WHERE
        (t1.Col2 in 
            (SELECT  t2.Col4
                FROM 
                    Table2 t2
                WHERE 
                    t2.Col1 = t1.Col1 AND 
                    t2.Col2 = 'AAA' AND 
                    t2.t3 <> 0)
        )
end

This is commonly referred to as the N+1 problem.
You're doing a select in table 1 and for each record you find you will go look for something in table 2.
By setting your @Param1 to an value which will never be found in your select the sql engine will skip the subquery.
To avoid this behavior you could use a JOIN statement to join both tables together and afterwards filter the results with a where statement. The join statement will be a bit slower then a single subquery because you are matching 2 tables to eachother but because you only need to do the join once(vs N times) you'll gain a serious performance boost.
Example code :

DECLARE @Param1 BIT
SET @Param1 = 1

SELECT t1.Col1,t1.Col2
FROM Table1 t1
INNER JOIN Table2 t2 on t1.Col1 = t2.Col1
WHERE @Param1=0 
OR t2.Col2 = 'AAA' 
AND t2.t3 <> 0