It appears that SQL Server will use the lower storage option first (5 bytes), before reaching the max for that precision (9 bytes). But both (11,3) and (18,3) max out at 9 bytes. See script below.
As for your conversion of 1 billion rows, since it will be a widening conversion I don't think you will have any problems. You should test this with as close to to he full data set as possible. I'm not sure how long this alter would actually take.
USE tempdb
GO
CREATE TABLE MyTest (TestCol NUMERIC(11,3))
INSERT INTO dbo.MyTest ( TestCol ) VALUES( 99999999.999 )
INSERT INTO dbo.MyTest ( TestCol ) VALUES( 9.999 )
SELECT Testcol, DATALENGTH(Testcol) AS Size
FROM dbo.MyTest
CREATE TABLE MyTest2 (TestCol NUMERIC(18,3))
INSERT INTO dbo.MyTest2 ( TestCol ) VALUES( 999999999999999.999 )
INSERT INTO dbo.MyTest2 ( TestCol ) VALUES( 9.999 )
SELECT Testcol, DATALENGTH(Testcol) AS Size
FROM dbo.MyTest2
DROP TABLE dbo.MyTest
DROP TABLE dbo.MyTest2