Tables in a relational database represent unordered sets. There is no such thing as the "order of rows" in a table.
If you need a specific sort order, the only way is to use an order by in a select statement as you did.
If you don't want to type the order by each time, you can create a view that does that:
create view sorted_price
as
select *
from price
order by created_on;
But be warned: if you sort the rows from the view in a different way, e.g. select * from sorted_price order by created_on desc Postgres will actually apply two sorts. The query optimizer is unfortunately not smart enough to remove the one store in the view's definition.