GatsbyJS is a great static site generator, but I think it is possible to just use create-react-app to produce a build that can be put on a static server.
Read the create-react-app documentation about serving apps with client-side-routing
Normally yarn build produces a build directory which contains everything you need to put on a (static) webserver.
There is however a problem when you use React Router with browserHistory. A static file server doesn't know how to handle requests for specific pages and hand them over to react router.
You can however use React Router with hashHistory. Use this when you set up ReactRouter:
<Router history={hashHistory}>
It will add # strings (hash strings) at the end of your URLs. The router uses the information contained in the string to render the correct components for the particular page that is requested. That way your server doesn't need any configuration and you can just use a static server.
Here is some info on the difference between browser history (nice looking url's) and hash history (# sign in the url)
/static/bundletostatic/bundle), otherwise I don't think it will work properly.