If memory serves me right, since 6-th version Xcode doesn't support so-called "universal" frameworks (frameworks that contains architectures for arm-family and x86_64/i386). So now when you build a framework Xcode will make two separate bundles for iphoneos and iphonesimulator (you can find them under corresponding folders in your derived data folder). The architectures these frameworks will contain depends on ARCHS variable. By default it's set to $ARCHS_STANDARD, that varies between platforms. You can play around with this setting, mixing architectures you really need, but Xcode will fail at build time if these architectures are incompatible (presumably x86_64 and arm-family architectures are considered incompatible).
To get round this limitation you actually have to do a little bit of "hard-work" yourself and "merge" framework bundles for simulators and devices manually.
1. Build your framework for iOS device and simulator
Here is nothing fancy, just go to your target build settings and ensure that "Build Active Architecture Only" (ONLY_ACTIVE_ARCH) is set to NO and add all required architectures for "Valid Architectures"(VALID_ARCHS, you have already done that):

Now build your framework, find the bundle file under the Product group in Xcode, and open it in Finder:

You should find two folders, one for each set of architectures:

2. Merge two frameworks into one
Now go into Debug-iphoneos folder, copy framework from there and paste it somewhere else, for example in the parent folder:

It will contain our universal framework in a short while. In order to do that we need first create universal dynamic library with the lipo tool. Open terminal, navigate to the folder where you are now (with the copied framework file) and type as follow:
$ lipo -create -output 'MyFramework' 'Debug-iphoneos/MyFramework.framework/MyFramework' 'Debug-iphonesimulators/MyFramework.framework/MyFramework'
If you are not in the derived data folder your paths for the framework libraries will differ of course. This command will produce new universal library containing both sets of architectures. Now just drag and drop it into your framework bundle and replace the existing one.
It's likely that your framework has architecture slices under a folder like MyFramework.framework/Modules/MyFramework.swiftmodule. Our universal framework should have slices for all supported architectures. You should already have arm slices inside, since we copied our framework from the Debug-iphoneos folder, so just find the same folder for the Debug-iphonesimulator and copy files from the folder to the destination framework's folder. You now should have this folder look something like that:

That's it! Now you have a universal framework that should work properly on both an iOS simulator and an iOS device.
3. Slice your framework when submitting to the Appstore
If you try to submit an application alongside a universal framework, it will be automatically rejected. So before submitting you will need to extract only devices' architectures one by one and then merge them into one framework using lipo:
$lipo -extract 'arm' 'MyFramework.framework/MyFramework' -output 'MyFramework-arm'
$lipo -extract 'arm64' 'MyFramework.framework/MyFramework' -output 'MyFramework-arm64'
$lipo -create -output 'MyFramework' 'MyFramework-arm' 'MyFramework-arm64'
Then copy it into your framework and remove redundant slices.
P.S.
Hopefully it helps, however this is kind of well known issue and there is already a lot of solutions in the internet. I suggest you googling terms like 'Create fat/universal framework iOS', and I believe you will find a lot of working scripts that will do all this stuff for you. Here is one I find the most helpful.