Git core uses libcurl for HTTP(S), with its various TLS backends. I don't know what Git-LFS uses for TLS, but if it relies on the same http.ssl* configuration as Git core according to your question, then it wouldn't be wise to use a format that Git-LFS supports but Git core does not. So I'll assume that you're configuring Git core and that Git-LFS expects exactly the same kind of configuration.
I am having some difficulty working out where each part of the keystore needs to be set in the http options
According to git help config, Git now supports PKCS#12 with certain TLS backends (practically – all of them as long as they're sufficiently recent):
If that doesn't work (and it probably won't because Git-LFS is written in Go and unlikely to be using libcurl), then the traditional "OpenSSL style" is to have two separate files for the certificate and the private key:
http.sslCertType should remain unset.
The path to the certificate goes into http.sslCert or GIT_SSL_CERT.
The certificate chain is expected to be in "PEM" text-based format, that is, each certificate individually Base64-encoded using "BEGIN CERTIFICATE" framing, and the leaf (client or server) certificate generally being the topmost one.
The path to the key file goes into http.sslKey or the GIT_SSL_KEY environment variable.
The key file is expected to be in "PEM" text-based format, that is, either PKCS#8 using the same kind of Base64 "BEGIN [ENCRYPTED] PRIVATE KEY" framing as with the certificate, or "legacy" PKCS#1 using very similar "BEGIN RSA PRIVATE KEY" framing.
Running just openssl pkcs12 -in $file will output both the certificate chain and the private key in the correct format.
Adding -nokeys will make it output only certificates, adding -nocerts will make it output only keys, and you can also split them using a text editor. (The extra text between the certificates and keys is ignored and just acts as a comment.)
Optionally you can try using -noenc or -nodes to output the key unencrypted for testing; later you can use openssl pkey -aes128 to (re-)encrypt it.
There's a possibility that pointing http.sslCert to a combined "certificate + PKCS#8 key" file will work, but I wouldn't bank on that.
I am unsure if git-bash on Windows would use the global OS user certificate store for LFS,
git-bash is irrelevant. It's a command-line shell.
The important part is libcurl for Git core, as it can be built either with SChannel or OpenSSL as the TLS backend, and the "Git for Windows" packaging seems to enable both with the http.sslBackend option to select between the two on the fly – if you choose http.sslBackend=schannel then you'll be using the Windows-provided TLS library, which will use Windows CAPI certificate and key storage, whereas if you choose openssl it expects the PEM-format files as described above.
Because Git-LFS is a separate program, I don't know whether it uses the same libcurl and whether it supports the same TLS backends, but given that it was written in Go and not C, I strongly suspect it does not.
and I cannot seem to figure out how one would install a client certificate system-wide on Ubuntu either.
There isn't a system-wide client certificate store on Ubuntu, as far as I know.
There's a PKCS#11-based framework (p11-kit) which ought to allow it to be implemented, but at the moment it has no default software-based backend that would actually work (in theory GNOME Keyring should have one; in practice I don't recall actually getting it to work) – and more importantly, OpenSSL is just… bad… at using PKCS#11. In older versions you'd need to specify http.sslKeyType=ENG (engine) but then there seems to be no option to tell Git or libcurl which engine to load (you'd need the 'pkcs11' engine from the libp11 package). In newer versions a similar situation exists with "providers" (with either 'pkcs11-provider' or 'libp11' providing a PKCS#11 provider, those now being loadable from openssl.cnf globally but breaking SSH by doing so).