How can I use dependency injection with .NET Core 2 API options?

However, my interface is registered as a transient service...

Yes, but Json.NET is not integrated with ASP.NET Core dependency injection. So the deserializer just does not know how to create an instance of ITestInterface.

do I need to set something else in the JSON options or use my own contract resolver.

Yes, you should put some efforts to cover such a scenario and a custom contract resolver seems the most proper way for this. The steps that you should take are the following:

1. Define a custom contract resolver. You could base it on DefaultContractResolver used by Json.NET. You should pass instance of IServiceProvider to this resolver which will be used for instantiation of registered types.

2. In Startup.ConfigureServices() register all types that could be used by contract resolver during Json.NET deserialization.

3. Create instance of custom contract resolver and set it to SerializerSettings.ContractResolver. In Startup.ConfigureServices() you don't yet have access to IServiceProvider because it's not yet created (since services are still registered). However, you could create it by yourself by calling services.BuildServiceProvider(). But make sure you have registered all required services before this call.

Here is a sample that performs all described steps:

DependencyInjectionContractResolver class:

public class DependencyInjectionContractResolver : DefaultContractResolver
{
    private readonly IServiceProvider serviceProvider;
    private readonly List<Type> registeredTypes;

    public DependencyInjectionContractResolver(IServiceProvider serviceProvider, IEnumerable<Type> registeredTypes)
    {
        this.serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
        this.registeredTypes = registeredTypes?.ToList() ?? throw new ArgumentNullException(nameof(registeredTypes));
    }

    protected override JsonContract CreateContract(Type objectType)
    {
        JsonContract contract = base.CreateContract(objectType);

        if (registeredTypes.Contains(objectType))
        {
            contract.DefaultCreator = () => serviceProvider.GetService(objectType);
        }

        return contract;
    }
}

Startup.ConfigureServices(IServiceCollection services) method:

public void ConfigureServices(IServiceCollection services)
{
    services.AddTransient<ITestInterface, TestClass>();
    //    Register all other services here
    //    ...

    services.AddMvc(options => options.RespectBrowserAcceptHeader = true)
        .AddXmlDataContractSerializerFormatters()
        .AddXmlSerializerFormatters()
        .AddJsonOptions(o =>
        {
            o.SerializerSettings.ContractResolver = new DependencyInjectionContractResolver(services.BuildServiceProvider(),
                new[] { typeof(ITestInterface) } );

            o.SerializerSettings.Formatting = Newtonsoft.Json.Formatting.Indented;
        });
}