Here's a small experiment about handling boilerplate tasks via service pipelines. It's inspired bysimilar to the htmlHTML pipeline but forsimpler and multipurpose.
I've updated the question and this particular use case greatly simplifiedis the 2nd version of this project. I'llThe first one turned out to have some substantial weaknesses like inability to invoke more than one controller service to handle requests. The code is written in C# 10 and .net-6.
Example
Let's start with an examplea demo-app demonstrating how I'd use it:
public static class ServicePipelineDemoServicePipelineDemo2
{
public static async Task Test()
{
// Compose the container.
var builder = new ContainerBuilder();
// Register middlewarecache servicesdependency,
as a builder.RegisterInstance(new MemoryCache(new MemoryCacheOptions())).As<IMemoryCache>().SingleInstance();
// Register service-pipeline-builder and associate it with ReadFile.Text by name.
builder.Register(_c => new ServicePipeline<string>Service.PipelineBuilder
{
// This pipeline should resolve environment variables that might be used by the .Name property.
new EnvironmentVariableService<string>EnvironmentVariableService(PropertyService.For<IReadFile>.Select(x => x.Name)),
// OverrideAll cache-lifetimetext files should be cached for all30min txt-and other files for 15min.
new CacheLifetimeService<string>CacheLifetimeService(TimeSpan.FromMinutes(15))
{
Rules =
{
// Add a condition for read-file request.
Condition.For<IReadFile>.When(x => x.KeyName.EndsWith(".txt") ?, ".txt").Then(TimeSpan.FromMinutes(30))
: TimeSpan }
},
new CacheService(c.ZeroResolve<IMemoryCache>(), PropertyService.For<IReadFile>.Select(x => x.Name)),
// This pipeline should look in embedded resources first before it reads files.
// Since this node doesn't have to succeed, the next one is called.
new CacheService<string>(EmbeddedResourceService<ServicePipelineDemo2> { MustSucceed = false },
// This overrides the file-service for testing.
new MemoryCacheConstantService.Text("This is not a real file!"),
// Finally this node tries to read a file.
new MemoryCacheOptionsFileService.Read()))
}).InstancePerDependency().Named<ServicePipeline<string>>Named<Service.PipelineBuilder>(nameof(ReadFile.Text));
await using var container = builder.Build();
await using var scope = container.BeginLifetimeScope();
// Set some environment variable.
Environment.SetEnvironmentVariable("HOME", @"c:\temp");
// EverythingCreate isthe setuprequest soand let'sinvoke go!it.
var result =
await new ReadFile.Text(@"%HOME%\notes.txt")
.CacheLifetime(TimeSpan.FromMinutes(10))
.Bind(scope)
.InvokeAsync(scope);
Console.WriteLine(result); // --> c:\temp\notes"This is not a real file!" or FileNotFoundException (as there is no "notes.txttxt".
}
}
I'll try to anticipate and answer some of the questions that might arise:
FAQ
- Why do I use
new ReadFile.Textinstead of resolving the serivce via dependency injection? -- The constructor allows me to better enforce required parameters. This way I can also useSince this is only a consistentIService<T>interface for all servicesrequest and connect themdoesn't require any dependencies here, I'd say it's fine. Additionally it also specifies the expected return type. - How do I associate middlewarepipelines with the actual servicerequests? -- I use
Namedservices. By default the name of the request type is used (like Text here). Such a pipeline would apply to all requests of this type but it's possible to add aTagto theItemsdictionary and use this for the lookup. When there is a middleware then the target service is attached as the last node. - Why do I register the pipeline as a
Func<>? -- Every pipelineIt should be possible to create independent herepipelines for each request but at the actual usageend it depends on the actual use case so everything is possiblethere can be a single pipeline for a lifetime scope. - In which order are the middleware evaluated? -- They are evaluated in the same order as registered.
- Why Autofac? -- I like Autofac. Microsoft's DI is too limited.
- C# & .net versions? -- C# 10 and .net-6.
- Is there a GitHub link? -- There is - everything as a single fileThere is.
- What would be some other usage scenarios?
- I canTo completely replace the middleware with something else for testing and throw exceptions or return other results skippingwithout modyfing the actual service doing file readingrequesting code.
- I canTo add another middleware to create file copies or backups without modyfing the code where I would call
WriteFileis requested. - I canTo add anothervalidation middleware and validate files.
- I can create an email-service andTo add json de/serialization middleware attaching files or signatures.
- I canTo create an email-service and middleware attaching files or signatures.
- To create a config-service that today might read settings from
appsettings.jsonand tommorow from a database by only chanigng the middleware. Another middleware could at the same time validate settings. - I canTo add a telemetry middleware that would log the usage of each servicerequest.
- To add multiple services of the same type and try to read a file from different locations and use the first successful result.
- To add a retry middleware.
- ...
In this project every taskLike the HTTP pipeline there is represend by an interface andalso a base class that implements itrequest but no context and response objects. The result is returned directly. The request specifies what you'd like to do e.g.: ReadFile or CopyFile. It defines a linktravels through the pipeline where each node can do anything with it.
Nodes or middleware are called ISerivce. They point to the next one and have one additional property NextMustSucceed service. By default this is false for normal nodes and true for controller nodes like ReadFile when it's the last one in the chain and a method to invoke itresult must be returned or an exception should be thrown.
The Items dictionary allows to pass additional data.
public interface IItemsIRequest
{
//IDictionary<string, Allowsobject> toItems store{ additionalget; metadata.}
}
public interface IRequest<T> : IRequest { }
public abstract class Request<T> : IRequest<T>
{
public IDictionary<string, object> Items { get; } = new Dictionary<string, object>(SoftString.Comparer);
}
public sealed class Unit
{
private Unit() { }
public static readonly Unit Default = new();
}
public interface IService<T>IService : IItems, IEnumerable<IService<T>>IEnumerable<IService>
{
bool MustSucceed { get; }
// Points to the next service in a pipelinemiddleware.
IService<T>IService? Next { get; set; }
Task<T>Task<object> InvokeAsync(IRequest request);
}
public abstract class Service<T>Service : IService<T>IService
{
public IService<T>?bool NextMustSucceed { get; set; }
public IDictionary<string, object>IService? ItemsNext { get; } = new Dictionary<string,set; object>(SoftString.Comparer);}
public abstract Task<T>Task<object> InvokeAsync(IRequest request);
public IEnumerator<IService<T>>IEnumerator<IService> GetEnumerator() => this.Enumerate().GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
//protected Null-Serviceasync Task<object> InvokeNext(IRequest request)
{
return
Next is { } next
? await next.InvokeAsync(request)
public : Unit.Default;
}
private class Empty : Service<T>Service
{
public override async Task<T>Task<object> InvokeAsync(IRequest request) => await InvokeNext(request);
}
public class PipelineBuilder : IEnumerable<IService>
{
private IService First { get; } = new Empty();
// Adds the specified service at the end of the pipeline.
public void Add(IService last) => First.Enumerate().Last().Next? = last;
public IEnumerator<IService> GetEnumerator() => First.InvokeAsyncEnumerate().GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
public IService Build() => First;
}
}
The Service also provides the PipelineBuilder that knows how to connect serivces that are added to it.
For the sake of this experiment I'veI created a single servicetwo ReadFile that would normally read a file. Here it returns a string but it could also be a Stream etc.requests:
public abstract class ReadFile<T> : Service<T>, IVariable<string>, IAssociable<string>
{
protected ReadFile(string name) => Name = name;
public string Name { get; set; }
string IVariable<string>.Value
{
get => Name;
set => Name = value;
}
string IAssociable<string>.Key => Name;
}
public static class ReadFile
{
// Reads files as string.
public class Text : ReadFile<string>
{
public Text(string name) : base(name) { }
public override Task<string> InvokeAsync()
{
return Task.FromResult($"Hallo from '{Name}'.");
}
}
}
It implements a couple of interfaces that are supported by middleware services:
Textgets astringStreamgets aFileStream.
public interface IVariable<T>IReadFile : IRequest
{
public Tstring ValueName { get; set; }
public FileShare Share { get; }
}
public interfaceabstract IAssociable<outclass T>ReadFile<T> : Request<T>, IReadFile
{
protected ReadFile(string name) => Name = name;
public string Name { get; set; }
public TFileShare KeyShare { get; set; } = FileShare.Read;
}
public abstract class ReadFile
{
public class Text : ReadFile<string>
{
public Text(string name) : base(name) { }
public Encoding Encoding { get; set; } = Encoding.UTF8;
}
public class Stream : ReadFile<FileStream>
{
public Stream(string name) : base(name) { }
}
}
Middleware
Middleware
As far as reading files is concered there are twoit's common to perform such tasks that many apps implementas:
- resolving environment variables in paths andor
- caching results.
Both of them are handled by two middleware serivces that look for particular interfaces they support. TheThey work dynamically with properties specified via the constructor. (The cache-service is a dummy but the environment-variable service updates the file-name.)
// Provides caching for service results.
public class CacheService<T>EnvironmentVariableService : Service<T>Service
{
public CacheServiceEnvironmentVariableService(IMemoryCacheIPropertyAccessor<string> cacheproperty) => CacheProperty = cache;property;
private IMemoryCacheIPropertyAccessor<string> CacheProperty { get; }
public override async Task<T>Task<object> InvokeAsync(IRequest request)
{
if Property.SetValue(thisrequest, Environment.LastExpandEnvironmentVariables()Property.CacheLifetimeGetValue(request) is var cacheLifetime && cacheLifetime > TimeSpan.Zero)
{
Console.WriteLine(cacheLifetime);
}
return await Next?.InvokeAsyncInvokeNext(request);
}
}
// Allows to associate cache-lifetime with a service.
public class CacheLifetimeService<T>CacheService : Service<T>Service
{
public CacheLifetimeServiceCacheService(Func<IAssociable<string>IMemoryCache cache, TimeSpan>IPropertyAccessor<string> lifetimeFunckey) => LifetimeFunc(Cache, Key) = lifetimeFunc;(cache, key);
private Func<IAssociable<string>,IMemoryCache TimeSpan>Cache LifetimeFunc{ get; }
private IPropertyAccessor<string> Key { get; }
public override async Task<T>Task<object> InvokeAsync(IRequest request)
{
if (thisrequest.LastCacheLifetime() is var lastcacheLifetime && cacheLifetime > TimeSpan.Zero)
{
if Console.WriteLine(last$"Cache-lifetime: is{cacheLifetime}");
IAssociable<string> associable && LifetimeFunc(associable) is var lifetime)
}
return await {InvokeNext(request);
}
}
// Allows to associate cache-lifetime to with a service.
public class CacheLifetimeService : last.CacheLifetime(lifetime);Service
{
public CacheLifetimeService(TimeSpan fallback)
{
}
if (fallback }
== TimeSpan.Zero) throw new ArgumentException("Fallback value needs to returnbe awaitgreater Next?than zero.InvokeAsync(");
}
}
public class EnvironmentVariableService<T> : Service<T>
{
Fallback = fallback;
public override async}
Task<T> InvokeAsync()
private TimeSpan Fallback { get; }
public List<ConditionBag<TimeSpan>> Rules { ifget; (this.Last} = new();
is IVariable<string> identifiable)
public override async Task<object> InvokeAsync(IRequest request)
{
var lifetime = Rules.Where(c => identifiablec.ValueEvaluate(request)).Select(c ==> Environmentc.ExpandEnvironmentVariablesGetValue(identifiable)).ValueFirstOrDefault();
}request.CacheLifetime(lifetime > TimeSpan.Zero ? lifetime : Fallback);
return await Next?.InvokeAsyncInvokeNext(request)!;
}
}
Pipeline
Controllers
The pipeline is build bySerives that handle the SerivcePipelineReadFile. It knows how to connect serivces that request are added to the and of the pipeline.:
FileServiceandEmbeddedResourceService
I call them controllers although they are like every other middleware and can be placed anywhere in the pipeline.
//public Buildsabstract serviceclass pipeline.FileService : Service
{
protected FileService()
{
MustSucceed = true;
}
public class ServicePipeline<T>Read : IEnumerable<IService<T>>FileService
{
private IService<T> First public override async Task<object> InvokeAsync(IRequest request)
{
get; } if (request is IReadFile file)
{
if (File.Exists(file.Name))
{
if (request is ReadFile.Text t)
{
await using var stream = new Service<T>FileStream(file.EmptyName, FileMode.Open, FileAccess.Read, file.Share);
return await stream.ReadTextAsync(t.Encoding);
}
// Adds the specified service at the end of the pipeline if (request is ReadFile.Stream s)
public void Add {
return new FileStream(IService<T>file.Name, lastFileMode.Open, FileAccess.Read, file.Share).ToTask();
=> First }
}
else
{
return
MustSucceed
? throw DynamicException.EnumerateCreate("FileNotFound", $"There is no such file as '{file.Name}'.")
: await InvokeNext(request);
}
}
throw DynamicException.LastCreate("UnknownRequest", $"{request.GetType().NextToPrettyString()} is not supported by this {nameof(FileService)}.");
}
}
}
public class EmbeddedResourceService : Service
{
public EmbeddedResourceService(Assembly assembly)
{
Assembly = last;assembly;
MustSucceed = true;
}
private Assembly Assembly { get; }
public IEnumerator<IService<T>>override GetEnumeratorasync Task<object> InvokeAsync(IRequest request)
=> First {
if (request is IReadFile file)
{
var name = Normalize(file.EnumerateName);
if (FindResource(name) is { } stream)
{
if (request is ReadFile.GetEnumeratorText t)
{
await using (stream)
{
return await stream.ReadTextAsync();
}
IEnumerator IEnumerable }
if (request is ReadFile.GetEnumeratorStream)
{
return stream;
}
}
else
{
Console.WriteLine($"Embedded resource '{name}' not found.");
return
MustSucceed
? throw DynamicException.Create("EmbeddedResourceNotFound", $"There is no such file as '{file.Name}'.")
: await InvokeNext(request);
}
}
throw DynamicException.Create("UnknownRequest", $"{request.GetType().ToPrettyString()} is not supported by this {nameof(EmbeddedResourceService)}.");
}
// Embedded resource names are separated by '.' so replace the windows separator.
private static string Normalize(string name) => GetEnumeratorRegex.Replace(name, @"\\|\/", ".");
private Stream? FindResource(string name)
{
// Embedded resource names are case sensitive so find the actual name of the resource.
var actualName = Assembly.GetManifestResourceNames().FirstOrDefault(current => current.EndsWith(name, StringComparison.OrdinalIgnoreCase));
return actualName is { } ? Assembly.GetManifestResourceStream(actualName) : default;
}
}
public class EmbeddedResourceService<T> : EmbeddedResourceService
{
public EmbeddedResourceService() : base(typeof(T).Assembly) { }
}
Dependency injection
Dependency injection
There's no way it would work without dependency injection in the long run so of course there is one here too. It's supported where the pipelines is resolved from the container by the BindRequest helper extension InvokeAsync. It looks for properties marked withThis is where the request's DependencyAttribute<T> and triesparameter is used to resolve them. The below code contains also other helper extensions that I use in this projectcast the result to what is expected.
public static class ServiceBindingRequestExtensions
{
private static readonly ConcurrentDictionary<Type, IEnumerable<Dependency>> Properties = new();
// Resolves dependencies for properties marked with the [DependencyAttribute] and the service-pipeline.
public static IService<T>async Bind<T>Task<T> InvokeAsync<T>(this IService<T>IRequest<T> servicerequest, IComponentContext servicescomponents)
{
foreach (var dependency in Properties.GetOrAdd(typeof(T), type => type.DependentProperties().ToList()))
node {=
dependencycomponents.Resolve(service, services);
}
// To resolve the pipeline use either a custom tag or the typenameResolveOptionalNamed<Service.
var serviceTag = servicePipelineBuilder>(request.Tag()) is { } tag ? tag : service.GetType().Name;
return
// Try to resolve the middleware for this service.
services.ResolveOptionalNamed<ServicePipeline<string>>(serviceTag) is ServicePipeline<T> pipeline
// Attach current service at the end of the pipeline and return the first service.builder
? pipeline.Also(p => p.Last().Next = service)builder.FirstBuild()
// Otherwise use the current service.
: service;
}
// Gets info about dependentthrow propertiesDynamicException.
private static IEnumerable<Dependency> DependentPropertiesCreate(this IReflect type)
{
return
"PipelineNotFound", $"There is no frompipeline propertyto ininvoke type{request.GetPropertiesGetType(BindingFlags.Public | BindingFlags.Instance)
let dependency = property.GetCustomAttribute<DependencyAttribute>()
where dependency is { }
select new DependencyToPrettyString(property, dependency.Required);
}
// Enumerates service pipeline.
public static IEnumerable<IService<T>> Enumerate<T>(this IService<T> service")
{
for (var current = service; current is { }; current = current.Next)
{
yield return current;
}
}
// Stores property dependency info and resolvesawait itnode.
private record DependencyInvokeAsync(PropertyInfo Property, bool Requiredrequest)
{
public void Resolve(object target,is IComponentContextT scope)result
{
if? (Required)result
{
: throw PropertyDynamicException.SetValueCreate(target"Request", scope.Resolve(Property.PropertyType));
}
else
$"{
if (scoperequest.TryResolveGetType(Property.PropertyType, out var dependency))
{
Property.SetValueToPrettyString(target, dependency);
}
}
}
}
}
// Marks property thatdid requiresnot anreturn externalany dependency.
[AttributeUsage(AttributeTargetsresult.Property")]
public class DependencyAttribute : Attribute
{;
public bool Required { get; set; } = true;
}
Utilities
There's also a classThe first version used interfaces to access properties that handlesmiddleware nodes can modify or read. I found this was too inflexible so I created the ItemsPropertyAccessor to be able to select any property and makes it easiera Condition with ConditionBag to use by providing extensionsbe fully flexible when setting rules for commong properties:caching-lifetimes
PropertyAccesor
public staticinterface classIPropertyAccessor<TValue>
{
ServiceItems Func<object, TValue> GetValue { get; }
Action<object, TValue> SetValue { get; }
}
public record PropertyAccessor<TValue>(Func<object, TValue> GetValue, Action<object, TValue> SetValue) : IPropertyAccessor<TValue>;
public static Tclass GetItem<T>(thisPropertyService
{
IItems service private static readonly ConcurrentDictionary<(Type, string name)
, object> Cache = {new();
public abstract class For<TSource> where TSource : returnIRequest
{
service.Items.TryGetValue(typeofpublic static IPropertyAccessor<TValue> Select<TValue>(T).NameExpression<Func<TSource, out varTValue>> valueexpression)
&& value is T result {
if (expression.Body is not ?MemberExpression resultmemberExpression)
{
: throw DynamicException.Create("ItemNotFound",new $"CouldArgumentException($"Expression notmust findbe itema '{namenameof(MemberExpression)}'.");
}
public static T GetItemOrDefault<T>(this IItems service, string name, Treturn fallback(IPropertyAccessor<TValue>)Cache.GetOrAdd((typeof(TSource), memberExpression.Member.Name), _ =>
{
return{
service.Items.TryGetValue(name, out var value) &&var valuetargetParameter is= TExpression.Parameter(typeof(object), result"target");
?var result
valueParameter = Expression.Parameter(typeof(TValue), "value");
: fallback;
var casted = ParameterConverter<TSource>.Rewrite(memberExpression, }targetParameter);
// Sets CacheLifetime((T)target).Property
public static IService<T> CacheLifetime<T>(this IService<T> service, TimeSpan lifetime)
{ var getter =
return service.Also(s => s Expression.Items[nameofLambda<Func<object, TValue>>(CacheLifetime)]
= lifetime);
}
// Gets CacheLifetime. casted,
public static TimeSpan CacheLifetime<T>(this T service) where T : IItems
{ targetParameter
return service.GetItemOrDefault(nameof(CacheLifetime), TimeSpan ).ZeroCompile();
}
// Sets Tag((T)target).Property = value
public static IService<T> Tag<T>(this IService<T> service, string value)
{ var setter =
return service Expression.AlsoLambda<Action<object, TValue>>(s
=> s Expression.Items[nameofAssign(Tag)] =casted, valuevalueParameter);,
}
// Gets Tag.
public static string? Tag<T>(this T service) where T : IItems targetParameter, valueParameter
{
return service ).GetItemOrDefault(nameofCompile(Tag),;
default return new PropertyAccessor<TValue>(stringgetter, setter);
});
}
}
}
There's one more extension that I extensively use here. It allows me to create functional code without having to write code blocks with explicit returns in them.
Condition
/// <summary>
/// Allows to pipe an action on the current object in a functional way.
/// </summary>
[MustUseReturnValue]
public static T Also<T>(this T obj, Action<T>? next)
{
next?.Invoke(obj);
return obj;
}
public interface ICondition
{
Func<object, bool> Evaluate { get; }
}
public record Condition(Func<object, bool> Evaluate) : ICondition
{
private static readonly ConcurrentDictionary<(Type, string), object> Cache = new();
public abstract class For<TSource> where TSource : IRequest
{
public static ICondition When(Expression<Func<TSource, bool>> expression, string tag)
{
return (ICondition)Cache.GetOrAdd((typeof(TSource), tag), _ =>
{
var targetParameter = Expression.Parameter(typeof(object), "target");
var casted = ParameterConverter<TSource>.Rewrite(expression.Body, targetParameter);
// ((T)target)() -> bool
var evaluate =
Expression.Lambda<Func<object, bool>>(
casted,
targetParameter
).Compile();
return new Condition(evaluate);
});
}
}
}
public record ConditionBag<T>(Func<object, bool> Evaluate, Func<T> GetValue) : ICondition;
public static class ConditionBagFactory
{
public static ConditionBag<T> Then<T>(this ICondition condition, T value)
{
return new ConditionBag<T>(condition.Evaluate, () => value);
}
}