List<IRemoteStreamContent> when submitting http request has stream inconsistency. #9077

elie.khadij created 7 days ago

Hello, we are using IRemoteStreamContent to upload images in our solution.
let's say we have a DTO called A identified like so:

public record A {
    public string Name {get; init;}
    public B Bs {get; init;}
    public List <IRemoteStreamContent> Images {get; init} = [];
}

Let's say we have an Application Service named TestApplicationService. In this service, I'm calling a Custom Blob Service that uploads Azure blob images.

public class TestAppService : AppService {
    public Task<bool> CreateRecordA(A dto)
    {
        foreach(var image in dto.Images)
        {
         await blobService.SaveBlobAsync(image);
        }
        
        return true;
    }
}

as for the Custom blob Service I will expose the method as follows:

    public async Task<bool> SaveBlobAsync(IRemoteStreamContent file, CancellationToken cancellationToken = default)
    {
        using var memoryStream = new MemoryStream();
        await file.GetStream().CopyToAsync(memoryStream, cancellationToken);
        var fileContent = memoryStream.ToArray();
        await memoryStream.FlushAsync(cancellationToken);
        await blobContainer.SaveAsync(CleanFileName(file.FileName!), fileContent, true, cancellationToken);

        return true;
    }

In our host module we added this line in conventional controllers:

        Configure<AbpAspNetCoreMvcOptions>(options =>
        {
            options.ConventionalControllers.FormBodyBindingIgnoredTypes.Add(typeof(A));
            options.ConventionalControllers.Create(typeof(TestApplicationModule).Assembly);
        });

The problem is that whenever I have more than 1 image in my dto we are facing an exception of HTTP stream positioning changed unexpectedly, noting that we followed step by step
https://abp.io/docs/latest/framework/architecture/domain-driven-design/application-services#miscellaneous

Exception message and full stack trace:

System.InvalidOperationException: The inner stream position has changed unexpectedly.
   at Microsoft.AspNetCore.Http.ReferenceReadStream.VerifyPosition()
   at Microsoft.AspNetCore.Http.ReferenceReadStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
   at Totssy_Backend.BuildingBlocks.Services.Blob.BlobService.SaveBlobAsync(IRemoteStreamContent file, CancellationToken cancellationToken) in C:\Repos\Lykos\Totssy-Backend\src\Totssy_Backend.BuildingBlocks\Services\Blob\BlobService.cs:line 80
   at StockManager.Services.Catalog.CatalogsAppService.PostProductAsync(CreateProductDto productDto)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

Steps to reproduce the issue:

elie.khadij created 6 days ago

Do you have any reply? The problem is from an Extension method you created that is resetting the stream position CopyToAsync(). It is overriding the virtual method of Stream! it is resetting the position to 0 while on Http requests Asp net core itself manages the streams

maliming created 5 days ago

Support Team Fullstack Developer

Can you share full logs?

Thanks.

elie.khadij created 5 days ago

Hello

this is the full log. If you want we can schedule a Teams meeting to share my findings more in details since we are on NDA.

regards
Elie

maliming created 5 days ago

Support Team Fullstack Developer

Can you share the logs that contain the http request info?

The context logs .

https://abp.io/support/questions/8622/How-to-enable-Debug-logs-for-troubleshoot-problems

You can share it with liming.ma@volosoft.com

Thanks

elie.khadij created 5 days ago

Debug mode did not show the FormData anything required from my side?

regards
Elie

maliming created 5 days ago

Support Team Fullstack Developer

Can you just share the full logs?

I will check it.

Thanks.

elie.khadij created 5 days ago

Hello

sorry there was an override line on Microsoft log to Information, logs shared

thanks

maliming created 4 days ago

Support Team Fullstack Developer

Thanks. I will check it asap.

👍

maliming created 4 days ago

Support Team Fullstack Developer

I test a controller with two files.

But I didn't get the exception. Can you share a test project?

Thanks.

screenshot-2025-04-2 T05-22-00@2x.png

screenshot-2025-04-2 T05-21-38@2x.png

screenshot-2025-04-2 T05-21-49@2x.png

elie.khadij created 4 days ago

Hello maliming
Thank you for your reply, but the shared example is incompatible with my use case since you separated the streams into separate objects. Please try to put them in a List<IRemoteStreamContent>, and you will see that it won't work. If you try the example CreateMultipleFileInput here https://abp.io/docs/latest/framework/architecture/domain-driven-design/application-services#miscellaneous in the documentation, you will realize that the example shared does not work. I already shared an example with you when I opened the support ticket.

I encourage you to jump on a Teams call to elaborate more about this matter. Would Monday work for you?

maliming created 4 days ago

Support Team Fullstack Developer

Here is my test code, it works. What should I change to reproduce the exception?

Thanks

public class HomeController : AbpController
{
    public async Task<ActionResult> Index(MyFiles file)
    {
        using (var ms = new MemoryStream())
        {
            await file.Files.First().GetStream().CopyToAsync(ms);
            var filename = file.Files.First().FileName;
            var fileContent = ms.ToArray();
        }

        using (var ms = new MemoryStream())
        {
            await file.Files.Last().GetStream().CopyToAsync(ms);
            var filename = file.Files.Last().FileName;
            var fileContent = ms.ToArray();
        }

        return new OkResult();
    }
}

public class MyFiles
{
    public List<IRemoteStreamContent> Files { get; set; }
}

elie.khadij created 4 days ago

we are using Conventional Controllers, like the application Service. Maybe is it when you inject a scoped service and manipulate the stream from within?
The blow method is part of a custom IBlobService that we created, which has a scoped life cycle. and I'm calling its method in my application service.

    // IBlobService
    public async Task SaveBlobAsync(IRemoteStreamContent file, CancellationToken cancellationToken = default)
   {
       using var memoryStream = new MemoryStream();
       await file.GetStream().CopyToAsync(memoryStream, cancellationToken); // this line throwing exception since it is touching stream position of HTTP request
       var fileContent = memoryStream.ToArray();
       await memoryStream.FlushAsync(cancellationToken);
       await blobContainer.SaveAsync(CleanFileName(file.FileName!), fileContent, true, cancellationToken);

       return true;
   }
   // end blobService
   
   public class MyApplicationService(IBlobService blobService) : IMyApplicationService
   {
       public async Task TestAsync(List files)
       {
           foreach(var file in files)
           {
               await blobService.SaveBlobAsync(file);
           }
       }
   }

maliming created 4 days ago

Support Team Fullstack Developer

Can you share a test project?
I will download and check it ,then I will find out the reason.
Thanks.

elie.khadij created 4 days ago

Ok i will share a test sample within 2 3 hours. Do you want it via email or discord or what are your steps

maliming created 4 days ago

Support Team Fullstack Developer

liming.ma@volosoft.com

elie.khadij created 4 days ago

Hey

I shared with you via email github repository of the test solution

regards

maliming created 3 days ago

Support Team Fullstack Developer

The fix code:

AbpSolution1HttpApiHostModule:

public override void ConfigureServices(ServiceConfigurationContext context)
{


    PostConfigure<MvcOptions>(options =>
    {
        options.ModelBinderProviders.RemoveAll(x => x.GetType() == typeof(AbpRemoteStreamContentModelBinderProvider));
        options.ModelBinderProviders.Insert(2, new MyAbpRemoteStreamContentModelBinderProvider());
    });
}

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc.ModelBinding;
using Microsoft.AspNetCore.Mvc.ModelBinding.Validation;
using Volo.Abp.AspNetCore.Mvc.ContentFormatters;
using Volo.Abp.Content;

namespace AbpSolution1;

public class MyAbpRemoteStreamContentModelBinderProvider : IModelBinderProvider
{
    public IModelBinder? GetBinder(ModelBinderProviderContext context)
    {
        if (context == null)
        {
            throw new ArgumentNullException(nameof(context));
        }

        if (context.Metadata.ModelType == typeof(RemoteStreamContent) ||
            typeof(IEnumerable<RemoteStreamContent>).IsAssignableFrom(context.Metadata.ModelType))
        {
            return new MyAbpRemoteStreamContentModelBinder<RemoteStreamContent>();
        }

        if (context.Metadata.ModelType == typeof(IRemoteStreamContent) ||
            typeof(IEnumerable<IRemoteStreamContent>).IsAssignableFrom(context.Metadata.ModelType))
        {
            return new MyAbpRemoteStreamContentModelBinder<IRemoteStreamContent>();
        }

        return null;
    }
}

public class MyAbpRemoteStreamContentModelBinder<TRemoteStreamContent> : IModelBinder
    where TRemoteStreamContent: class, IRemoteStreamContent
{
    public async Task BindModelAsync(ModelBindingContext bindingContext)
    {
        if (bindingContext == null)
        {
            throw new ArgumentNullException(nameof(bindingContext));
        }

        var postedFiles = GetCompatibleCollection<TRemoteStreamContent>(bindingContext);

        // If we're at the top level, then use the FieldName (parameter or property name).
        // This handles the fact that there will be nothing in the ValueProviders for this parameter
        // and so we'll do the right thing even though we 'fell-back' to the empty prefix.
        var modelName = bindingContext.IsTopLevelObject
            ? bindingContext.BinderModelName ?? bindingContext.FieldName
            : bindingContext.ModelName;

        await GetFormFilesAsync(modelName, bindingContext, postedFiles);

        // If ParameterBinder incorrectly overrode ModelName, fall back to OriginalModelName prefix. Comparisons
        // are tedious because e.g. top-level parameter or property is named Blah and it contains a BlahBlah
        // property. OriginalModelName may be null in tests.
        if (postedFiles.Count == 0 &&
            bindingContext.OriginalModelName != null &&
            !string.Equals(modelName, bindingContext.OriginalModelName, StringComparison.Ordinal) &&
            !modelName.StartsWith(bindingContext.OriginalModelName + "[", StringComparison.Ordinal) &&
            !modelName.StartsWith(bindingContext.OriginalModelName + ".", StringComparison.Ordinal))
        {
            modelName = ModelNames.CreatePropertyModelName(bindingContext.OriginalModelName, modelName);
            await GetFormFilesAsync(modelName, bindingContext, postedFiles);
        }

        object value;
        if (bindingContext.ModelType == typeof(TRemoteStreamContent))
        {
            if (postedFiles.Count == 0)
            {
                // Silently fail if the named file does not exist in the request.
                return;
            }

            value = postedFiles.First();
        }
        else
        {
            if (postedFiles.Count == 0 && !bindingContext.IsTopLevelObject)
            {
                // Silently fail if no files match. Will bind to an empty collection (treat empty as a success
                // case and not reach here) if binding to a top-level object.
                return;
            }

            // Perform any final type mangling needed.
            var modelType = bindingContext.ModelType;
            if (modelType == typeof(TRemoteStreamContent[]))
            {
                value = postedFiles.ToArray();
            }
            else
            {
                value = postedFiles;
            }
        }

        // We need to add a ValidationState entry because the modelName might be non-standard. Otherwise
        // the entry we create in model state might not be marked as valid.
        bindingContext.ValidationState.Add(value, new ValidationStateEntry()
        {
            Key = modelName,
        });

        bindingContext.ModelState.SetModelValue(
            modelName,
            rawValue: null,
            attemptedValue: null);

        bindingContext.Result = ModelBindingResult.Success(value);
    }

    private async Task GetFormFilesAsync(
        string modelName,
        ModelBindingContext bindingContext,
        ICollection<TRemoteStreamContent> postedFiles)
    {
        var request = bindingContext.HttpContext.Request;
        if (request.HasFormContentType)
        {
            var form = await request.ReadFormAsync();

            var useMemoryStream = form.Files.Count > 1;
            foreach (var file in form.Files)
            {
                // If there is an <input type="file" ... /> in the form and is left blank.
                if (file.Length == 0 && string.IsNullOrEmpty(file.FileName))
                {
                    continue;
                }

                if (file.Name.Equals(modelName, StringComparison.OrdinalIgnoreCase))
                {
                    if (useMemoryStream)
                    {
                        var memoryStream = new MemoryStream();
                        await file.OpenReadStream().CopyToAsync(memoryStream);
                        memoryStream.Position = 0;
                        postedFiles.Add(new RemoteStreamContent(memoryStream, file.FileName, file.ContentType, file.Length, disposeStream: false).As<TRemoteStreamContent>());
                        bindingContext.HttpContext.Response.OnCompleted(async () =>
                        {
                            await memoryStream.DisposeAsync();
                        });
                    }
                    else
                    {
                        postedFiles.Add(new RemoteStreamContent(file.OpenReadStream(), file.FileName, file.ContentType, file.Length, disposeStream: false).As<TRemoteStreamContent>());
                    }
                }
            }
        }
        else if (bindingContext.IsTopLevelObject)
        {
            postedFiles.Add(new RemoteStreamContent(request.Body, null, request.ContentType, request.ContentLength).As<TRemoteStreamContent>());
        }
    }

    private static ICollection<T> GetCompatibleCollection<T>(ModelBindingContext bindingContext)
    {
        var model = bindingContext.Model;
        var modelType = bindingContext.ModelType;

        // There's a limited set of collection types we can create here.
        //
        // For the simple cases: Choose List<T> if the destination type supports it (at least as an intermediary).
        //
        // For more complex cases: If the destination type is a class that implements ICollection<T>, then activate
        // an instance and return that.
        //
        // Otherwise just give up.
        if (typeof(T).IsAssignableFrom(modelType))
        {
            return new List<T>();
        }

        if (modelType == typeof(T[]))
        {
            return new List<T>();
        }

        // Does collection exist and can it be reused?
        if (model is ICollection<T> collection && !collection.IsReadOnly)
        {
            collection.Clear();

            return collection;
        }

        if (modelType.IsAssignableFrom(typeof(List<T>)))
        {
            return new List<T>();
        }

        return (ICollection<T>)Activator.CreateInstance(modelType)!;
    }
}