Open Closed

Unable to call remote service and it is throwing error #10336

User avatar
0
Hon-Tre_IFS created

Hi,

We have 3 modular APIs (1 API,2 API,3 API) and we are trying to call (through remote service call) 2 modulars layered api (2 and 3 API) from single layer modular api (1 API). When all these modular APIs were developed in ABP 9, it was working fine. After upgrading to all the modular API from 9 to 10, we are getting 403 forbidden issue when calling the remote service.

The token is generating properly and infact the 2 modular layered api also returning the data but when we try to call from modular single layer api, it is throwing forbidden error. We looked at all the permissions and it looks fine.

Please help us to resolve the issue

Go to accepted answer
10 Answer(s)
  • User Avatar
    0
    AI-Bot created
    Support Team ABP Agent
    1. Understand what 403 means in this scenario

    A 403 from API‑2 / API‑3 means:

    • The HTTP call from API‑1 reaches API‑2/3.
    • The access token is present, but authorization fails (policy / permission / scope / client not allowed).

    So the problem is not “remote service call configuration”, but application‑to‑application authorization after the v9→v10 upgrade (where identity/auth stack has changed).

    1. Checklist on API‑2 / API‑3 (the services being called)

    On each modular layered API (2 & 3):

    1. Check the authorize attributes

      • If your application service or controller has:
    [Authorize("SomePolicy")]
     // or
     [Authorize(MyPermissions.Foo.Bar)]

     ensure that the **principal represented by the token coming from API‑1** is actually allowed for that policy / permission.

    1. If you use permissions, ensure the calling client has them

      When you use ABP’s permission system on APIs:

      • Each service must define its permissions in its Application.Contracts module.
      • Those permissions must be granted either:
        • to the user represented in the token, or
        • to the client (if you use client‑credentials flow and ClientPermissionValueProvider).

      In a microservice/zero‑trust setup this is typically done via a central identity/permission service; see the “Zero Trust Microservice Architecture with ABP Framework” article for a complete pattern:

    // Example central permission check service
   public interface IPermissionCheckerAppService : IApplicationService
   {
       Task<bool> CheckPermissionAsync(CheckPermissionInput input);
       Task<MultiplePermissionGrantResultDto> CheckPermissionsAsync(CheckPermissionsInput input);
   }

    1. Check that the token subject is what you expect

      • If API‑1 calls 2/3 using client credentials, the token will represent a client, not a user.
      • If your permissions / policies assume a user (e.g. they check ICurrentUser.IsAuthenticated or user‑based permissions), they will deny requests coming from a pure client token.
      • In that case either:
        • Switch your authorization to client‑based permissions, or
        • Call APIs using a user token (on behalf of the end user) instead of client credentials.
    1. Checklist on API‑1 (the caller)

    On the single‑layer API‑1, which calls 2 & 3 via remote service / HTTP client:

    1. Confirm how the token is added to outgoing calls

      You should have something equivalent to an IRemoteServiceHttpClientAuthenticator that sets the Authorization: Bearer <token> header for all dynamic client proxy calls, e.g.:

    [Dependency(ReplaceServices = true)]
   [ExposeServices(typeof(IRemoteServiceHttpClientAuthenticator))]
   public class SampleServiceHttpClientAuthenticator :
       IRemoteServiceHttpClientAuthenticator, ISingletonDependency
   {
       private readonly IRemotingTokenStore _tokenStore;

       public async Task Authenticate(RemoteServiceHttpClientAuthenticateContext context)
       {
           context.Request.Headers.Authorization = null;
           var token = await _tokenStore.GetTokenAsync(CancellationToken.None);
           if (!string.IsNullOrEmpty(token))
           {
               context.Request.Headers.Add("Authorization", $"Bearer {token}");
           }
       }
   }

    If after the upgrade this authenticator is not registered or not executed, API‑2/3 will receive a request without a valid token and return 401/403.

    1. Check client credentials used by API‑1

      If API‑1 uses client credentials to obtain a token (as in the “Zero Trust Microservice Architecture” sample):

      • Verify in your configuration (e.g. appsettings.json) that:
        • AuthServer:Authority points to the correct identity server.
        • Remoting:Client:ClientName, ClientSecret, and Scope (or equivalent) match a client that exists and is allowed to call API‑2/3.
      • Ensure that this client is granted all required scopes / API resources / permissions for endpoints in API‑2 and 3.

    2. Ensure RemoteServices base URLs are correct

      In API‑1, for each remote service:

    "RemoteServices": {
     "Service2": {
       "BaseUrl": "https://<api2-host>"
     },
     "Service3": {
       "BaseUrl": "https://<api3-host>"
     }
   }

    and in your module:

    context.Services.AddHttpClientProxies(
       typeof(Service2ApplicationContractsModule).Assembly,
       remoteServiceConfigurationName: "Service2"
   );

   context.Services.AddHttpClientProxies(
       typeof(Service3ApplicationContractsModule).Assembly,
       remoteServiceConfigurationName: "Service3"
   );

    Misconfigured remote service name or base URL won’t usually produce 403, but verify them while you’re here.

    1. Typical concrete fix path

    Based on similar ABP microservice setups (including the referenced zero‑trust sample):

    1. Create/verify a client in your identity server representing API‑1, with:

      • Grant type: client credentials.
      • Allowed scopes/resources: APIs exposed by services 2 and 3.

    2. Grant required permissions to that client (if you use centralized permission management and client permission provider).

    3. Configure API‑1 to use those client credentials when requesting tokens and to attach the token on all remote calls (via an IRemoteServiceHttpClientAuthenticator).

    4. Align [Authorize] usage on API‑2 / API‑3 so that:

      • They accept a client token where appropriate, or
      • They check permissions in a way compatible with your token (client vs user).

    5. Test directly with the same token

      • From API‑1 logs or debugger, copy the access token it is using.
      • Call an endpoint on API‑2 / API‑3 via Postman using that exact token.
      • If you still get 403, your issue is purely authorization (permissions / scopes) on API‑2/3.
    1. If you want a hardened pattern

    The Zero Trust Microservice Architecture with ABP Framework article shows a complete, working pattern of:

    • Centralized identity/permission microservice.
    • Each service having its own client.
    • IRemotingTokenStore + custom IRemoteServiceHttpClientAuthenticator for inter‑service calls.
    • Remote permission checks and auditing.

    You can align your solution with that pattern to make your inter‑service calls and permissions robust.

    1. Summary

    After upgrading to ABP 10, you must:

    • Ensure API‑1 obtains a valid token as a client and attaches it to all remote calls.
    • Ensure API‑2 & API‑3 accept that token and grant it the necessary permissions / scopes.
    • Check [Authorize] attributes and permission assignments accordingly.

    Once client credentials, scopes/permissions, and the HTTP client authenticator are correctly configured, the 403s from your modular APIs will stop.

    This response was generated by AI. If it didn’t solve your problem, feel free to post a new answer — our support team will jump in to help you soon.

  • User Avatar
    0
    Hon-Tre_IFS created

    it does not solve the problem

  • User Avatar
    0
    AI-Bot created
    Support Team ABP Agent
    1. From your description, what we know
    • Each API (1, 2, 3) can be called directly with a token → APIs 2 & 3 accept and process a token correctly.
    • When API‑1 calls API‑2/3 via ABP remote service, you now get 403 Forbidden after upgrading everything from v9 → v10.
    • Same architecture worked on v9.

    That almost always means:

    • The token that API‑1 sends to API‑2/3 is no longer authorized after the v10 upgrade, even though user tokens work.

    v10 moved everything fully to OpenIddict and tightened validation. You must realign:

    • The client representing API‑1 in the AuthServer.
    • The scopes/audiences that APIs 2 & 3 accept.
    • The token acquisition & attachment code used by API‑1.

    Below is a concrete, minimal set of checks/fixes that resolves this in practice.

    1. Verify the inter‑service token that API‑1 uses

    In API‑1:

    1. Log the access token just before ABP sends it:
    public class LoggingHttpClientAuthenticator : IRemoteServiceHttpClientAuthenticator, ISingletonDependency
{
    private readonly IRemotingTokenStore _tokenStore;
    private readonly ILogger<LoggingHttpClientAuthenticator> _logger;

    public LoggingHttpClientAuthenticator(
        IRemotingTokenStore tokenStore,
        ILogger<LoggingHttpClientAuthenticator> logger)
    {
        _tokenStore = tokenStore;
        _logger = logger;
    }

    public async Task Authenticate(RemoteServiceHttpClientAuthenticateContext context)
    {
        context.Request.Headers.Authorization = null;
        var token = await _tokenStore.GetTokenAsync(CancellationToken.None);

        _logger.LogInformation("Outgoing inter-service access token: {Token}", token);

        if (!string.IsNullOrEmpty(token))
        {
            context.Request.Headers.Add("Authorization", $"Bearer {token}");
        }
    }
}
    1. Take that token string and call an endpoint on API‑2 directly with Postman:
    • Authorization: Bearer <copied-token>

    If you still get 403, then the problem is not ABP remote services, but access token configuration / scopes / audiences between AuthServer, API‑1 client, and APIs 2 & 3.

    1. Fix OpenIddict client/scopes for API‑1 in AuthServer

    In your AuthServer (Identity service), open your OpenIddictDataSeedContributor (or equivalent) and check the application (client) that represents API‑1.

    It must have:

    • client_type = confidential
    • Grant type: client_credentials
    • Scopes including the API scopes of services 2 & 3.

    Roughly like the official template:

    // scopes
await CreateScopesAsync(context, new[]
{
    "Api2",
    "Api3",
    OpenIddictConstants.Scopes.Email,
    OpenIddictConstants.Scopes.Profile,
    OpenIddictConstants.Scopes.Roles
});

// client for API-1
await CreateApplicationAsync(
    name: "Api1_Internal_Client",
    type: OpenIddictConstants.ClientTypes.Confidential,
    consentType: OpenIddictConstants.ConsentTypes.Systematic,
    displayName: "Api1 internal client",
    secret: "VERY-SECRET",
    grantTypes: new[]
    {
        OpenIddictConstants.GrantTypes.ClientCredentials
    },
    scopes: new[]
    {
        "Api2",
        "Api3"
    });

    Make sure the scope names you put here match what APIs 2 & 3 are configured to accept (next section).

    If the client was migrated from IdentityServer4 config, the allowed scopes/resources often need to be re‑created in this OpenIddict seeding.

    1. Align API‑2 and API‑3 resource configuration (audience & scope)

    In each resource API (2 and 3) you normally have something like this in the module’s ConfigureServices:

    public override void ConfigureServices(ServiceConfigurationContext context)
{
    var configuration = context.Services.GetConfiguration();

    context.Services.AddAuthentication("Bearer")
        .AddJwtBearer("Bearer", options =>
        {
            options.Authority = configuration["AuthServer:Authority"]; // e.g. https://auth.mycompany.com
            options.RequireHttpsMetadata = true;
            options.Audience = "Api2"; // or "Api3" for the 3rd API
        });

    // ...
}

    Or, if you use OpenIddict validation:

    PreConfigure<OpenIddictValidationBuilder>(builder =>
{
    builder.AddAudiences("Api2"); // or "Api3"
    builder.UseLocalServer();
    builder.UseAspNetCore();
});

    Now match that:

    • Audience / AddAudiences("Api2") must equal the scope name you granted to API‑1’s client (previous step).
    • If API‑1 is requesting a token with scope Api2 Api3 but API‑2 is configured with Audience = "MyApi2" or something else, the token will be considered not for this API, and authorization will fail.

    So:

    • Decide a scope name for each API, e.g. "Api2" and "Api3".
    • Use those consistently:
      • In OpenIddictDataSeedContributor (scope + client’s allowed scopes).
      • In the resource APIs’ JWT/OpenIddict validation config (Audience or AddAudiences).
    1. Ensure API‑1 actually requests the correct scopes

    Your IRemotingTokenStore (or equivalent) in API‑1 must request the same scopes:

    public class RemotingClientOptions
{
    public string ClientName { get; set; }
    public string ClientSecret { get; set; }
    public string Scope { get; set; } // e.g. "Api2 Api3"
}

    appsettings.json in API‑1:

    "AuthServer": {
  "Authority": "https://<your-auth-server>"
},
"Remoting": {
  "Client": {
    "ClientName": "Api1_Internal_Client",
    "ClientSecret": "VERY-SECRET",
    "Scope": "Api2 Api3"
  }
}

    Token store (simplified):

    var tokenResponse = await client.RequestClientCredentialsTokenAsync(
    new ClientCredentialsTokenRequest
    {
        Address = discovery.TokenEndpoint,
        ClientId = clientId,
        ClientSecret = clientSecret,
        Scope = scope // "Api2 Api3"
    },
    cancellationToken);

    If in v9 you used older IdentityServer4 client‑credentials with different scope names, after migrating to OpenIddict those old names will not work. You must request scopes that actually exist in OpenIddict.

    1. Re‑confirm the remote service wiring in API‑1 (non‑auth piece)

    Once the token and scopes are fixed, your existing ABP remote service setup is usually fine:

    // appsettings.json of API-1
"RemoteServices": {
  "Api2": {
    "BaseUrl": "https://api2.yourdomain"
  },
  "Api3": {
    "BaseUrl": "https://api3.yourdomain"
  }
}

    // in API-1 module
context.Services.AddHttpClientProxies(
    typeof(Api2ApplicationContractsModule).Assembly,
    remoteServiceConfigurationName: "Api2"
);

context.Services.AddHttpClientProxies(
    typeof(Api3ApplicationContractsModule).Assembly,
    remoteServiceConfigurationName: "Api3"
);

    Together with an IRemoteServiceHttpClientAuthenticator that attaches the bearer token, those parts typically don’t change between v9 and v10.

    1. Summary of what to actually change

    2. In AuthServer

      • Define / verify scopes: one per API (e.g. Api2, Api3).
      • Define API‑1’s confidential client with grant type client_credentials and allowed scopes Api2, Api3.

    3. In APIs 2 & 3

      • Configure JWT/OpenIddict validation audience to Api2 / Api3 (same as scopes above).

    4. In API‑1

      • Configure the client credentials request to ask for Scope = "Api2 Api3".
      • Keep using IRemoteServiceHttpClientAuthenticator to attach that token.

    5. Debug once

      • Copy the exact token from API‑1 logs and call API‑2 with it in Postman.
      • When that returns 200, ABP dynamic remote service calls will also return 200 (403 disappears).
    1. Sources
    • Microservice / inter‑service communication & auth pattern (includes IRemotingTokenStore, IRemoteServiceHttpClientAuthenticator, scopes, and audiences):
      https://abp.io/community/articles/xpiz9nvh
    • Microservice solution basics (how remote services are wired):
      https://abp.io/docs/latest/get-started/microservice
    • Dynamic C# client proxies (how ABP maps your remote calls):
      https://abp.io/docs/latest/API/Dynamic-CSharp-API-Clients

    This response was generated by AI. If it didn’t solve your problem, feel free to post a new answer — our support team will jump in to help you soon.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you share all app logs when 403 error occurs?

    see: https://abp.io/support/questions/8622/How-to-enable-Debug-logs-for-troubleshoot-problems

    liming.ma@volosoft.com

    Thanks

  • User Avatar
    0
    Hon-Tre_IFS created

    I have shared the logs to above email please refer

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you also share all applications' logs.txt?

    The HAR file shows 403 with a custom response.

    {
    "error": {
        "code": null,
        "message": "Failed to retrieve editions.",
        "details": null,
        "data": {},
        "validationErrors": null
    }
}

    Thanks.

  • User Avatar
    0
    Hon-Tre_IFS created

    i have shared the Logs.txt. Please refer

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Didn't receive it. Do you share all the Logts.txt files for all apps?

    You can share it by https://wetransfer.com/

    liming.ma@volosoft.com

    Thanks.

  • User Avatar
    0
    Hon-Tre_IFS created

    I have shared again. Please refer

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    The AuthServer returns the The specified client credentials are invalid. error.

    Can you check the credentials of the reqeust?

    Thanks.

    [INF] Received HTTP response headers after 1050.7501ms - 401
[INF] End processing HTTP request after 1050.9061ms - 401
[ERR] Unexpected error while fetching editions from SiteHost.
System.Exception: Failed to get token from DepartmentHost. Status: Unauthorized, Error: {
  "error": "invalid_client",
  "error_description": "The specified client credentials are invalid.",
  "error_uri": "https://documentation.openiddict.com/errors/ID2055"
}
   at Hon.IFS.PlatformHost.Services.Integrations.TenantHostService.GetJwtTokenAsync(TenantType tenantType) in /home/runner/work/Hon.IFS.PlatformHost/Hon.IFS.PlatformHost/Hon.IFS.PlatformHost/Services/Integrations/TenantHostService.cs:line 117
   at Hon.IFS.PlatformHost.Services.Integrations.TenantHostService.GetAllHostEditionsAsync(TenantType tenantType) in /home/runner/work/Hon.IFS.PlatformHost/Hon.IFS.PlatformHost/Hon.IFS.PlatformHost/Services/Integrations/TenantHostService.cs:line 525
2026-01-23 20:42:50.708 +00:00 [ERR] Error retrieving editions.
Volo.Abp.UserFriendlyException: Unexpected error: Failed to get token from DepartmentHost. Status: Unauthorized, Error: {
  "error": "invalid_client",
  "error_description": "The specified client credentials are invalid.",
  "error_uri": "https://documentation.openiddict.com/errors/ID2055"
}
Assignee
User avatar maliming
Question details
Labels
None yet
Boost Your Development
ABP Live Training
Packages
See Trainings
Mastering ABP Framework Book
The Official Guide
Mastering
ABP Framework
Learn More
Mastering ABP Framework Book
Made with ❤️ on ABP v10.2.0-preview. Updated on January 22, 2026, 11:30
1
ABP Assistant
🔐 You need to be logged in to use the chatbot. Please log in first.