Previously, the lack of this capability has resulted in large organisations and enterprises needing to centrally govern Azure AI model usage via an AI gateway for compliance, governance and monitoring purposes having limited capabilities to do so.
This also prevented organisations using AI gateways from accessing the benefits of rapid prototyping and producing proof-of-concept solutions at pace using Azure Foundry no-code or low-code functionality.

The Pattern

The usage pattern has the three following components (from right to left):

• Azure Foundry - Core Instance and Project
  This is the centrally governed Azure Foundry item. It is where models are setup and managed and governance configuration is setup.

• Azure API Management
  This is an Azure API management instance configured to function as an AI Gateway, managing and monitoring AI inference access to the core Azure Foundry instance.

• Azure Foundry - Client Instance and Project
  This is the client Azure Foundry instance that could be used for a variety of purposes within the organisation such as solution development.
  It does not have any models deployed directly within it.

  The Client Azure Foundry instance & project has a model gateway connection configured within it which redirects internal model calls to the Azure API Management service. The Azure API management service in turn redirects calls back and forth with the Azure Foundry core instance.

Example Implementation

Details for an example implementation using Model Gateway Connection functionality with a self-managed APIM instance can be found here:

bdazuretech/foundry-apim-mg-example: An example foundry API Management AI Gateway implementation using model gateway configuration

Further Information

The following resources provide further details for Azure Foundry Model Gateway Connection setup and usage.

• Bring Your Own Model to Foundry Agent Service - Microsoft Foundry | Microsoft Learn

• foundry-samples/infrastructure/infrastructure-setup-bicep/01-connections/apim-and-modelgateway-integration-guide.md at main · microsoft-foundry/foundry-samples

• foundry-samples/infrastructure/infrastructure-setup-bicep/01-connections/model-gateway/README.md at main · microsoft-foundry/foundry-samples

Many enterprises are looking for patterns to govern and manage access to their Azure Open AI resources, one such pattern is to place Azure API Management in front of Azure Open AI providing a governance gateway.
Solutions requiring access to Azure Open AI for embedding and completion operations must therefore communicate via Azure API Management, including Azure AI Search.

This blog covers a pattern that allows Azure AI Search Integrated Vectorisation to undertake the generation of document embedding data using Azure Open AI where Azure API Management is present as middleware.

The Pattern

The goal of this pattern is to enable the Azure Search Indexer to generate embeddings using a skillset definition that points to Azure API Management.
The Azure API Management method in-turn relays the embeddings requests back and forth to Azure Open AI. The indexer then places the embedding vector data into the AI Search Index.
The steps within the process are listed in more detail below:

1. An Azure AI Search Indexer run is initiated, reading a document from the configured Data Source.
   The Skillset attached to the indexer has a ‘CustomWebAPISkill’ defined to generate embeddings, with a URI and credentials pointing to the API Management ‘Embeddings for Azure Search‘ endpoint.
   JSON data is sent to the API Management method by the Indexer.

2. The API Management endpoint ‘Embeddings for Azure Search‘ receives the strings to have embeddings generated for, records the ‘recordId’ values, transforms the JSON body to that expected by Azure Open AI and forwards on the request to Azure Open AI.

3. Azure Open AI receives the embedding request from API Management, generates the embedding data and sends the response back to the API Management resource.

4. The API Management resource receives and transforms the embedding data from Azure Open AI, placing the ‘recordId’s’ from the original input back onto the collection elements, ensuring the JSON schema is of a format expected by the Azure Search indexer.
   The response is then forwarded back to the indexer which stores the embedding data, along with the document metadata into the AI Search Index.

Challenges and Solutions

1. Azure AI Search indexers have no built-in support for generating embedded data where API Management is governing access to Azure Open AI.

2. An Azure AI Search Skillset can be pointed to an API Management method, however a ‘CustomWebAPISkill’ is required.

3. An Azure AI Search Skillset using a ‘CustomWebAPISkill’ for enrichment has a fixed input and output JSON schema that significantly differs from that used by Azure Open AI embeddings. Manipulation of the body data within API Management is required inbound and outbound.

4. The Azure AI Search Indexer passes ‘recordId’ values for each item within the JSON body which should be relayed back onto the same data blocks after Azure Open AI has generated the embedding values. This ensures existing enriched data can be updated. API Management processing must handle this ID mapping.

5. Azure API Management requires unique method names for it’s inputs and replays the method name to the output. Adding a new method e.g. ‘embeddingsForSearch’ would result in the request being sent to Azure Open AI at the same address which Azure Open AI does not support.
   The API Management resource must therefore be configured to ‘Rewrite’ the URL to simply be ‘embeddings‘ for the backend Azure Open AI call.

Implementation

Note: The examples shown below are provided ‘as-is’ as a proof-of-concept and should not be considered for production use.

Assumptions

1. Azure AI Search, Azure APIM and Azure Open AI Services have been deployed.

2. Azure APIM has been configured with the built-in Azure Open AI configuration.

3. Azure Open AI has the ‘text-embedding-ada-002‘ model deployed.

Azure AI Search Configuration

Skillset

{
    "name": "test-skillset",
    "description": "Skillset to chunk documents and generate embeddings",
    "skills": [
    {
        "@odata.type": "#Microsoft.Skills.Text.SplitSkill",
        "name": "#1",
        "description": "Split skill to chunk documents",
        "context": "/document",
        "inputs": [
         {
            "name": "text",
            "source": "/document/content",
            "inputs": []
         }
        ],
        "outputs": [
         {
            "name": "textItems",
            "targetName": "pages"
         }
        ],
        "defaultLanguageCode": "en",
        "textSplitMode": "pages",
        "maximumPageLength": 200,
        "pageOverlapLength": 5,
        "unit": "characters"
    },
    {
        "@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
        "name": "#2",
        "description": "APIM Embedding",
        "context": "/document/pages/*",
        "inputs": [
         {
            "name": "text",
            "source": "/document/pages/*",
            "inputs": []
         },
        "outputs": [
          {
            "name": "vector",
            "targetName": "vector"
           }
        ],
        "uri": "https://{APIMSERVICENAME}.azure-api.net/openai/deployments/text-embedding-ada-002/embeddings-ai-search?api-version=2023-05-15",
        "httpHeaders": {
            "{APIMHEADERKEYNAME}": "{APIMHEADERKEYVALUE}"
        },
        "httpMethod": "POST",
        "timeout": "PT30S",
        "batchSize": 5
     }
    ],
    "@odata.etag": "\"0x8DCFCEB6F53B10F\"",
    "indexProjections": {
     "selectors": [
      {
        "targetIndexName": "index1",
        "parentKeyFieldName": "parent_id",
        "sourceContext": "/document/pages/*",
        "mappings": [
         {
            "name": "vector",
            "source": "/document/pages/*/vector"
         },
         {
            "name": "chunk",
            "source": "/document/pages/*"
         },
         {
            "name": "metadata_storage_path",
            "source": "/document/metadata_storage_path"
         },
         {
            "name": "title",
            "source": "/document/title"
         }
        ]
        }
    ],
    "parameters": {
        "projectionMode": "skipIndexingParentDocuments"
    }
  }
}

Parameters

• {APIMSERVICENAME} - The name of the APIM service that forms part of the URL to access it.

• {APIMHEADERKEYNAME} - The APIM header key name used for authentication.

• {APIMHEADERKEYVALUE} - The APIM header key value used for authentication.

Index

Indexer

{
    "@odata.context":"https://{CONTEXTNAME}/\(metadata#indexers/\)entity",
    "@odata.etag":"\"0x8DCFCEBAA8B8DFC\"",
    "name":"indexer1",
    "description":null,
    "dataSourceName":"{DATASOURCENAME}",
    "skillsetName":"{SKILLSETNAME}",
    "targetIndexName":"{INDEXNAME}",
    "disabled":null
    "schedule":null,
    "parameters":{
     "batchSize":1,
     "maxFailedItems":null,
     "maxFailedItemsPerBatch":null,
     "base64EncodeKeys":null,
     "configuration":{
        "dataToExtract":"contentAndMetadata",
        "parsingMode":"default"
     }
    },
    "fieldMappings":[
     {
        "sourceFieldName":"metadata_storage_name",
        "targetFieldName":"title",
        "mappingFunction":null
     }
    ],
    "outputFieldMappings":[
     {
        "sourceFieldName":"/docemnt/pages/*/vector",
        "targetFieldName":"vector",
        "mappingFunction":null
     }
    ],
    "cache":null,
    "encryptionKey":null
}

References

Custom Web API skill in an Azure AI Search enrichment pipeline

Azure Open AI REST API - Embeddings