Calling Durable Functions From Fabric Pipelines

Azure Durable Functions are convenient when you need to do long-running custom operations. Now, like Synapse and Azure Data Factory, we can call Durable Functions from Fabric pipelines. Let’s dive into how.
Calling Durable Functions From Fabric Pipelines
Calling Durable Functions From Fabric Pipelines

Fabric pipelines are growing daily and include more and more Synapse Analytics and Azure Data Factory features. One feature I have repeatedly used lately is the Azure Function activity to call Durable Functions for long-running operations that I can’t simply do with Synapse or Fabric tooling.

There could be many reasons for using Durable Functions; mine is that the Data Factory becomes slow and expensive when it calls APIs from within Pipelines or Dataflows or overloads the API with calls. I create batch files from the Pipeline, then utilise Durable Functions to process them by first queueing them in storage queues and executing in my own rhythm. You can also use Service Bus or other queueing components; I like the Durable Function experience.

The Azure Function activity in Fabric can call HTTP-triggered Functions, but Durable ones run asynchronously. That means they run in the background, and you need to query the status to see if they’ve finished. Fortunately, Durable Functions return async-HTTP-pattern compatible URLs, so it’s a breeze to do so.

For this example, we will use the code sample from our own GitHub repo, which you can find below. You can deploy that to any Function app of your choice, as long as it’s accessible from the Internet.

GitHub - ThatBlueCloud/Samples.DurableFunction
Contribute to ThatBlueCloud/Samples.DurableFunction development by creating an account on GitHub.

Let’s dive in.

Create a Pipeline in Fabric

Go to your favourite Fabric Workspace and add a new Pipeline. Name it anything you like.

Add Azure Function Activity and Configure

Click on Azure Function activity under the Activities top ribbon. Name it as you wish, then go to the Settings page. Click on Add Connection. You can copy the Function App’s URL from its Azure Portal blade and the Function Key under your Function on the same page.

Creating a Function Connection

Azure Function activity currently doesn’t support Azure AD authentication or Service Principals, so we must use the Account Key authentication.

After creating the connection, configure the rest of the Settings as below. Function Relative URL is the route of your Function, which you can find under Azure Portal. In my example, it’s sayhello, but make sure you don’t include the api part in this box.

Function activity settings

Now, save your changes and give it a run. If you configured it correctly, you should see a green light on the Output page with the following output:

Durable Function output

As mentioned, it’s running asynchronously. Now we have to get the URL to listen from our Function activity, and wait for it to complete.

Subscribe to TBC Weekly Newsletter!

Receive articles, tips and tricks from us and the Fabric community

Subscribe

Listen To What The Function Says

We’ll use Web activity next to call the statusQueryGetUri URL that Function gave us. While this was a breeze in Synapse and ADF, it’s a bit tricky in Fabric as it requires a Connection (a.k.a. Linked Service) to be created for the Web activity.

Web activity connection

This requirement for Connection screws up the relative path that it needs, as the status URL comes back as a full URL, not relative. We'll circumvent this by replacing the Function App’s URL with an empty string in the Web activity:

Removing Function App url from status query URL

Then, configure the Method as GET on the Settings page. That’s it.

Save your changes and give it another go. Your Function should have executed nicely, and your Pipeline will drink a cold beer whilst waiting for its order:

Successful output

If you click on the output of the Web activity, you should be seeing the Function App saying Hello That Blue Cloud!:

Conclusion

Function Apps are a big part of the Azure ecosystem and can help enrich your Pipeline experience in many scenarios. They may not be for business teams or citizen developers, but they certainly are easy to develop by IT teams. When Microsoft brings the Azure AD authentication capability to the Function activity, it’ll be even more accessible.

How are you using Azure Functions with your Fabric, Synapse or ADF instances? Let us know in the comments!


Remember to subscribe to our TBC Weekly newsletter! You'll get a summary of the latest articles from us and the Fabric and Azure communities.

Harun Legoz

Harun Legoz

I’m a cloud solutions architect with a coffee obsession. Have been building apps and data platforms for over 18 years, I also blog on Azure & Microsoft Fabric. Feel free to say hi on Twitter/X!

That Blue Cloud

Design awesome data platforms using Azure & Microsoft Fabric

That Blue Cloud

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to That Blue Cloud.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.