Welcome to Azure Data Factory’s August monthly update! Here, we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all our updates in our What's New in ADF page.
We’ll also be hosting our monthly livestream THIS week on September 15th at 11am PST / 2pm EST! Join us to see some live demos and to ask us your ADF questions!
Join the livestream here.
Table of Contents
Continuous integration and continuous delivery (CI/CD)
Data flow
- Appfigures REST-based connector added as Source (Preview) [Doc]
- Cast transformation added – visually convert data types [Blog]
- New UI for inline datasets - categories added to easily find data sources [Blog]
Data movement
- Default activity time-out changed from 7 days to 12 hours [Blog]
- Service principal authentication type added for Blob storage [Doc]
User interface
- New data factory creation experience - 1 click to have your factory ready within seconds [Blog]
- Expression builder UI update – categorical tabs added for easier use [Blog]
Continuous integration and continuous delivery
Exclude pipeline triggers that did not change in deployments
We’ve improved the CI/CD process by only requiring the triggers that are changed or updated that will be stopped and started during deployments using the updated PowerShell. This will help reduce overall deployment time and the need to stop all triggers during deployment!
To learn more about this update, read CI/CD trigger deployment improvement in ADF (microsoft.com).
Data flow
Appfigures REST-based connector added (Preview)
We’ve added a new REST-based connector to mapping data flows! Users can now read their tables from Appfigures. Note: This connector is only available when using inline datasets.
For more information, read Transform data in AppFigures (Preview) - Azure Data Factory & Azure Synapse | Microsoft Docs.
Cast transformation added – visually convert data types
It is very common to transform data types into data integration workloads. For example, you might need to convert strings to dates. Previously to do this in mapping data flows, you had to use Expression Builder and remember the correct syntax or keywords to convert data types in mapping data flows.
For example, the syntax to convert a string ‘Jul-01-2022’ to date 2022-07-01 is toDate(‘Jul-01-2022’, ‘MMM-dd-yyyy’, ‘en-US’).
Now, you can use the cast transformation to quickly transform data types visually!
To learn more about this transformation in mapping dataflows, read Cast transformation in mapping data flow.
New UI for inline datasets - categories added to easily find data sources
We’ve updated our data flow source UI to make it easier to find your inline dataset type. Previously, you would have to scroll through the list or filter to find your inline dataset type. Now, we have categories that group your dataset types, making it easier to find what you’re looking for.
To learn more about this update, read Mapping data flows - Inline Datasets.
Data movement
Default activity time-out changed from 7 days to 12 hours
We heard your feedback, and it was clear that the default pipeline activity timeout of 7 days was too long! While you can still customize the timeout, the default activity time out for Azure Data Factory and Azure Synapse Analytics is now 12 hours.
To learn more about this change, read Azure Data Factory Changing Default Pipeline Activity.
Service principal authentication type added for Blob storage
Now, users can use the Service Principal authentication type for Azure Blob Storage! This allows you to use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user, group, or application service principal, who can then authenticate with Azure AD to authorize a request.
To learn more, read Copy and transform data in Azure Blob storage - Azure Data Factory & Azure Synapse | Microsoft Docs.
User interface
New data factory creation experience - 1 click to have your factory ready within seconds
A new user experience is ready in the landing page of adf.azure.com or the "Switch Data Factory" tab of Data Factory Studio portal to enable users to create a Data Factory with only one click! All you need is a factory name, location, and subscription and the Data Factory created will be available for use within seconds. The new creation experience will not only empower new users to quickly use and learn more about Data Factory, but also let experienced users easily create a new factory for testing.
To learn more about this new experience, read New Experience for Creating Data Factory within Seconds (microsoft.com).
Expression builder UI update - categorical tabs added for easier use
We’ve updated our expression builder UI to make pipeline designing easier. Our pipeline expression builder used to list out all components in a single pane.
Now, to make it easier to work with different pipeline components, we’ve created new content category tabs to make it easier to find what you’re looking for.
To learn more, read More Pipeline Expression Builder Ease-of-Use Enhancements!
We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in these blogs or livestreams. We love hearing your feedback!