David Conrad

Design Leader and Community Activator

1/

Azure Data Factory

Design Direction: David Conrad
Designers: Jung Hung, Watanya Morrison

Azure Data Factory is a tool that lets users copy, move, and transform data. One example of this could be a company that wants to copy their sales transaction data from an internally-hosted SQL Server database to an Azure SQL database where they could use PowerBI to report on sales. As part of that process, they might want to remove personally identifying information from the data to avoid privacy concerns. ADF would let the user define the source (SQL Server Database), transform the data (remove the personally identifying info), and copy to a cloud database (Azure SQL database). This would be the end of the process fro ADF. The rest would be done in a tool like PowerBI.

Work

 

Challenges

  • Aure Data Factory must serve both coders and non-coders. This requires the voice of the product be both technically accurate and humane.
  • With a highly complex ecosystem of integrations, the product experience needs to normalize many disparate systems.
  • Given the exploratory nature of ADF, reducing the cost to get the product to market required a disciplined approach to prioritizing work.

History

ADF initially launched in 2015, primarily as an SDK with some visual monitoring for users. As time went by, a second audience became apparent – non-coders. These data engineers were not proficient in the coding languages used for data management and were handling their responsibilities with competitive products which allowed for visual authoring.

At the same time, it was clear there was a need for a solution to help SQL Server Integration Studio (SSIS) customers migrate their processes to the cloud.

As the redesign for the ADF version one product began to be discussed, we considered incorporating the visual authoring tools into another product – Azure Machine Learning Workbench (AMLW) – but reconsidered when it became clear that our target users didn’t have a need for machine learning tools. We did, however, opt to share the visual design system AMLW had created in the interest of trying to create as much unity across the products as possible.

Process

Our small design team met with product management daily to present concepts and respond to requests from engineering. These meetings, each lasting about thirty-minutes, were critical to keeping us focused and agile. We also met several times a week with the engineering team to discuss progress and blocking items.

Node and Connection diagram guidelines by Jung Hung
Using the JavaScript library GoJS to support our pipeline building experience, we set about designing the visual system based on the constraints of the library.

Version control management explorations by Watanya Morrison and David Conrad

Source control management was identified as an important feature as we iterated on design. This forced a number of rounds of factoring and refactoring UI to support repository management features.

Working with two researchers from the Azure Data team, we conducted a round of customer interviews on preview releases of the product which helped validate the conceptual model as well as identifying some gaps that needed to be addressed. We also maintained these customer relationships and extended the discussions into more of a longitudinal study which provides a different kind of feedback about the product.

I’ve worked many years with SAP and with other clouds and Microsoft is clearly on the right track. Nothing matches it in terms of interface and user friendliness
– Alexandre G.