Dataverse Migration, Power Apps vs Logic Apps Speed Test

-

Recently I’ve been looking into fine tuning Power Automate for performance, a customer needs a large regular data load into the Dataverse, the number of records keeps growing and my fine tuning keeps improving performance. A thank you to John Liu and some information from the 2019 power platform conference (check this youtube video out to get some great performance pointers).

My next evolution was to see what would happen if I used Logic Apps, that’ll surely make things quicker right?

My scenario is grabbing circa half a million records using GraphQL http requests and adding/updating them to Dataverse, doing some logic and lookups along the way.

I decided to make a crude example just to see what the performance was like, I created a source repository to mimic a large data set, added 10,000 items, with 20 fields, 19 of which were guids and 1 date field just to represent a reasonable chunk of data.

Both of the examples below I run the apply to each with concurrency set to max! (50)

Power Automate Results

The Power Automate to move this from one table to another you can see below:

Total run time: 9m 39s

Logic Apps Results

Next onto a representative Logic App, note I couldn’t get the Preview list rows to work, so I went legacy here, but the Add rows is the new preview version:

Total run time 11m 53s

I must confess, I ran this once, they are on different tenants (both test tenants so the 10k run was fresh on each). My main goal was to see if there was a scale difference in performance like wow it’s 10x faster on a logic app. For me I’m going to stick with Power Automate for now.

Hope this might be useful.

 

 

Side note – an interesting observation, when I do the data import from my GraphQL requests, I originally retrieved all the existing ID’s of the records in my target table using list rows, to then use the inbuilt “Filter” step to check if I’m creating a new item or updating an existing.

Don’t do this – the array manipulation is slower than just querying the dataverse for field=id and checking if there was an item returned! For over 500 test records it halved the processing time – it’ll probably be more on a larger dataset.

 

Comments

Post a Comment

Popular posts from this blog

Use Power Automate to create a Document Set – The easy way!

-
Image
NOTE: This makes a “pseudo” document set, use the VTI bin method to create a full blown document set, however perhaps some of the steps could be useful to you. I’ve recently been working with improving some workflows which use Document Sets and stumbled into a problem. Online when you search for “Document Sets using Power Automate” you get posts such as:  https://karinebosch.wordpress.com/2018/11/08/microsoft-flow-create-a-document-set/ https://knowhere365.space/microsoft-flow-sharepoint-document-set-creation/ Now these posts have helped a lot of people but there is one drawback to them and most of the others I can find online. They rely on the use of _vti_bin (sometimes referred to as SP 2010 REST API) If you don’t know much about the history of VTI go check this out: https://www.spjeff.com/2009/03/13/history-of-_vti_bin/ It’s pretty vintage now. The customer I’ve been working with has a handful of SharePoint 2010 workflows we discovered using the SharePoint Modernisation Scann...
Read more

Power Automate - Get SharePoint items using Created By/Modified By

-
Image
Recently I had an interesting request where someone was trying to filter items based on the Created By field in SharePoint using Power Automate. I thought, that should be pretty straight forward, make a claim or something similar and I headed to Google. Surprisingly there's not actually that many hits and it took quite a bit of digging. The answer once you realise is pretty straight forward. SharePoint holds a hidden(ish) list, the "User Information List" you can view it by adding the following to a Site Collections URL:  /_catalogs/users/simple.aspx This list holds all the "Ensured Users" on the site - you can google that one if you want to find out more. It also holds the information on which you can use in the filters for People Fields. Here are all the useful filters you can use: "ItemInternalId":"11", "ID":11, "Title":"Martin Pyman", "Name":"i:0#.f|membership|martin@marnotechnology.com", ...
Read more

Power Automate - Update SharePoint item metadata using only variables

-
Image
 When you use the inbuilt Power Automate SharePoint "Update item" or "Update file properties" and want to use a variable for that, you lose all the prepopulated columns: For general quick Power Automates - "Update item" is fine, but if you work with other developers or promote these across Power Platform Environments and hook them up to different lists/libraries, you may want to be able to quickly set that information at the top of the Automate. Not have to scroll through tens/hundreds of steps to change the target list URL everywhere. This is where "Send an HTTP Request to SharePoint" comes in, you just have to know how to use the SharePoint REST API, or follow my guide below. My Target library has two additional metadata columns which I'll show you how to update, "Document Type" and "Item Types", the latter being a choice field. For the example below I show what this might look if you are passing an Id of an item from a ...
Read more