Dataverse Migration, Power Apps vs Logic Apps Speed Test

-

Recently I’ve been looking into fine tuning Power Automate for performance, a customer needs a large regular data load into the Dataverse, the number of records keeps growing and my fine tuning keeps improving performance. A thank you to John Liu and some information from the 2019 power platform conference (check this youtube video out to get some great performance pointers).

My next evolution was to see what would happen if I used Logic Apps, that’ll surely make things quicker right?

My scenario is grabbing circa half a million records using GraphQL http requests and adding/updating them to Dataverse, doing some logic and lookups along the way.

I decided to make a crude example just to see what the performance was like, I created a source repository to mimic a large data set, added 10,000 items, with 20 fields, 19 of which were guids and 1 date field just to represent a reasonable chunk of data.

Both of the examples below I run the apply to each with concurrency set to max! (50)

Power Automate Results

The Power Automate to move this from one table to another you can see below:

Total run time: 9m 39s

Logic Apps Results

Next onto a representative Logic App, note I couldn’t get the Preview list rows to work, so I went legacy here, but the Add rows is the new preview version:

Total run time 11m 53s

I must confess, I ran this once, they are on different tenants (both test tenants so the 10k run was fresh on each). My main goal was to see if there was a scale difference in performance like wow it’s 10x faster on a logic app. For me I’m going to stick with Power Automate for now.

Hope this might be useful.

 

 

Side note – an interesting observation, when I do the data import from my GraphQL requests, I originally retrieved all the existing ID’s of the records in my target table using list rows, to then use the inbuilt “Filter” step to check if I’m creating a new item or updating an existing.

Don’t do this – the array manipulation is slower than just querying the dataverse for field=id and checking if there was an item returned! For over 500 test records it halved the processing time – it’ll probably be more on a larger dataset.

 

Comments

Post a Comment

Popular posts from this blog

Use Power Automate to create a Document Set – The easy way!

-
Image
NOTE: This makes a “pseudo” document set, use the VTI bin method to create a full blown document set, however perhaps some of the steps could be useful to you. I’ve recently been working with improving some workflows which use Document Sets and stumbled into a problem. Online when you search for “Document Sets using Power Automate” you get posts such as:  https://karinebosch.wordpress.com/2018/11/08/microsoft-flow-create-a-document-set/ https://knowhere365.space/microsoft-flow-sharepoint-document-set-creation/ Now these posts have helped a lot of people but there is one drawback to them and most of the others I can find online. They rely on the use of _vti_bin (sometimes referred to as SP 2010 REST API) If you don’t know much about the history of VTI go check this out: https://www.spjeff.com/2009/03/13/history-of-_vti_bin/ It’s pretty vintage now. The customer I’ve been working with has a handful of SharePoint 2010 workflows we discovered using the SharePoint Modernisation Scanner  
Read more

Power Automate - Get SharePoint items using Created By/Modified By

-
Image
Recently I had an interesting request where someone was trying to filter items based on the Created By field in SharePoint using Power Automate. I thought, that should be pretty straight forward, make a claim or something similar and I headed to Google. Surprisingly there's not actually that many hits and it took quite a bit of digging. The answer once you realise is pretty straight forward. SharePoint holds a hidden(ish) list, the "User Information List" you can view it by adding the following to a Site Collections URL:  /_catalogs/users/simple.aspx This list holds all the "Ensured Users" on the site - you can google that one if you want to find out more. It also holds the information on which you can use in the filters for People Fields. Here are all the useful filters you can use: "ItemInternalId":"11", "ID":11, "Title":"Martin Pyman", "Name":"i:0#.f|membership|martin@marnotechnology.com",
Read more

Handling British Summer time when querying SharePoint from Power Automate

-
Image
This also applies to other TimeZones. When you query SharePoint in Power Automate using steps like "Get Items" the raw information that is returned when you are working with dates and currently inside British Summer Time will be out by an hour. Below I've created a step which runs Get Items and I add a compose to check the Created date: Going into SharePoint you can see the date time returned is an hour out (I'm currently inside British Summer Time and my laptops date/time is set as such). Whenever you are handling dates from SharePoint in Power Automate you have to remember that the original value stored in SharePoint for date/time is the UTC value. Depending on your regional settings the value is converted for you when it's rendered. Fear not! Power Automate has conversions, here's one way: In the example above I'm taking the source Created date/time which is in UTC and converting it to my location - Dublin, Edinburgh, Lisbon, London. The expected Create
Read more