This post illustrates a cool technique that I learned at the MVP summit last week, allowing us to use Power Query to merge multiple files with properties from the file in the output.  A specific example of where this is useful is where you have several files with transactional data, saved with the month as the file name, but no date records in the file itself.  What we’d want to do is merge all the contents, but also inject the filename into the records as well.

The funny thing about this technique is that it’s eluded me for a long time, mainly because I way over thought the methods needed to pull it off.  Once you know how, it’s actually ridiculously simple, and gives us some huge flexibility to do other things.  Let’s take a look at how it works.

If you’d like to download the files I’m using for this example, you can get them here.  You'll find that there are 3 files in total: Jan 2008.csv, Feb 2008.csv and Mar 2008.csv.

Step 1: Create your query for one file

The first thing we need to do is connect to the Jan 2008.csv file and pull in it’s contents.  So let’s do that:

• Power Query –> From File –> From CSV
• Browse to the Jan 2008.csv file and import it
• Rename the “Sum of Amount” column to “Amount”

Perfect, we now have a basic query:

Notice here how the January file has no dates in it?  That’s not good, so that’s what we’re trying to fix.

Step 2: Turn the query into a function

At this point we need to do a little code modification.  Let’s go into the Advanced editor:

• View –> Advanced Editor

We need to do two things to the M code here:

• Insert a line at the top that reads:  (filepath) =>
• Replace the file path in the Source step (including the quotes) with:  filepath

At that point our M will read as follows:

We can now:

• Click Done
• Rename our Query to something like fnGetFileContents
• Save and close the query

Power Query doesn’t do a lot for us yet, just giving us this in the Workbook Queries pane:

Step 3: List all files in the folder

Now we’re ready to make something happen.  Let’s create a new query…

• Power Query –> From File –> From Folder
• Browse to the folder that holds the CSV files
• Remove all columns except the Name and Folder Path

Wait, what?  I removed the column with the binary content?  The column that holds the details of the files?  You bet I did!  You should now have a nice, concise list of files like this:

Next, we need to add a column to pull in our content, via our function.  So go to:

• Add Column –> Add Custom Column
• Enter the following formula:  =fnGetFileContents([Folder Path]&[Name])

Remember it is case sensitive, but when you get it right, some magic happens.  We get a bunch of “Table” objects in our new column… and those Table objects hold the contents of the files!

I know you’re eager to expand them, but let’s finish prepping the rest of the data first.

Step 4: Prep the rest of the data you want on the table rows

Ultimately, what I want to do here is convert the file name into the last day of the month.  In addition, I don’t need the Folder Path any more. So let’s take care of business here:

• Remove the “Folder Path” column
• Select the “Name” column –> Transform –> Replace Values –> “.csv” with nothing
• Select the “Name” column –> Transform –> Date Type –> Date
• Select the “Name” column –> Transform –> Date –> Month –> End of Month

And we’ve now got a pretty table with our dates all ready to go:

Step 5: Expand the table

The cool thing here is that, when we expand the table, each row of the table will inherit the appropriate value in the first column.  (So all rows of the table in row 1 will inherit 2/29/2008 as their date.)

• Click the little icon to the top right of the Custom column
• Click OK (leaving the default of expanding all columns)
• Rename each of the resulting columns to remove the Custom. prefix

And that’s it!  You can save it and close it, and it’s good to go.

A little bit of thanks

I want to throw a shout out to Miguel Llopis and Faisal Mohamood from the Power Query team for demonstrating this technique for the MVP’s at the summit.  I’ve been chasing this for months, and for some reason tried to make it way more complicated than it needs to be.

What’s the next step?

The next logical step is to extend this to working with Excel files, consolidating multiple Excel files together; something we can’t do through the UI right now.  Watch this space, as that technique is coming soon!