Importing data from Folder through Power Query

earthworm

earthworm

Well-known Member
Joined
May 19, 2009
Messages
766
Office Version
  1. 2019
  2. 2016
Platform
  1. Windows
I have 30 files . Each files contains data of approx 6 Million. I imported the folder through Power query. Then I get and transform data and then I used the formula Excel.Workbook(Content). When I am expanding the column power query is taking too much time in loading file before I move to next step of expanding the column to show data. Is there any faster way ? Because I have to wait for hours before moving to next step. Please advice
 

Excel Facts

Bring active cell back into view
Start at A1 and select to A9999 while writing a formula, you can't see A1 anymore. Press Ctrl+Backspace to bring active cell into view.
Sort by date Sort by votes
What file format are the source files?
Are the files/folder on a network drive? If so, move them to your local drive and it should be a lot better.
 
Upvote 0
You say each file contains "approx 6 million". 6 million what? It can't be rows. When you say "I used the formula Excel.Workbook(Content)" did you do that manually, or did you use the UI to combine the files automatically?
 
Upvote 0
Matt Allington said:
You say each file contains "approx 6 million". 6 million what? It can't be rows. When you say "I used the formula Excel.Workbook(Content)" did you do that manually, or did you use the UI to combine the files automatically?
Click to expand...
6 Million Rows. I used the formula manually
 
Upvote 0
You are not telling me something. You say your source files are Excel files and you say they contain more than 6 million rows. An excel sheet has a maximum of 1 million rows, so something else is going on. Are the 6 million rows spread across more than 6 sheets in the same book?
 
Upvote 0
Matt Allington said:
You are not telling me something. You say your source files are Excel files and you say they contain more than 6 million rows. An excel sheet has a maximum of 1 million rows, so something else is going on. Are the 6 million rows spread across more than 6 sheets in the sa
Click to expand...

Matt Allington said:
You are not telling me something. You say your source files are Excel files and you say they contain more than 6 million rows. An excel sheet has a maximum of 1 million rows, so something else is going on. Are the 6 million rows spread across more than 6 sheets in the same book?
Click to expand...

I need a merge a data from each excel file that contains 800,000 on each excel file
 
Upvote 0
Here are some thoughts:
  1. are you merging or appending? Appending is generally faster than merging but you may not have a choice
  2. I suggest you use the UI, not manually written queries. Usually the UI knows best
  3. You say it is "too slow" but you don't specify how long it is taking. I would expect such a query to take about 3-5 mins assuming pure append.
  4. What is the file extension? if it is XLSX it will probably be faster than XLSB
 
Upvote 0
You must log in or register to reply here.

Similar threads

D
  • Solved
PQ slow when refreshing, load to as pivot table
Replies
1
Views
106
Matt Allington
M
D
  • Question
Power Query refresh taking HOURS after adding custom column
Replies
3
Views
546
Matt Allington
M
M
  • Question
Can you change a source in power query for excel?
Replies
1
Views
158
alansidman
A
B
  • Question
How to use the lookup capability in Power Query
Replies
1
Views
229
Matt Allington
M
M
  • Question
Power Query imports date as number and is off by 4 years and one day
Replies
1
Views
201
KRice
K

Forum statistics

Threads
1,216,499
Messages
6,131,012
Members
449,613
Latest member
MedDash99

Share this page

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top