Best method to copy a multitude of files to a folder?
Asked by
se_ven (
789
)
January 15th, 2010
[Windows Network Environment] I need to copy 140,000 files from many sub folders on a server to a common folder (on the server or a different server) so I can burn to DVD(s).
All these files are within one folder directory and in various sub folders. Mixed in with these files are files that I do not want to copy. I was able to get the exact file path for each of these documents. I created a batch file with the copy statements for each file and am having them copied from one server onto another. The batch file is being run from the desktop machine in my office. The process is taking very long, and I was wondering if I was doing this the most efficient way.
How could I adjust the method to be more efficient? Copy the files to a different folder on the same machine? Run the batch file from the server with the files on it? Use a different method than a batch file?
I’ve already figured out that I could have split the batch file into multiple and run it from multiple computers, but I don’t want to interrupt the process after it’s been running for so long.
Observing members:
0
Composing members:
0
11 Answers
Create new folders, drag and drop the stuff you don’t want to include. Provided the stuff you don’t want to include is considerably less.
Or pick a folder to burn and once you are in the burn process you can pick out and delete the stuff you do not want to burn.
You’re probably doing it the best way as it is, especially if you don’t want to stop it and start it again. Sometimes copying over the network can take a dog’s age to finish.
Although, if I remember right, early Vista releases had a problem with the network copying code that made it especially slow so in that case you may want to have the server copy from the workstation. This was fixed in SP1, I think, though.
@Pandora I don’t think you realize how many files there are. The set I want is 140,000 of probably 3 times that.
@mrentropy That makes me feel a little better, and this is all XP and Windows Server so no Vista slow down. I also ran a small portion on my Windows 7 machine and it seemed to do ok.
I find drag and drop fails with a large number of files. Your batch file is much better. It might be faster to run from the server.
It’s been a long time, but I kind of recall testing that scenario, and finding that running from the source was a lot faster than from the destination. Best to test that, though, my memory isn’t great.
@jaytkay Yeah, I should have done some testing first, but I didn’t really start thinking about that until I had already started. I’d imagine the way it’s running is less efficient because I have 3 machines involved (the source server, the destination server, and the workstation that is executing the batch file).
Click on the first file so it highlights. If everything is in a sub directory you just need to pull over the file that contains all the other stuff.
Or do Ctrl+A. That means ALL and will select (highlight) all the files at once.
Then click on one of the highlighted files and drag to the new folder or drive.
Let go and all will dump over.
If you right click instead of left click (when dragging) you will get an option to either move to new folder or copy to new folder.
Warning! If you accidentally double click in the process of dragging. You will/could duplicated every file you are trying to move.
Creating an additional 140,000 files would not be a fun experience.
Side note:
Key shortcuts I use the most.
Ctrl+A = All
Ctrl+C = Copy
Ctrl+V = Paste
Ctrl+Z = Un-do
Sounds like a headache. Good luck!
If you want to use the command line, I would do robocopy.
@mowens I think robocopy will only copy folders (directories), but I needed to grab specific files out of many different folders.
@shadling21 thanks!
Well, it looks like I found a solution that worked much better than the way I started out, and thought I would post it here if anyone needs it for future reference.
——————-
To recap, I had the exact filepath of over 140,000 files which were in 1 main directory, in many different sub-directories, and mixed among these files were many more files (~300,000) that I did not want to copy. So copying an entire directory or sub-directory was out of the question.
Since I had the exact filepath for the files I created a batch file utilizing the copy command. I started with 1 batch file with 140,000+ statements and started running it on one computer yesterday. After running overnight for about 17 hrs it had only copied about 50,000 of the files.
I then broke the remaining copy statements (~90,000) into batch files of 1,000 to 3,000 statements [CopyFilesXX.bat] and created 3 batch files [RunCopyXX.bat] that would call the 51 CopyFilesXX.bat with the copy statements in them. I then ran each RunCopyXX.bat from three different machines (the source server, my XP desktop, and my Windows7 laptop). The quickest file was done in about 25 min, and the longest file was done in 70 min.
I think the keys in this method improvement were using multiple machines to run the batch files, and breaking the batch files into smaller chunks which I think allowed the memory to be cleared or something. I’m not sure why the smaller chunks helped, but I’m pretty certain they helped quite a bit.
Hope this helps someone in a similar situation…
@jaytkay Thanks. I guess I didn’t look enough into robocopy, so that’ll be a potential upgrade if I have to do this again (hopefully not).
Answer this question 
This question is in the General Section. Responses must be helpful and on-topic.
Composing members:
0
