python help (copying files from one directory to another)

[JeenLeen]

I already solved this in SAS, but I'm trying to learn Python (within Anaconda) so I wanted to ask around. From some Googling, I got some code written but it doesn't work.

My google results are saved as comments in the code.
I have a set of CSV files that I need to distribute to school districts. My goal is to move the files at O:\data\myfiles to a sever with a folder structure like \\webftp\districts\Acct\<DistrictCode>\<DistrictCo de>000 for each district, where DistrictCode is a 4 digit number possibly with leading zeroes (e.g., 0900, 1000, 1100, etc.).

The DistrictCode is embedded in the CSV files I'm moving, and I figured out how to get it out, but the actual transfer of files isn't happening. I'm designing the output directory correctly (as confirmed by the commented out print(output) line of code).

Also, do you need multiple backslashes to make true backslashes (e.g., // means /)?
Note that where I'm copying data to starts with two slashes instead of just one.

import shutil
import os
#https://stackoverflow.com/questions/25070614/python-moving-multiple-files-from-one-folder-to-the-other-based-on-text-characte
#https://guide.freecodecamp.org/python/is-there-a-way-to-substring-a-string-in-python/

dest1='O:\\data\\myfiles'
dest2='\\\webftp\\districts\\Acct\\'

files=os.listdir(dest1)
for f in files:
    DistrictCode=f[9:13]
    output=dest2+DistrictCode+'\\'+DistrictCode+'000'
    #print(output)
    shutil.move(f,output)

[DavidSh]

It doesn't work? What happens when you try? Do you get an error message?

And, did the directories \\webftp\districts\Acct\<DistrictCode> already exist when you ran it?

[JeenLeen]

The folder locations do exist.

I tried re-running the code just now, and I got the message below. This seems to make sense as I had already written the file out there via SAS, so I guess this function only puts data out there; it doesn't overwrite a file already present.

  File "C:\Users\<redacted>\anaconda3\lib\shutil.py", line 561, in move
    raise Error("Destination path '%s' already exists" % real_dst)

Error: Destination path '\\webftp-0\0160\0160000\district_0160__DistName.csv' already exists

So I changed dest2 to a dummy folder I just made that fits the same schema.

Then I re-ran and got the same error as before.

  File "C:\Users\<redacted>\anaconda3\lib\shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:

FileNotFoundError: [Errno 2] No such file or directory: 'district_0160__DistName.csv'

From the error message, it looks like it's trying to copy the csv to a folder named "district_0160__DistName.csv". When, if ever, should the name of the csv be present in the shutil.move call?

[Jasdoif]

Then I re-ran and got the same error as before.

  File "C:\Users\<redacted>\anaconda3\lib\shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:

FileNotFoundError: [Errno 2] No such file or directory: 'district_0160__DistName.csv'

From the error message, it looks like it's trying to copy the csv to a folder named "district_0160__DistName.csv".

Actually, it's getting the error trying to open the source file (notice "fsrc" in the traceback). You'd want the full path to the file as the input, not just filename (which f is).

Personally, I'd recommend using os.path.join for the path manipulations:

import os.path
# Your stuff

for thisFilename in files:
    DistrictCode=thisFilename[9:13]
    inputPath=os.path.join(dest1,thisFilename) # ...dest1 and dest2 instead of srcFolder and dstFolder?
    outputPath=os.path.join(dest2,DistrictCode,(DistrictCode+'000'))
    print("Input Path: {0}\nOutput Path: {1}\n\n".format(inputPath, outputPath))
    # Do real stuff here, obviously

[JeenLeen]

Cool and thanks.

Is there an easy modification to tell it to create the folder structure if it doesn't already exist?

[Jasdoif]

Is there an easy modification to tell it to create the folder structure if it doesn't already exist?

I just noticed I can link directly to functions in the official documentation....os.makedirs will make an empty directory structure for a path, and if you give it "exist_ok=True" as a parameter it won't give an error if the folder already exists; you should be able to call it for outputPath that way to automatically get the folder structure (and not have to explicitly check whether it already exists). And conveniently you're already importing os, so you should just need to add a line before the move:

os.makedirs(outputPath, exist_ok=True)

[JeenLeen]

Thank you. It's working great now. My first work-operational Python program. Feels good!

I also learned that what I really want was .copy instead of .move!

[Jasdoif]

Oh, almost forgot!

Also, do you need multiple backslashes to make true backslashes (e.g., // means /)?
Note that where I'm copying data to starts with two slashes instead of just one.

Assuming you meant to type \ (backslash) instead of / ((forward) slash)....

Normally yes; as is the case in many languages, a backslash is how you reference special characters in a string ("\n" for newline, "\t" for tab, "\'" and '"' for quote characters that could clash with the string delimiter and/or your eyes, etc. etc.) so a literal backslash is treated as a special character ("") so you can still use a backslash in a string.

Python does have a "raw string notation" for string literals which disables nearly all of that, just put an "r" in front of the opening string delimiter....While it's fine for most cases, it interacts rather unintuitively with the string delimiter; so it looks weird for a string that ends with a backslash...but since you're using os.path.join, you don't need to end dest2 with a backslash (it's the main reason I recommend it; "wait, did I end this string with a path separator or not" is both critical to proper functioning and a giant pain to handle, so having a function take care of all of that is very nice/productive). So you could do this:

dest1=r'O:\data\myfiles'
dest2=r'\\webftp\districts\Acct'