File Management

3.1. File Management#

Hide code cell source

 
import sys
from pathlib import Path

current = Path.cwd()
for parent in [current, *current.parents]:
    if (parent / '_config.yml').exists():   ### root
        project_root = parent               ### Add project root
        break
else:
    project_root = Path.cwd().parent.parent

sys.path.insert(0, str(project_root))

from shared import thinkpython, diagram, jupyturtle, download, structshape

# Register as top-level modules so direct imports work in subsequent cells
sys.modules['thinkpython'] = thinkpython
sys.modules['diagram'] = diagram
sys.modules['jupyturtle'] = jupyturtle
sys.modules['download'] = download
sys.modules['structshape'] = structshape

3.1.1. Download Files#

3.1.2. Custom Code#

If you download the notebook files and place them in a folder in your project directory, or if you write your own notebooks and want to use features such as jupyturtle, you may need additional helper files. The cell below downloads the files used specifically for this book. You don’t need to understand this code yet, but you can see that it downloads several *.py files from Dr. Allen Downey’s GitHub repository to enable the required functionality.

You then create a folder in your project directory called shared, place the downloaded files in it, and add an empty file called __init__.py, which makes the shared folder a package. Later, in any project notebook, you can import modules from that package and use their functions:

from shared import [module]

The download function:

from os.path import basename, exists

def download(url):
    filename = basename(url)
    if not exists(filename):
        from urllib.request import urlretrieve

        local, _ = urlretrieve(url, filename)
        print("Downloaded " + str(local))
    return filename

To use the downloaded files as modules, create a directory named shared in the project root and place the files in it. In each notebook that needs these modules, put the code snippet below at the beginning of the notebook.

import sys
from pathlib import Path

current = Path.cwd()
for parent in [current, *current.parents]:
    if (parent / '_config.yml').exists():
        project_root = parent  # ← Add project root, not chapters
        break
else:
    project_root = Path.cwd().parent.parent

sys.path.insert(0, str(project_root))

from shared import thinkpython, diagram, jupyturtle

3.1.3. The with Statement#

Before writing to or reading from a file, Python needs to open it — and when you’re done, the file must be closed to flush data and release system resources. Forgetting to close a file can cause data loss or resource leaks.

The with statement handles this automatically:

with open('data.txt') as f:
    content = f.read()
# file is closed here automatically, even if an error occurred inside the block

This is equivalent to:

f = open('data.txt')
try:
    content = f.read()
finally:
    f.close()          # runs even if an exception is raised

Any object that supports the with statement is called a context manager. open() returns one. The with block guarantees that setup (opening) and teardown (closing) happen in the right order, even if something goes wrong.

Note

You will see with used throughout this course with files, database connections, network sockets, and more. Ch13 explains how to build your own context managers.

text = """Hello, this is the first line of a new text file.
This is the second line."""

with open('../../../data/text_file.txt', 'w') as f:
    f.write(text)

with open('../../../data/text_file.txt') as f:
    file_content = f.read()       ### file object
    print(file_content)

Hello, this is the first line of a new text file.
This is the second line.

3.1.4. Path()#

The Path() function is from the pathlib module — the modern way to work with file paths in Python. To use it, you

  • import Path() from Python’s pathlib module

  • build the path pointing to the directory or to the file.

from pathlib import Path

data = project_root / 'data'                  ### path of data folder
words_path = project_root / 'data' / 'words.txt'   ### path of words.txt file

After you have the path object, you can do many things with it.

# Building paths (works on Windows and Mac/Linux)
f = data / "words.txt"          ### use data folder and file name to build path

# File info
print(f.name)          # "words.txt"
print(f.stem)          # "words"
print(f.suffix)        # ".txt"
print(data.parent)     # parent directory of data/

# Checking existence
print(f.exists())
print(f.is_file())
print(f.is_dir())

# Reading and writing
### uncomment the following line to read the file
### the following line will throw an error if the file does not exist
# print(f.read_text())
tmp = Path('tmp_example.txt')
tmp.write_text("hello")         # write to a temp file, not words.txt
tmp.unlink()                    # clean up

# Current directory and home
print(Path.cwd())      # current working directory
print(Path.home())     # home directory (~)

# Listing files
print(list(data.iterdir()))            # all files in folder
print(list(data.glob("*.csv")))        # all CSVs
print(list(data.rglob("*.txt")))       # recursive

words.txt
words
.txt
/home/runner/work/py/py
True
True
False
/home/runner/work/py/py/chapters/appendices/packaging
/home/runner
[PosixPath('/home/runner/work/py/py/data/dr_jekyll.txt'), PosixPath('/home/runner/work/py/py/data/pg345_cleaned_10_lines.txt'), PosixPath('/home/runner/work/py/py/data/pg345_cleaned.txt'), PosixPath('/home/runner/work/py/py/data/pg345.txt'), PosixPath('/home/runner/work/py/py/data/pg345_cleaned_100_lines.txt'), PosixPath('/home/runner/work/py/py/data/words.txt'), PosixPath('/home/runner/work/py/py/data/text_file.txt'), PosixPath('/home/runner/work/py/py/data/pg43.txt')]
[]
[PosixPath('/home/runner/work/py/py/data/dr_jekyll.txt'), PosixPath('/home/runner/work/py/py/data/pg345_cleaned_10_lines.txt'), PosixPath('/home/runner/work/py/py/data/pg345_cleaned.txt'), PosixPath('/home/runner/work/py/py/data/pg345.txt'), PosixPath('/home/runner/work/py/py/data/pg345_cleaned_100_lines.txt'), PosixPath('/home/runner/work/py/py/data/words.txt'), PosixPath('/home/runner/work/py/py/data/text_file.txt'), PosixPath('/home/runner/work/py/py/data/pg43.txt')]

An example of using Path():

  1. Import Path()

  2. Build the path

  3. Check to see if file exists. If not, do something.

from pathlib import Path
words_path = project_root / 'data' / 'words.txt'
if not words_path.exists():
    download('https://raw.githubusercontent.com/AllenDowney/ThinkPython/v3/words.txt', words_path)