2

I have several python files, and I want to run in a different directory where it will search for a particular pattern from 100s of files. Let's say I have the python file in /home/jay directory, and I have 100s of files in /home/jay/data directory.

What can I do to achieve this?

My python code is :

import re
import os

os.chdir(r'/home/jay/data/')
files = open('*')
for line in files :
    line = line.rstrip()
    if re.search('Model' , line):
        print(line)

I'm getting following error:

Traceback (most recent call last):
  File "/home/jay/test.py", line 4, in ?
    files = open('*')
IOError: [Errno 2] No such file or directory: '*'
7
  • Go into your Python site-packages folder and open up a setup.py from NumPy or other to see how they are getting all the file names good luck Commented Aug 1, 2017 at 18:08
  • don't see setup.py file in site-packages. not sure which files i should look for. there other module imported such as BeautifulSoup, requests etc.. beginner in python, so pardon if this is silly question. Commented Aug 1, 2017 at 18:30
  • If you really wrote files = open(*) you wouldn't be getting an IOError but rather a SyntaxError. Please post your actual code. Commented Aug 1, 2017 at 18:33
  • print(line)` is also syntax error. Commented Aug 1, 2017 at 18:38
  • Also, you can't get an error saying No such file or directory: '/home/jay/Data/*' if your line really reads os.chdir(r'/home/jay/data/'). Please fix your code! Commented Aug 1, 2017 at 18:40

1 Answer 1

2

You are looking for os.listdir. It will give you a list of all the file names in the specified directory, which defaults to the current directory. The reason that '*' does not work is that it is a command-line construct that is expanded by your shell. You can only really use it in a shell or script that supports that sort of expansion. Since open does not go through the shell, it tries to find a file actually named *. Also, open can only deal with one file at a time.

import os, os.path, re

os.chdir(r'/home/jay/data/')
files = os.listdir()
for name in files:
    # Skip directories
    if os.path.isdir(name):
        continue
    with open(name) as file:
        for line in file:
            line = line.rstrip()
            if re.search('Model' , line):
                print(line)

That being said, as a matter of personal preference, I generally avoid using os.chdir. Instead, I prefer specifying full paths using os.path.join. Here is your example rewritten to do that:

from os import listdir
from os.path import join, isdir

folder = '/home/jay/data'
files = listdir(folder)
for name in files:
    name = join(folder, name)
    # Skip directories
    if isdir(name):
        continue
    with open(name) as file:
        for line in file:
            line = line.rstrip()
            if 'Model' in line:
                print(line)

I took the liberty of removing the regex completely since it only serves to slow things down in you have many files. If you do use a regex for some more complicated scenario, compile if before you use it using re.compile.

Furthermore, you are free to use relative paths here if you want. For example, if you are always running from /home/jay, you can set folder = 'data' instead of folder = '/home/jay/data' in the second example.

Sign up to request clarification or add additional context in comments.

17 Comments

@YakymPirozhenko. Thanks for that.
Try that code. gave me syntax error: File "test.py", line 6 with open(name) as file: ^ SyntaxError: invalid syntax
Tired your 2nd exapmple without regex : File "test.py", line 7 with open(join(folder, name)) as file: ^ SyntaxError: invalid syntax
@Jay. It looks like you are having issues with with. I have no problem running this code. How are you running your script exactly?
I'm running code normal like. " python test.py " from my company unix server.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.