How to differentiate between "a string" and "a actual code" in python?

Question

My works relates to instrumentation of code fragments in python code. So in my work i would be writing a script in python such that I take another python file as input and insert any necessary code in the required place with my script.

The following code is a sample code of a file which i would be instrumenting:

A.py #normal un-instrumented code

statements
....
....

def move(self,a):
    statements
    ......
    print "My function is defined" 
    ......

statements 
......

My script what actually does is to check each lines in the A.py and if there is a "def" then a code fragment is instrumented on top of the code the def function

The following example is how the final out put should be:

A.py #instrumented code

statements
....
....

@decorator    #<------ inserted code
def move(self,a):
    statements
    ......
    print "My function is defined" 
    ......

statements 
......

But I have been resulted with different output. The following code is the final output which i am getting:

A.py #instrumented code

statements
....
....

@decorator    #<------ inserted code
def move(self,a):
    statements
    ......
    @decorator #<------ inserted code [this should not occur]
    print "My function is defined" 
    ......

statements 
......

I can understand that in the instrumented code it recognizes "def" in the word "defined" and so it instruments the a code above it.

In realty the instrumented code has lots of these problems I was not able to properly instrument the given python file. Is there any other way to differentiate the actual "def" from string?

Thank you

How are you finding def in the instrumentation? If using a Regular Expresssion then try r'\bdef\b'. The \b marks a word boundary. — cdarke
– cdarke, Commented May 29, 2013 at 8:50
Will it work even when there is a statement like this print" This is a def" — Kaushik
– Kaushik, Commented May 29, 2013 at 8:55
no. To deal with text embedded in quotes you will need negative look-arounds. — cdarke
– cdarke, Commented May 29, 2013 at 9:05

Janne Karila · Accepted Answer · 2013-05-29 09:02:38Z

3

Use the ast module to parse the file properly.

This code prints the line number and column offset of each def statement:

import ast
with open('mymodule.py') as f:
    tree = ast.parse(f.read())
for node in ast.walk(tree):
    if isinstance(node, ast.FunctionDef):
        print node.lineno, node.col_offset

answered May 29, 2013 at 9:02

Janne Karila

25.3k6 gold badges60 silver badges97 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Kaushik Over a year ago

How do i use the column offset value when i am instrumenting a statement above the def function so that its is aligned correctly as the def function.

Janne Karila Over a year ago

@karthik I'm not sure how tab characters affect col_offset; try it out. I think you need to copy col_offset characters from beginning of the line and use that string to indent @decorator.

cdarke · Accepted Answer · 2013-05-29 09:15:30Z

0

You could use a Regular Expression. To avoid def inside quotes then you can use negative look-arounds:

import re

for line in open('A.py'):
    m = re.search(r"(?!<[\"'])\bdef\b(?![\"'])", line)
    if m:
        print r'@decorator    #<------ inserted code' 

    print line

However, there might be other occurances of def that you or I can't think of, and if we are not careful we end-up writing the Python parser all over again. @Janne Karila's suggestion of using ast.parse is probably safer in the long term.

answered May 29, 2013 at 9:15

cdarke

44.8k8 gold badges85 silver badges86 bronze badges

2 Comments

Janne Karila Over a year ago

Then there are multi-line strings.

cdarke Over a year ago

@JanneKarila: yup, that's one I didn't think of. Which only goes to show that your answer is preferred.

Collectives™ on Stack Overflow

How to differentiate between "a string" and "a actual code" in python?

2 Answers 2

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related