0

I have a TON of files (~260,000) sorted into files with lists of files by type. (binary, binary+Dx, text, text+x, etc...)

EI -

binary:

D:/Workspace/folder1/file.dll
D:/Workspace/folder2/file.res
D:/Workspace/folder3/file.pyd

text:

D:/Workspace/folder1/file.txt
D:/Workspace/folder2/file.txt
D:/Workspace/folder3/file.doc

etc, etc....

Using p4python, is there a way to add a list of files to a changelist? I have managed to find a way to add these line-by-line, but that will take FOREVER; I think I calculated it at like 3 days.... This method takes about 1 sec per line:

for lines in fileList:
    p4.run_edit(f"-c{changeNum}", f"-t{fileName}", lines )

The command I would try to mimic is like:

p4 add -t text -c 1214 D:/Workspace/folder1/file.txt, D:/Workspace/folder2/file.txt, D:/Workspace/folder3/file.doc

The documentation for p4python suggests that there is a way to use files that are already marked for add (in the default CL?), but that still requires these files to be added.

I have also found the "-x" option for add Here, but p4python doesn't seem to like this option. At least I can't get it to accept it...

In a nut-shell: I am just looking for a relatively time-efficient way to get the files from my disk to the depot, 100% in-code. My knowledge of Python is still pretty limited, so please forgive me if there is a super-obvious way to do this that I don't know about.

Here is what I have so far....

def generate_changelist( fileLoc, fileType ):
    fileList = []
    ## starts 'New' changelist
    print("Generating Changelist...")
    change = p4.fetch_change()
    change._description = f"P4 Script; Uploading files of type: {fileType}"
    
    ## give it a Number
    CL = p4.save_change( change ) # returns a list with the string "Change ### created"
    print(CL)
    for item in CL: changeNum = item.split()[1]
    
    print(f"Attaching Files to CL {changeNum}")
    with open(fileLoc, 'r') as f:
        for line in f:
            line = line[:-1]    # trims the '\n' off the end...
            fileList.append(line) # p4 command would go here....
    ## tinkering with these right now....
    p4.run_edit(f"-c{changeNum}", f"-t{fileType}", f"{fileList}")
    #p4.run("-s", f"-x {fileList}", "add" )
    
    ## works....but painfully slow....
    ## I know I can add this to the for loop above...
    #for lines in fileList:
    #    p4.run_edit(f"-c{changeNum}", f"-t{fileType}", f"{lines}")
    return changeNum
    

EDIT / Solution-ish: SO i was able to get this to work, kinda:

print(f"Attaching Files to CL {changeNum}")
with open(fileLoc, 'r') as f:
    for line in f:
        line = line[:-1]    # trims the '\n' off the end...
        if (DEBUG): print(line)
        fileList.append(line)
p4.run_edit(f"-c{changeNum}", f"-t{fileName}", fileList)

There seems to be an issue that P4Python is not throwing the error:

Screenshot

So, I think it might be that the list is just too large. Might need to have a counter to break up the fileList into chunks... 5k per?

Thanks all for the help!!

5
  • Use line = line.strip() to remove the newline. Commented Aug 24, 2023 at 23:45
  • The loop can be replaced with fileList = [line.strip() for line in f] Commented Aug 24, 2023 at 23:46
  • Are you sure you need to specify the filetype of each file individually? The type should have been auto-detected (or set based on the typemap) when the files were originally added; you don't need to re-specify it each time you edit them. If you eliminate that complication you can simply do p4 edit //... to open everything in your workspace for edit, and it will be much faster than opening each one for edit individually. Commented Aug 25, 2023 at 0:38
  • @Samwise - In our case, yes. the typemap is not picking up some types properly (utf16 for example). Upper management wants "Exact copies". :shrug: Commented Aug 25, 2023 at 16:27
  • The typemap isn't the same as the auto-detection; you configure it based on file path (usually file extension) and it's applied exactly as defined, every time. There's no way for it to "not pick up" a type. perforce.com/blog/vcs/perforce-p4-typemap But note that it's applied on add, not on subsequent edits (so that if you manually override the typemap for a specific file, it doesn't get set back the next time someone edits it). Commented Aug 25, 2023 at 16:55

2 Answers 2

2

Don't put filelist in a single argument, they should be separate arguments.

p4.run_edit(f"-c{changeNum}", f"-t{fileType}", *fileList)
Sign up to request clarification or add additional context in comments.

3 Comments

Removing the F string did the trick..... I forgot that that's for string references, not lists. Stoopid n00b error. TY for the help!
I think you still need * to spread it, unless run_edit() does that automatically.
looks like run_edit() does it for me.
0

Focusing on the "in a nutshell" part:

I am just looking for a relatively time-efficient way to get the files from my disk to the depot, 100% in-code.

Generating big lists of files and then using them to drive a script is certainly possible, but it's significantly less efficient than doing it the conventional way, and smacks of "XY problem".

Getting files from your disk to the depot in a Python script should usually be as simple as:

p4.run_reconcile()
p4.run_submit("-d", "P4 script uploading files")

If you want to set custom filetypes based on extension, this is best done via the p4 typemap command; the typemap is persistently stored on the server and is automatically applied when you add a file. A file's type "sticks" with it unless explicitly changed, so it's not necessary to re-specify it every time you make an edit.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.