aboutsummaryrefslogtreecommitdiffstats
path: root/t
AgeCommit message (Collapse)AuthorFilesLines
2021-09-28rm: skip sparse paths with missing SKIP_WORKTREEDerrick Stolee1-2/+17
If a path does not match the sparse-checkout cone but is somehow missing the SKIP_WORKTREE bit, then 'git rm' currently succeeds in removing the file. One reason a user might be in this situation is a merge conflict outside of the sparse-checkout cone. Removing such a file might be problematic for users who are not sure what they are doing. Add a check to path_in_sparse_checkout() when 'git rm' is checking if a path should be considered for deletion. Of course, this check is ignored if the '--sparse' option is specified, allowing users who accept the risks to continue with the removal. This also removes a confusing behavior where a user asks for a directory to be removed, but only the entries that are within the sparse-checkout definition are removed. Now, 'git rm <dir>' will fail without '--sparse' and will succeed in removing all contained paths with '--sparse'. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-28rm: add --sparse optionDerrick Stolee1-0/+12
As we did previously in 'git add', add a '--sparse' option to 'git rm' that allows modifying paths outside of the sparse-checkout definition. The existing checks in 'git rm' are restricted to tracked files that have the SKIP_WORKTREE bit in the current index. Future changes will cause 'git rm' to reject removing paths outside of the sparse-checkout definition, even if they are untracked or do not have the SKIP_WORKTREE bit. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-28add: update --renormalize to skip sparse pathsDerrick Stolee1-1/+11
We added checks for path_in_sparse_checkout() to portions of 'git add' that add warnings and prevent stagins a modification, but we skipped the --renormalize mode. Update renormalize_tracked_files() to ignore cache entries whose path is outside of the sparse-checkout cone (unless --sparse is provided). Add a test in t3705. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-28add: update --chmod to skip sparse pathsDerrick Stolee1-1/+9
We added checks for path_in_sparse_checkout() to portions of 'git add' that add warnings and prevent staging a modification, but we skipped the --chmod mode. Update chmod_pathspec() to ignore cache entries whose path is outside of the sparse-checkout cone (unless --sparse is provided). Add a test in t3705. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-28add: implement the --sparse optionDerrick Stolee2-19/+27
We previously modified 'git add' to refuse updating index entries outside of the sparse-checkout cone. This is justified to prevent users from accidentally getting into a confusing state when Git removes those files from the working tree at some later point. Unfortunately, this caused some workflows that were previously possible to become impossible, especially around merge conflicts outside of the sparse-checkout cone. These were documented in tests within t1092. We now re-enable these workflows using a new '--sparse' option to 'git add'. This allows users to signal "Yes, I do know what I'm doing with these files," and accept the consequences of the files leaving the worktree later. We delay updating the advice message until implementing a similar option in 'git rm' and 'git mv'. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-28add: skip tracked paths outside sparse-checkout coneDerrick Stolee3-8/+27
When 'git add' adds a tracked file that is outside of the sparse-checkout cone, it checks the SKIP_WORKTREE bit to see if the file exists outside of the sparse-checkout cone. This is usually correct, except in the case of a merge conflict outside of the cone. Modify add_pathspec_matched_against_index() to be more careful about paths by checking the sparse-checkout patterns in addition to the SKIP_WORKTREE bit. This causes 'git add' to no longer allow files outside of the cone that removed the SKIP_WORKTREE bit due to a merge conflict. With only this change, users will only be able to add the file after adding the file to the sparse-checkout cone. A later change will allow users to force adding even though the file is outside of the sparse-checkout cone. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-28add: fail when adding an untracked sparse fileDerrick Stolee1-9/+28
The add_files() method in builtin/add.c takes a set of untracked files that are being added by the input pathspec and inserts them into the index. If these files are outside of the sparse-checkout cone, then they gain the SKIP_WORKTREE bit at some point. However, this was not checked before inserting into the index, so these files are added even though we want to avoid modifying the index outside of the sparse-checkout cone. Add a check within add_files() for these files and write the advice about files outside of the sparse-checkout cone. This behavior change modifies some existing tests within t1092. These tests intended to document how a user could interact with the existing behavior in place. Many of these tests need to be marked as expecting failure. A future change will allow these tests to pass by adding a flag to 'git add' that allows users to modify index entries outside of the sparse-checkout cone. The 'submodule handling' test is intended to document what happens to directories that contain a submodule when the sparse index is enabled. It is not trying to say that users should be able to add submodules outside of the sparse-checkout cone, so that test can be modified to avoid that operation. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-28t1092: behavior for adding sparse filesDerrick Stolee1-0/+28
Add some tests to demonstrate the current behavior around adding files outside of the sparse-checkout cone. Currently, untracked files are handled differently from tracked files. A future change will make these cases be handled the same way. Further expand checking that a failed 'git add' does not stage changes to the index. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-24t3705: test that 'sparse_entry' is unstagedDerrick Stolee1-0/+14
The tests in t3705-add-sparse-checkout.sh check to see how 'git add' behaves with paths outside the sparse-checkout definition. These currently check to see if a given warning is present but not that the index is not updated with the sparse entries. Add a new 'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving correctly. We need to modify setup_sparse_entry to actually commit the sparse_entry file so it exists at HEAD and as an entry in the index, but its exact contents are not staged in the index. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-09sparse-index: integrate with cherry-pick and rebaseDerrick Stolee1-2/+37
The hard work was already done with 'git merge' and the ORT strategy. Just add extra tests to see that we get the expected results in the non-conflict cases. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-09t1092: add cherry-pick, rebase testsDerrick Stolee1-6/+9
Add tests to check that cherry-pick and rebase behave the same in the sparse-index case as in the full index cases. These tests are agnostic to GIT_TEST_MERGE_ALGORITHM, so a full CI test suite will check both the 'ort' and 'recursive' strategies on this test. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-09merge-ort: expand only for out-of-cone conflictsDerrick Stolee1-2/+28
Merge conflicts happen often enough to want to avoid expanding a sparse index when they happen, as long as those conflicts are within the sparse-checkout cone. If a conflict exists outside of the sparse-checkout cone, then we still need to expand before iterating over the index entries. This is critical to do in advance because of how the original_cache_nr is tracked to allow inserting and replacing cache entries. Iterate over the conflicted files and check if any paths are outside of the sparse-checkout cone. If so, then expand the full index. Add a test that demonstrates that we do not expand the index, even when we hit a conflict within the sparse-checkout cone. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-09merge: make sparse-aware with ORTDerrick Stolee1-2/+10
Allow 'git merge' to operate without expanding a sparse index, at least not immediately. The index still will be expanded in a few cases: 1. If the merge strategy is 'recursive', then we enable command_requires_full_index at the start of the merge_recursive() method. We expect sparse-index users to also have the 'ort' strategy enabled. 2. With the 'ort' strategy, if the merge results in a conflicted file, then we expand the index before updating the working tree. The loop that iterates over the worktree replaces index entries and tracks 'origintal_cache_nr' which can become completely wrong if the index expands in the middle of the operation. This safety valve is important before that loop starts. A later change will focus this to only expand if we indeed have a conflict outside of the sparse-checkout cone. 3. Other merge strategies are executed as a 'git merge-X' subcommand, and those strategies are currently protected with the 'command_requires_full_index' guard. Some test updates are required, including a mistaken 'git checkout -b' that did not specify the base branch, causing merges to be fast-forward merges. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-07sparse-checkout: clear tracked sparse dirsDerrick Stolee1-0/+59
When changing the scope of a sparse-checkout using cone mode, we might have some tracked directories go out of scope. The current logic removes the tracked files from within those directories, but leaves the ignored files within those directories. This is a bit unexpected to users who have given input to Git saying they don't need those directories anymore. This is something that is new to the cone mode pattern type: the user has explicitly said "I want these directories and _not_ those directories." The typical sparse-checkout patterns more generally apply to "I want files with with these patterns" so it is natural to leave ignored files as they are. This focus on directories in cone mode provides us an opportunity to change the behavior. Leaving these ignored files in the sparse directories makes it impossible to gain performance benefits in the sparse index. When we track into these directories, we need to know if the files are ignored or not, which might depend on the _tracked_ .gitignore file(s) within the sparse directory. This depends on the indexed version of the file, so the sparse directory must be expanded. We must take special care to look for untracked, non-ignored files in these directories before deleting them. We do not want to delete any meaningful work that the users were doing in those directories and perhaps forgot to add and commit before switching sparse-checkout definitions. Since those untracked files might be code files that generated ignored build output, also do not delete any ignored files from these directories in that case. The users can recover their state by resetting their sparse-checkout definition to include that directory and continue. Alternatively, they can see the warning that is presented and delete the directory themselves to regain the performance they expect. By deleting the sparse directories when changing scope (or running 'git sparse-checkout reapply') we regain these performance benefits as if the repository was in a clean state. Since these ignored files are frequently build output or helper files from IDEs, the users should not need the files now that the tracked files are removed. If the tracked files reappear, then they will have newer timestamps than the build artifacts, so the artifacts will need to be regenerated anyway. Use the sparse-index as a data structure in order to find the sparse directories that can be safely deleted. Re-expand the index to a full one if it was full before. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-07t7519: rewrite sparse index testDerrick Stolee1-17/+21
The sparse index is tested with the FS Monitor hook and extension since f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test was very fragile because it shared an index across sparse and non-sparse behavior. Since that expansion and contraction could cause the index to lose its FS Monitor bitmap and token, behavior is fragile to changes in 'git sparse-checkout set'. Rewrite the test to use two clones of the original repo: full and sparse. This allows us to also keep the test files (actual, expect, trace2.txt) out of the repos we are testing with 'git status'. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-10Merge branch 'ds/add-with-sparse-index' into ds/sparse-index-ignored-filesJunio C Hamano1-9/+58
* ds/add-with-sparse-index: add: remove ensure_full_index() with --renormalize add: ignore outside the sparse-checkout in refresh() pathspec: stop calling ensure_full_index add: allow operating on a sparse-only index t1092: test merge conflicts outside cone
2021-08-06Merge branch 'cb/t7508-regexp-fix'Junio C Hamano1-1/+1
* cb/t7508-regexp-fix: t7508: avoid non POSIX BRE
2021-08-06Merge branch 'fc/disable-checkwinsize'Junio C Hamano1-0/+6
* fc/disable-checkwinsize: test: fix for COLUMNS and bash 5
2021-08-06test: fix for COLUMNS and bash 5Felipe Contreras1-0/+6
Since c49a177bec (test-lib.sh: set COLUMNS=80 for --verbose repeatability, 2021-06-29) multiple tests have been failing when using bash 5 because checkwinsize is enabled by default, therefore COLUMNS is reset using TIOCGWINSZ even for non-interactive shells. It's debatable whether or not bash should even be doing that, but for now we can avoid this undesirable behavior by disabling this option. Reported-by: Fabian Stelzer <fabian.stelzer@campoint.net> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> [jc: with SZEDER Gábor's suggestion to do this before setting COLUMNS] Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-04Merge branch 'tb/mingw-rmdir-symlink-to-directory'Junio C Hamano2-0/+16
Windows rmdir() equivalent behaves differently from POSIX ones in that when used on a symbolic link that points at a directory, the target directory gets removed, which has been corrected. * tb/mingw-rmdir-symlink-to-directory: mingw: align symlinks-related rmdir() behavior with Linux
2021-08-04Merge branch 'ab/getcwd-test'Junio C Hamano4-1/+32
Portability test update. * ab/getcwd-test: t0001: fix broken not-quite getcwd(3) test in bed67874e2
2021-08-04Merge branch 'pb/merge-autostash-more'Junio C Hamano1-0/+19
The local changes stashed by "git merge --autostash" were lost when the merge failed in certain ways, which has been corrected. * pb/merge-autostash-more: merge: apply autostash if merge strategy fails merge: apply autostash if fast-forward fails Documentation: define 'MERGE_AUTOSTASH' merge: add missing word "strategy" to a message
2021-08-04Merge branch 'en/ort-perf-batch-14'Junio C Hamano1-1/+1
Further optimization on "merge -sort" backend. * en/ort-perf-batch-14: merge-ort: restart merge with cached renames to reduce process entry cost merge-ort: avoid recursing into directories when we don't need to merge-ort: defer recursing into directories when merge base is matched merge-ort: add a handle_deferred_entries() helper function merge-ort: add data structures for allowable trivial directory resolves merge-ort: add some more explanations in collect_merge_info_callback() merge-ort: resolve paths early when we have sufficient information
2021-08-04Merge branch 'ds/commit-and-checkout-with-sparse-index'Junio C Hamano2-23/+221
"git checkout" and "git commit" learn to work without unnecessarily expanding sparse indexes. * ds/commit-and-checkout-with-sparse-index: unpack-trees: resolve sparse-directory/file conflicts t1092: document bad 'git checkout' behavior checkout: stop expanding sparse indexes sparse-index: recompute cache-tree commit: integrate with sparse-index p2000: compress repo names p2000: add 'git checkout -' test and decrease depth
2021-08-04Merge branch 'ar/submodule-add'Junio C Hamano2-6/+17
Rewrite of "git submodule" in C continues. * ar/submodule-add: submodule: drop unused sm_name parameter from show_fetch_remotes() submodule--helper: introduce add-clone subcommand submodule--helper: refactor module_clone() submodule: prefix die messages with 'fatal' t7400: test failure to add submodule in tracked path
2021-08-02mingw: align symlinks-related rmdir() behavior with LinuxThomas Bétous2-0/+16
When performing a rebase, rmdir() is called on the folder .git/logs. On Unix rmdir() exits without deleting anything in case .git/logs is a symbolic link but the equivalent functions on Windows (_rmdir, _wrmdir and RemoveDirectoryW) do not behave the same and remove the folder if it is symlinked even if it is not empty. This creates issues when folders in .git/ are symlinks which is especially the case when git-repo[1] is used: It replaces `.git/logs/` with a symlink. One such issue is that the _target_ of that symlink is removed e.g. during a `git rebase`, where `delete_reflog("REBASE_HEAD")` will not only try to remove `.git/logs/REBASE_HEAD` but then recursively try to remove the parent directories until an error occurs, a technique that obviously relies on `rmdir()` refusing to remove a symlink. This was reported in https://github.com/git-for-windows/git/issues/2967. This commit updates mingw_rmdir() so that its behavior is the same as Linux rmdir() in case of symbolic links. To verify that Git does not regress on the reported issue, this patch adds a regression test for the `git rebase` symptom, even if the same `rmdir()` behavior is quite likely to cause potential problems in other Git commands as well. [1]: git-repo is a python tool built on top of Git which helps manage many Git repositories. It stores all the .git/ folders in a central place by taking advantage of symbolic links. More information: https://gerrit.googlesource.com/git-repo/ Signed-off-by: Thomas Bétous <tomspycell@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-02t7508: avoid non POSIX BRECarlo Marcelo Arenas Belón1-1/+1
24c30e0b6 (wt-status: tolerate dangling marks, 2020-09-01) adds a test that uses a BRE which breaks at least with OpenBSD's grep. switch to an ERE as it is done for similar checks and while at it, remove the now obsolete test_i18ngrep call. Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-02Merge branch 'jt/bulk-prefetch'Junio C Hamano1-0/+33
"git read-tree" had a codepath where blobs are fetched one-by-one from the promisor remote, which has been corrected to fetch in bulk. * jt/bulk-prefetch: cache-tree: prefetch in partial clone read-tree unpack-trees: refactor prefetching code
2021-08-02Merge branch 'ab/bundle-tests'Junio C Hamano1-16/+56
"git bundle" gained more test coverage. * ab/bundle-tests: bundle tests: use test_cmp instead of grep bundle tests: use ">file" not ": >file"
2021-08-02Merge branch 'ps/perf-with-separate-output-directory'Junio C Hamano3-13/+24
Test update. * ps/perf-with-separate-output-directory: perf: fix when running with TEST_OUTPUT_DIRECTORY
2021-07-30t0001: fix broken not-quite getcwd(3) test in bed67874e2Ævar Arnfjörð Bjarmason4-1/+32
With a54e938e5b (strbuf: support long paths w/o read rights in strbuf_getcwd() on FreeBSD, 2017-03-26) we had t0001 break on systems like OpenBSD and AIX whose getcwd(3) has standard (but not like glibc et al) behavior. This was partially fixed in bed67874e2 (t0001: skip test with restrictive permissions if getpwd(3) respects them, 2017-08-07). The problem with that fix is that while its analysis of the problem is correct, it doesn't actually call getcwd(3), instead it invokes "pwd -P". There is no guarantee that "pwd -P" is going to call getcwd(3), as opposed to e.g. being a shell built-in. On AIX under both bash and ksh this test breaks because "pwd -P" will happily display the current working directory, but getcwd(3) called by the "git init" we're testing here will fail to get it. I checked whether clobbering the $PWD environment variable would affect it, and it didn't. Presumably these shells keep track of their working directory internally. There's possible follow-up work here in teaching strbuf_getcwd() to get the working directory with whatever method "pwd" uses on these platforms. See [1] for a discussion of that, but let's take the easy way out here and just skip these tests by fixing the GETCWD_IGNORES_PERMS prerequisite to match the limitations of strbuf_getcwd(). 1. https://lore.kernel.org/git/b650bef5-d739-d98d-e9f1-fa292b6ce982@web.de/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-29add: ignore outside the sparse-checkout in refresh()Derrick Stolee1-5/+1
Since b243012 (refresh_index(): add flag to ignore SKIP_WORKTREE entries, 2021-04-08), 'git add --refresh <path>' will output a warning message when the path is outside the sparse-checkout definition. The implementation of this warning happened in parallel with the sparse-index work to add ensure_full_index() calls throughout the codebase. Update this loop to have the proper logic that checks to see if the pathspec is outside the sparse-checkout definition. This avoids the need to expand the sparse directory entry and determine if the path is tracked, untracked, or ignored. We simply avoid updating the stat() information because there isn't even an entry that matches the path! Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-29pathspec: stop calling ensure_full_indexDerrick Stolee1-4/+3
The add_pathspec_matches_against_index() focuses on matching a pathspec to file entries in the index. This already works correctly for its only use: checking if untracked files exist in the index. The compatibility checks in t1092 already test that 'git add <dir>' works for a directory outside of the sparse cone. That provides coverage for removing this guard. This finalizes our ability to run 'git add .' without expanding a sparse index to a full one. This is evidenced by an update to t1092 and by these performance numbers for p2000-sparse-operations.sh: Test HEAD~1 HEAD -------------------------------------------------------------------------------- 2000.10: git add . (full-index-v3) 0.37(0.28+0.07) 0.36(0.27+0.06) -2.7% 2000.11: git add . (full-index-v4) 0.33(0.26+0.06) 0.32(0.28+0.05) -3.0% 2000.12: git add . (sparse-index-v3) 0.57(0.53+0.07) 0.06(0.06+0.07) -89.5% 2000.13: git add . (sparse-index-v4) 0.57(0.53+0.07) 0.05(0.03+0.09) -91.2% While the ~90% improvement is shown by the test results, it is worth noting that expanding the sparse index was adding overhead in previous commits. Comparing to the full index case, we see the performance go from 0.33s to 0.05s, an 85% improvement. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-29add: allow operating on a sparse-only indexDerrick Stolee1-7/+18
Disable command_requires_full_index for 'git add'. This does not require any additional removals of ensure_full_index(). The main reason is that 'git add' discovers changes based on the pathspec and the worktree itself. These are then inserted into the index directly, and calls to index_name_pos() or index_file_exists() already call expand_to_path() at the appropriate time to support a sparse-index. Add a test to check that 'git add -A' and 'git add <file>' does not expand the index at all, as long as <file> is not within a sparse directory. This does not help the global 'git add .' case. We can measure the improvement using p2000-sparse-operations.sh with these results: Test HEAD~1 HEAD ------------------------------------------------------------------------------ 2000.6: git add -A (full-index-v3) 0.35(0.30+0.05) 0.37(0.29+0.06) +5.7% 2000.7: git add -A (full-index-v4) 0.31(0.26+0.06) 0.33(0.27+0.06) +6.5% 2000.8: git add -A (sparse-index-v3) 0.57(0.53+0.07) 0.05(0.04+0.08) -91.2% 2000.9: git add -A (sparse-index-v4) 0.58(0.55+0.06) 0.05(0.05+0.06) -91.4% While the 91% improvement seems impressive, it's important to recognize that previously we had significant overhead for expanding the sparse-index. Comparing to the full index case, 'git add -A' goes from 0.37s to 0.05s, which is "only" an 86% improvement. This modification to 'git add' creates some behavior change depending on the use of a sparse index. We modify a test in t1092 to demonstrate these changes which will be remedied in future changes. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-29t1092: test merge conflicts outside coneDerrick Stolee1-0/+43
Conflicts can occur outside of the sparse-checkout definition, and in that case users might try to resolve the conflicts in several ways. Document a few of these ways in a test. Make it clear that this behavior is not necessarily the optimal flow, since users can become confused when Git deletes these files from the worktree in later commands. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-28Merge branch 'ps/t0000-output-directory-fix'Junio C Hamano2-2/+14
"TEST_OUTPUT_DIRECTORY=there make test" failed to work, which has been corrected. * ps/t0000-output-directory-fix: t0000: fix test if run with TEST_OUTPUT_DIRECTORY
2021-07-28Merge branch 'tb/reverse-midx'Junio C Hamano1-0/+5
The code that gives an error message in "git multi-pack-index" when no subcommand is given tried to print a NULL pointer as a strong, which has been corrected. * tb/reverse-midx: multi-pack-index: fix potential segfault without sub-command
2021-07-28Merge branch 'pb/dont-complete-aliased-options'Junio C Hamano1-0/+13
The completion support used to offer alternate spelling of options that exist only for compatibility, which has been corrected. * pb/dont-complete-aliased-options: parse-options: don't complete option aliases by default
2021-07-28Merge branch 'ds/status-with-sparse-index'Junio C Hamano2-11/+206
"git status" codepath learned to work with sparsely populated index without hydrating it fully. * ds/status-with-sparse-index: t1092: document bad sparse-checkout behavior fsmonitor: integrate with sparse index wt-status: expand added sparse directory entries status: use sparse-index throughout status: skip sparse-checkout percentage with sparse-index diff-lib: handle index diffs with sparse dirs dir.c: accept a directory as part of cone-mode patterns unpack-trees: unpack sparse directory entries unpack-trees: rename unpack_nondirectories() unpack-trees: compare sparse directories correctly unpack-trees: preserve cache_bottom t1092: add tests for status/add and sparse files t1092: expand repository data shape t1092: replace incorrect 'echo' with 'cat' sparse-index: include EXTENDED flag when expanding sparse-index: skip indexes with unmerged entries
2021-07-28Merge branch 'ab/pkt-line-tests'Junio C Hamano5-34/+24
Tests that cover protocol bits have been updated and helpers used there have been consolidated. * ab/pkt-line-tests: test-lib-functions: use test-tool for [de]packetize()
2021-07-28Merge branch 'jk/t0000-subtests-fix'Junio C Hamano1-10/+29
Test fix. * jk/t0000-subtests-fix: t0000: clear GIT_SKIP_TESTS before running sub-tests
2021-07-28Merge branch 'ab/attribute-format'Junio C Hamano1-1/+1
Many "printf"-like helper functions we have have been annotated with __attribute__() to catch placeholder/parameter mismatches. * ab/attribute-format: advice.h: add missing __attribute__((format)) & fix usage *.h: add a few missing __attribute__((format)) *.c static functions: add missing __attribute__((format)) sequencer.c: move static function to avoid forward decl *.c static functions: don't forward-declare __attribute__
2021-07-28Merge branch 'jk/log-decorate-optim'Junio C Hamano1-0/+14
Optimize "git log" for cases where we wasted cycles to load ref decoration data that may not be needed. * jk/log-decorate-optim: load_ref_decorations(): fix decoration with tags add_ref_decoration(): rename s/type/deco_type/ load_ref_decorations(): avoid parsing non-tag objects object.h: add lookup_object_by_type() function object.h: expand docstring for lookup_unknown_object() log: avoid loading decorations for userformats that don't need it pretty.h: update and expand docstring for userformat_find_requirements()
2021-07-28Merge branch 'sm/worktree-add-lock'Junio C Hamano1-1/+15
"git worktree add --lock" learned to record why the worktree is locked with a custom message. * sm/worktree-add-lock: worktree: teach `add` to accept --reason <string> with --lock worktree: mark lock strings with `_()` for translation t2400: clean up '"add" worktree with lock' test
2021-07-28Merge branch 'ew/many-alternate-optim'Junio C Hamano4-0/+100
Optimization for repositories with many alternate object store. * ew/many-alternate-optim: oidtree: a crit-bit tree for odb_loose_cache oidcpy_with_padding: constify `src' arg make object_directory.loose_objects_subdir_seen a bitmap avoid strlen via strbuf_addstr in link_alt_odb_entry speed up alt_odb_usable() with many alternates
2021-07-28Merge branch 'hj/commit-allow-empty-message'Junio C Hamano1-1/+1
"git commit --allow-empty-message" won't abort the operation upon an empty message, but the hint shown in the editor said otherwise. * hj/commit-allow-empty-message: commit: remove irrelavent prompt on `--allow-empty-message` commit: reorganise commit hint strings
2021-07-23merge: apply autostash if merge strategy failsPhilippe Blain1-0/+8
Since 'git merge' learned '--autostash' in a03b55530a (merge: teach --autostash option, 2020-04-07), 'cmd_merge', once it is determined that we have to create a merge commit, calls 'create_autostash' if '--autostash' is given. As explained in a03b55530a, and made more abvious by the tests added in that commit, the autostash is then applied if the merge succeeds, either directly or by committing (after conflict resolution or if '--no-commit' was given), or if the merge is aborted with 'git merge --abort'. In some other cases, like the user calling 'git reset --merge' or 'git merge --quit', the autostash is not applied, but saved in the stash list. However, there exists a scenario that creates an autostash but does not apply nor save it to the stash list: if the chosen merge strategy completely fails to handle the merge, i.e. 'try_merge_strategy' returns 2. Apply the autostash in that case also. An easy way to test that is to try to merge more than two commits but explicitely ask for the 'recursive' merge strategy. Signed-off-by: Philippe Blain <levraiphilippeblain@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-23merge: apply autostash if fast-forward failsPhilippe Blain1-0/+11
Since 'git merge' learned '--autostash' in a03b55530a (merge: teach --autostash option, 2020-04-07), 'cmd_merge', in the fast-forward case, calls 'create_autostash' before calling 'checkout_fast_forward' if '--autostash' is given. However, if 'checkout_fast_forward' fails, the autostash is not applied to the working tree, nor saved in the stash list, since the code simply calls 'goto done'. Be more helpful to the user by applying the autostash in that case. An easy way to test a failing fast-forward is when we are merging a branch that has a tracked file that conflicts with an untracked file in the working tree. Signed-off-by: Philippe Blain <levraiphilippeblain@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-23cache-tree: prefetch in partial clone read-treeJonathan Tan1-0/+33
"git read-tree" checks the existence of the blobs referenced by the given tree, but does not bulk prefetch them. Add a bulk prefetch. The lack of prefetch here was noticed at $DAYJOB during a merge involving some specific commits, but I couldn't find a minimal merge that didn't also trigger the prefetch in check_updates() in unpack-trees.c (and in all these cases, the lack of prefetch in cache-tree.c didn't matter because all the relevant blobs would have already been prefetched by then). This is why I used read-tree here to exercise this code path. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-22bundle tests: use test_cmp instead of grepÆvar Arnfjörð Bjarmason1-13/+53
Change the bundle tests to fully compare the expected "git ls-remote" or "git bundle list-heads" output, instead of merely grepping it. This avoids subtle regressions in the tests. In f62e0a39b6 (t5704 (bundle): add tests for bundle --stdin, 2010-04-19) the "bundle --stdin <rev-list options>" test was added to make sure we didn't include the tag. But since the --stdin mode didn't work until 5bb0fd2cab (bundle: arguments can be read from stdin, 2021-01-11) our grepping of "master" (later "main") missed the important part of the test. Namely that we should not include the "refs/tags/tag" tag in that case. Since the test only grepped for "main" in the output we'd miss a regression in that code. So let's use test_cmp instead, and also in the other nearby tests where it's easy. This does make things a bit more verbose in the case of the test that's checking the bundle header, since it's different under SHA1 and SHA256. I think this makes test easier to follow. I've got some WIP changes to extend the "git bundle" command to dump parts of the header out, which are easier to understand if we test the output explicitly like this. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-22bundle tests: use ">file" not ": >file"Ævar Arnfjörð Bjarmason1-3/+3
Change uses of ":" on the LHS of a ">" to the more commonly used ">file" pattern in t/t5607-clone-bundle.sh. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-22Merge branch 'bc/rev-list-without-commit-line'Junio C Hamano1-2/+78
"git rev-list" learns to omit the "commit <object-name>" header lines from the output with the `--no-commit-header` option. * bc/rev-list-without-commit-line: rev-list: add option for --pretty=format without header
2021-07-22Merge branch 'ab/send-email-optim'Junio C Hamano1-0/+29
"git send-email" optimization. * ab/send-email-optim: perl: nano-optimize by replacing Cwd::cwd() with Cwd::getcwd() send-email: move trivial config handling to Perl perl: lazily load some common Git.pm setup code send-email: lazily load modules for a big speedup send-email: get rid of indirect object syntax send-email: use function syntax instead of barewords send-email: lazily shell out to "git var" send-email: lazily load config for a big speedup send-email: copy "config_regxp" into git-send-email.perl send-email: refactor sendemail.smtpencryption config parsing send-email: remove non-working support for "sendemail.smtpssl" send-email tests: test for boolean variables without a value send-email tests: support GIT_TEST_PERL_FATAL_WARNINGS=true
2021-07-20unpack-trees: resolve sparse-directory/file conflictsDerrick Stolee1-12/+12
When running unpack_trees() with a sparse index, we attempt to operate on the index without expanding the sparse directory entries. Thus, we operate by manipulating entire directories and passing them to the unpack function. In the case of the 'git checkout' command, this is the twoway_merge() function. There are several cases in twoway_merge() that handle different situations. One new one to add is the case of a directory/file conflict where the directory is sparse. Before the sparse index, such a conflict would appear as a list of file additions and deletions. Now, twoway_merge() initializes 'current', 'oldtree', and 'newtree' from src[0], src[1], and src[2], then sets 'oldtree' to NULL because it is equal to the df_conflict_entry. The way to determine that we have a directory/file conflict is to test that 'current' and 'newtree' disagree on being sparse directory entries. When we are in this case, we want to resolve the situation by calling merged_entry(). This allows replacing the 'current' entry with the 'newtree' entry. This is important for cases where we want to run 'git checkout' across the conflict and have the new HEAD represent the new file type at that path. The first NEEDSWORK comment dropped in t1092 demonstrates this necessary behavior. However, we still are in a confusing state when 'current' corresponds to a staged change within a sparse directory that is not present at HEAD. This should be atypical, because it requires adding a change outside of the sparse-checkout cone, but it is possible. Since we are unable to determine that this is a staged change within twoway_merge(), we cannot add a case to reject the merge at this point. I believe this is due to the use of df_conflict_entry in the place of 'oldtree' instead of using the valud at HEAD, which would provide some perspective to this decision. Any change that would allow this differentiation for staged entries would need to involve information further up in unpack_trees(). That work should be done, sometime, because we are further confusing the behavior of a directory/file conflict when staging a change in the directory. The two cases 'checkout behaves oddly with df-conflict-?' in t1092 demonstrate that even without a sparse-checkout, Git is not consistent in its behavior. Neither of the two options seems correct, either. This change makes the sparse-index behave differently than the typcial sparse-checkout case, but it does match the full checkout behavior in the df-conflict-2 case. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-20t1092: document bad 'git checkout' behaviorDerrick Stolee1-2/+140
Add new branches to the test repo that demonstrate directory/file conflicts in different ways. Since the directory 'folder1/' has adjacent files 'folder1-', 'folder1.txt', and 'folder10' it causes searches for 'folder1/' to land in a different place in the index than a search for 'folder1'. This causes a change in behavior when working with the df-conflict-1 and df-conflict-2 branches, whose only difference is that the first uses 'folder1' as the conflict and the other uses 'folder2' which does not have these adjacent files. We can extend two tests that compare the behavior across different 'git checkout' commands, and we see already that the behavior will be different in some cases and not in others. The difference between the two test loops is that one uses 'git reset --hard' between iterations. Further, we isolate the behavior of creating a staged change within a directory and then checking out a branch where that directory is replaced with a file. A full checkout behaves differently across these two cases, while a sparse-checkout cone behaves consistently. In both cases, the behavior is wrong. In one case, the staged change is dropped entirely. The other case the staged change is kept, replacing the file at that location, but none of the other files in the directory are kept. Likely, the correct behavior in this case is to reject the checkout and report the conflict, leaving HEAD in its previous location. None of the cases behave this way currently. Use comments to demonstrate that the tested behavior is only a documentation of the current, incorrect behavior to ensure we do not _accidentally_ change it. Instead, we would prefer to change it on purpose with a future change. At this point, the sparse-index does not handle these 'git checkout' commands correctly. Or rather, it _does_ reject the 'git checkout' when we have the staged change, but for the wrong reason. It also rejects the 'git checkout' commands when there is no staged change and we want to replace a directory with a file. A fix for that unstaged case will follow in the next change, but that will make the sparse-index agree with the full checkout case in these documented incorrect behaviors. Helped-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-20merge-ort: restart merge with cached renames to reduce process entry costElijah Newren1-1/+1
The merge algorithm mostly consists of the following three functions: collect_merge_info() detect_and_process_renames() process_entries() Prior to the trivial directory resolution optimization of the last half dozen commits, process_entries() was consistently the slowest, followed by collect_merge_info(), then detect_and_process_renames(). When the trivial directory resolution applies, it often dramatically decreases the amount of time spent in the two slower functions. Looking at the performance results in the previous commit, the trivial directory resolution optimization helps amazingly well when there are no relevant renames. It also helps really well when reapplying a long series of linear commits (such as in a rebase or cherry-pick), since the relevant renames may well be cached from the first reapplied commit. But when there are any relevant renames that are not cached (represented by the just-one-mega testcase), then the optimization does not help at all. Often, I noticed that when the optimization does not apply, it is because there are a handful of relevant sources -- maybe even only one. It felt frustrating to need to recurse into potentially hundreds or even thousands of directories just for a single rename, but it was needed for correctness. However, staring at this list of functions and noticing that process_entries() is the most expensive and knowing I could avoid it if I had cached renames suggested a simple idea: change collect_merge_info() detect_and_process_renames() process_entries() into collect_merge_info() detect_and_process_renames() <cache all the renames, and restart> collect_merge_info() detect_and_process_renames() process_entries() This may seem odd and look like more work. However, note that although we run collect_merge_info() twice, the second time we get to employ trivial directory resolves, which makes it much faster, so the increased time in collect_merge_info() is small. While we run detect_and_process_renames() again, all renames are cached so it's nearly a no-op (we don't call into diffcore_rename_extended() but we do have a little bit of data structure checking and fixing up). And the big payoff comes from the fact that process_entries(), will be much faster due to having far fewer entries to process. This restarting only makes sense if we can save recursing into enough directories to make it worth our while. Introduce a simple heuristic to guide this. Note that this heuristic uses a "wanted_factor" that I have virtually no actual real world data for, just some back-of-the-envelope quasi-scientific calculations that I included in some comments and then plucked a simple round number out of thin air. It could be that tweaking this number to make it either higher or lower improves the optimization. (There's slightly more here; when I first introduced this optimization, I used a factor of 10, because I was completely confident it was big enough to not cause slowdowns in special cases. I was certain it was higher than needed. Several months later, I added the rough calculations which make me think the optimal number is close to 2; but instead of pushing to the limit, I just bumped it to 3 to reduce the risk that there are special cases where this optimization can result in slowing down the code a little. If the ratio of path counts is below 3, we probably will only see minor performance improvements at best anyway.) Also, note that while the diffstat looks kind of long (nearly 100 lines), more than half of it is in two comments explaining how things work. For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 205.1 ms ± 3.8 ms 204.2 ms ± 3.0 ms mega-renames: 1.564 s ± 0.010 s 1.076 s ± 0.015 s just-one-mega: 479.5 ms ± 3.9 ms 364.1 ms ± 7.0 ms Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-20t0000: fix test if run with TEST_OUTPUT_DIRECTORYPatrick Steinhardt2-2/+14
Testcases in t0000 are quite special given that they many of them run nested testcases to verify that testing functionality itself works as expected. These nested testcases are realized by writing a new ad-hoc test script which again sources test-lib.sh, where the new script is created in a nested subdirectory located beneath the current trash directory. We then execute the new test script with the nested subdirectory as current working directory and explicitly re-export TEST_OUTPUT_DIRECTORY to point to that directory. While this works as expected in the general case, it falls apart when the developer has TEST_OUTPUT_DIRECTORY explicitly defined either via the environment or via config.mak and runs "make test". In that case, test-lib.sh will clobber the value that we've just carefully set up to instead contain what the developer has defined. As a result, the TEST_OUTPUT_DIRECTORY continues to point at the root output directory, not at the nested one. This issue causes breakage in the 'test_atexit is run' test case: the nested test case writes files into "../../", which is assumed to be the parent's trash directory. But because TEST_OUTPUT_DIRECTORY already points to to the root output directory, we instead end up writing those files outside of the output directory. The parent test case will then try to check whether those files still exist in its own trash directory, which thus must fail now. Fix the issue by adding a new TEST_OUTPUT_DIRECTORY_OVERRIDE variable. If set, then we'll always override the TEST_OUTPUT_DIRECTORY with its value after sourcing GIT-BUILD-OPTIONS. Signed-off-by: Patrick Steinhardt <ps@pks.im> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-19multi-pack-index: fix potential segfault without sub-commandTaylor Blau1-0/+5
Since cd57bc41bb (builtin/multi-pack-index.c: display usage on unrecognized command, 2021-03-30) we have used a "usage" label to avoid having two separate callers of usage_with_options (one when no arguments are given, and another for unrecognized sub-commands). But the first caller has been broken since cd57bc41bb, since it will happily jump to usage without arguments, and then pass argv[0] to the "unrecognized subcommand" error. Many compilers will save us from a segfault here, but the end result is ugly, since it mentions an unrecognized subcommand when we didn't even pass one, and (on GCC) includes "(null)" in its output. Move the "usage" label down past the error about unrecognized subcommands so that it is only triggered when it should be. While we're at it, bulk up our test coverage in this area, too. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-19t0000: clear GIT_SKIP_TESTS before running sub-testsJeff King1-10/+29
In t0000, we run several fake "sub-test" suites to verify the behavior of the test suite. But because we don't clear the parent environment completely, the sub-tests can be fooled by variables meant for the parent. For example: GIT_SKIP_TESTS=t1234 ./t0000-basic.sh fails when a sub-test expects its fake t1234 to actually run. This particular pattern is unlikely in practice; we're running a single script, and there is no t1234 in the real test suite anyway (not yet, at least). A more real-world example is: GIT_SKIP_TESTS=t[^0]* make test to run only the t0* tests. The fix is conceptually simple: we should clear the GIT_SKIP_TESTS variable when running the sub-tests, because its contents (if any) will be meant for the main test suite. This is easy to do centrally in our sub-test helper. But there's a catch: some of our tests do set GIT_SKIP_TESTS intentionally to test the feature. We need to allow them to continue to set it, but clear it for all the other tests. And the sub-test helper can't tell if the GIT_SKIP_TESTS it sees is from a test or not. We can handle this by adding a new option to the helper to let callers specify the skip list. I considered adding a more general "--eval" option to let callers set up the env for the sub-test however they like. That would cover this case and possible future ones. But the quoting gets awkward for the callers (since we're now 2 layers deep in evals!), so I went with the simpler more specific solution. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-19test-lib-functions: use test-tool for [de]packetize()Ævar Arnfjörð Bjarmason5-34/+24
The shell+perl "[de]packetize()" helper functions were added in 4414a150025 (t/lib-git-daemon: add network-protocol helpers, 2018-01-24), and around the same time we added the "pkt-line" helper in 74e70029615 (test-pkt-line: introduce a packet-line test helper, 2018-03-14). For some reason it seems we've mostly used the shell+perl version instead of the helper since then. There were discussions around 88124ab2636 (test-lib-functions: make packetize() more efficient, 2020-03-27) and cacae4329fa (test-lib-functions: simplify packetize() stdin code, 2020-03-29) to improve them and make them more efficient. There was one good reason to do so, we needed an equivalent of "test-tool pkt-line pack", but that command wasn't capable of handling input with "\n" (a feature) or "\0" (just because it happens to be printf-based under the hood). Let's add a "pkt-line-raw" helper for that, and expose is at a packetize_raw() to go with the existing packetize() on the shell level, this gives us the smallest amount of change to the tests themselves. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-16Merge branch 'ds/gender-neutral-doc'Junio C Hamano1-1/+0
Update the documentation not to assume users are of certain gender and adds to guidelines to do so. * ds/gender-neutral-doc: *: fix typos comments: avoid using the gender of our users doc: avoid using the gender of other people
2021-07-16Merge branch 'jt/partial-clone-submodule-1'Junio C Hamano4-0/+68
Prepare the internals for lazily fetching objects in submodules from their promisor remotes. * jt/partial-clone-submodule-1: promisor-remote: teach lazy-fetch in any repo run-command: refactor subprocess env preparation submodule: refrain from filtering GIT_CONFIG_COUNT promisor-remote: support per-repository config repository: move global r_f_p_c to repo struct
2021-07-16Merge branch 'dd/test-stdout-count-lines'Junio C Hamano3-76/+98
Tiny test clean-up. * dd/test-stdout-count-lines: t6402: preserve git exit status code t6400: preserve git ls-files exit status code test-lib-functions: introduce test_stdout_line_count
2021-07-16Merge branch 'hn/refs-test-cleanup'Junio C Hamano2-8/+6
Test clean-up. * hn/refs-test-cleanup: t7509: avoid direct file access for writing CHERRY_PICK_HEAD t1415: avoid direct filesystem access for writing refs
2021-07-16Merge branch 'ab/mktag-tests'Junio C Hamano1-15/+106
Fill test gaps. * ab/mktag-tests: mktag tests: test fast-export mktag tests: test for-each-ref mktag tests: test update-ref and reachable fsck mktag tests: test hash-object --literally and unreachable fsck mktag tests: invert --no-strict test mktag tests: parse out options in helper
2021-07-16Merge branch 'ab/show-branch-tests'Junio C Hamano2-70/+149
Fill test gaps. * ab/show-branch-tests: show-branch tests: add missing tests show-branch: don't <COLOR></RESET> for space characters show-branch tests: modernize test code show-branch tests: rename the one "show-branch" test file
2021-07-16Merge branch 'ab/fetch-negotiate-segv-fix'Junio C Hamano1-0/+16
Code recently added to support common ancestry negotiation during "git push" did not sanity check its arguments carefully enough. * ab/fetch-negotiate-segv-fix: fetch: fix segfault in --negotiate-only without --negotiation-tip=* fetch: document the --negotiate-only option send-pack.c: move "no refs in common" abort earlier
2021-07-16Merge branch 'tb/midx-use-checksum'Junio C Hamano1-0/+13
When rebuilding the multi-pack index file reusing an existing one, we used to blindly trust the existing file and ended up carrying corrupted data into the updated file, which has been corrected. * tb/midx-use-checksum: midx: report checksum mismatches during 'verify' midx: don't reuse corrupt MIDXs when writing commit-graph: rewrite to use checksum_valid() csum-file: introduce checksum_valid()
2021-07-16Merge branch 'en/merge-dir-rename-corner-case-fix'Junio C Hamano1-0/+175
The merge code had funny interactions between content based rename detection and directory rename detection. * en/merge-dir-rename-corner-case-fix: merge-recursive: handle rename-to-self case merge-ort: ensure we consult df_conflict and path_conflicts t6423: test directory renames causing rename-to-self
2021-07-16Merge branch 'en/ort-perf-batch-13'Junio C Hamano1-0/+440
Performance tweaks of "git merge -sort" around lazy fetching of objects. * en/ort-perf-batch-13: merge-ort: add prefetching for content merges diffcore-rename: use a different prefetch for basename comparisons diffcore-rename: allow different missing_object_cb functions t6421: add tests checking for excessive object downloads during merge promisor-remote: output trace2 statistics for number of objects fetched
2021-07-16Merge branch 'en/ort-perf-batch-12'Junio C Hamano1-1/+1
More fix-ups and optimization to "merge -sort". * en/ort-perf-batch-12: merge-ort: miscellaneous touch-ups Fix various issues found in comments diffcore-rename: avoid unnecessary strdup'ing in break_idx merge-ort: replace string_list_df_name_compare with faster alternative
2021-07-16parse-options: don't complete option aliases by defaultPhilippe Blain1-0/+13
Since 'OPT_ALIAS' was created in 5c387428f1 (parse-options: don't emit "ambiguous option" for aliases, 2019-04-29), 'git clone --git-completion-helper', which is used by the Bash completion script to list options accepted by clone (via '__gitcomp_builtin'), lists both '--recurse-submodules' and its alias '--recursive', which was not the case before since '--recursive' had the PARSE_OPT_HIDDEN flag set, and options with this flag are skipped by 'parse-options.c::show_gitcomp', which implements 'git <cmd> --git-completion-helper'. This means that typing 'git clone --recurs<TAB>' will yield both '--recurse-submodules' and '--recursive', which is not ideal since both do the same thing, and so the completion should directly complete the canonical option. At the point where 'show_gitcomp' is called in 'parse_options_step', 'preprocess_options' was already called in 'parse_options', so any aliases are now copies of the original options with a modified help text indicating they are aliases. Helpfully, since 64cc539fd2 (parse-options: don't leak alias help messages, 2021-03-21) these copies have the PARSE_OPT_FROM_ALIAS flag set, so check that flag early in 'show_gitcomp' and do not print them, unless the user explicitely requested that *all* completion be shown (by setting 'GIT_COMPLETION_SHOW_ALL'). After all, if we want to encourage the use of '--recurse-submodules' over '--recursive', we'd better just suggest the former. The only other options alias is 'log' and friends' '--mailmap', which is an alias for '--use-mailmap', but the Bash completion helpers for these commands do not use '__gitcomp_builtin', and thus are unnaffected by this change. Test the new behaviour in t9902-completion.sh. As a side effect, this also tests the correct behaviour of GIT_COMPLETION_SHOW_ALL, which was not tested before. Note that since '__gitcomp_builtin' caches the options it shows, we need to re-source the completion script to clear that cache for the second test. Signed-off-by: Philippe Blain <levraiphilippeblain@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-15worktree: teach `add` to accept --reason <string> with --lockStephen Manz1-0/+14
The default reason stored in the lock file, "added with --lock", is unlikely to be what the user would have given in a separate `git worktree lock` command. Allowing `--reason` to be specified along with `--lock` when adding a working tree gives the user control over the reason for locking without needing a second command. Signed-off-by: Stephen Manz <smanz@alum.mit.edu> Reviewed-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14checkout: stop expanding sparse indexesDerrick Stolee1-1/+9
Previous changes did the necessary improvements to unpack-trees.c and diff-lib.c in order to modify a sparse index based on its comparision with a tree. The only remaining work is to remove some ensure_full_index() calls and add tests that verify that the index is not expanded in our interesting cases. Include 'switch' and 'restore' in these tests, as they share a base implementation with 'checkout'. Here are the relevant performance results from p2000-sparse-operations.sh: Test HEAD~1 HEAD -------------------------------------------------------------------------------- 2000.18: git checkout -f - (full-v3) 0.49(0.43+0.03) 0.47(0.39+0.05) -4.1% 2000.19: git checkout -f - (full-v4) 0.45(0.37+0.06) 0.42(0.37+0.05) -6.7% 2000.20: git checkout -f - (sparse-v3) 0.76(0.71+0.07) 0.04(0.03+0.04) -94.7% 2000.21: git checkout -f - (sparse-v4) 0.75(0.72+0.04) 0.05(0.06+0.04) -93.3% It is important to compare the full index case to the sparse index case, as the previous results for the sparse index were inflated by the index expansion. For index v4, this is an 88% improvement. On an internal repository with over two million paths at HEAD and a sparse-checkout definition containing ~60,000 of those paths, 'git checkout' went from 3.5s to 297ms with this change. The theoretical optimum where only those ~60,000 paths exist was 275ms, so the extra sparse directory entries contribute a 22ms overhead. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14commit: integrate with sparse-indexDerrick Stolee1-4/+43
Update 'git commit' to allow using the sparse-index in memory without expanding to a full one. The only place that had an ensure_full_index() call was in cache_tree_update(). The recursive algorithm for update_one() was already updated in 2de37c536 (cache-tree: integrate with sparse directory entries, 2021-03-03) to handle sparse directory entries in the index. Most of this change involves testing different command-line options that allow specifying which on-disk changes should be included in the commit. This includes no options (only take currently-staged changes), -a (take all tracked changes), and --include (take a list of specific changes). To simplify testing that these options do not expand the index, update the test that previously verified that 'git status' does not expand the index with a helper method, ensure_not_expanded(). This allows 'git commit' to operate much faster when the sparse-checkout cone is much smaller than the full list of files at HEAD. Here are the relevant lines from p2000-sparse-operations.sh: Test HEAD~1 HEAD ---------------------------------------------------------------------------------- 2000.14: git commit -a -m A (full-v3) 0.35(0.26+0.06) 0.36(0.28+0.07) +2.9% 2000.15: git commit -a -m A (full-v4) 0.32(0.26+0.05) 0.34(0.28+0.06) +6.3% 2000.16: git commit -a -m A (sparse-v3) 0.63(0.59+0.06) 0.04(0.05+0.05) -93.7% 2000.17: git commit -a -m A (sparse-v4) 0.64(0.59+0.08) 0.04(0.04+0.04) -93.8% It is important to compare the full-index case to the sparse-index case, so the improvement for index version v4 is actually an 88% improvement in this synthetic example. In a real repository with over two million files at HEAD and 60,000 files in the sparse-checkout definition, the time for 'git commit -a' went from 2.61 seconds to 134ms. I compared this to the result if the index only contained the paths in the sparse-checkout definition and found the theoretical optimum to be 120ms, so the out-of-cone paths only add a 12% overhead. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14p2000: compress repo namesDerrick Stolee1-10/+10
By using shorter names for the test repos, we will get a slightly more compressed performance summary without comprimising clarity. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14p2000: add 'git checkout -' test and decrease depthDerrick Stolee1-7/+20
As we increase our list of commands to test in p2000-sparse-operations.sh, we will want to have a slightly smaller test repository. Reduce the size by a factor of four by reducing the depth of the step that creates a big index around a moderately-sized repository. Also add a step to run 'git checkout -' on repeat. This requires having a previous location in the reflog, so add that to the initialization steps. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14t1092: document bad sparse-checkout behaviorDerrick Stolee1-4/+6
There are several situations where a repository with sparse-checkout enabled will act differently than a normal repository, and in ways that are not intentional. The test t1092-sparse-checkout-compatibility.sh documents some of these deviations, but a casual reader might think these are intentional behavior changes. Add comments on these tests that make it clear that these behaviors should be updated. Using 'NEEDSWORK' helps contributors find that these are potential areas for improvement. Helped-by: Elijah Newren <newren@gmail.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14fsmonitor: integrate with sparse indexDerrick Stolee1-0/+49
If we need to expand a sparse-index into a full one, then the FS Monitor bitmap is going to be incorrect. Ensure that we start fresh at such an event. While this is currently a performance drawback, the eventual hope of the sparse-index feature is that these expansions will be rare and hence we will be able to keep the FS Monitor data accurate across multiple Git commands. These tests are added to demonstrate that the behavior is the same across a full index and a sparse index, but also that file modifications to a tracked directory outside of the sparse cone will trigger ensure_full_index(). Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14wt-status: expand added sparse directory entriesDerrick Stolee1-0/+33
It is difficult, but possible, to get into a state where we intend to add a directory that is outside of the sparse-checkout definition. Add a test to t1092-sparse-checkout-compatibility.sh that demonstrates this using a combination of 'git reset --mixed' and 'git checkout --orphan'. This test failed before because the output of 'git status --porcelain=v2' would not match on the lines for folder1/: * The sparse-checkout repo (with a full index) would output each path name that is intended to be added. * The sparse-index repo would only output that "folder1/" is staged for addition. The status should report the full list of files to be added, and so this sparse-directory entry should be expanded to a full list when reaching it inside the wt_status_collect_changes_initial() method. Use read_tree_at() to assist. Somehow, this loop over the cache entries was not guarded by ensure_full_index() as intended. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14status: use sparse-index throughoutDerrick Stolee1-4/+9
By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). In refresh_index(), we loop through the index entries to refresh their stat() information. However, sparse directories have no stat() information to populate. Ignore these entries. This allows 'git status' to no longer expand a sparse index to a full one. This is further tested by dropping the "-uno" option and adding an untracked file into the worktree. The performance test p2000-sparse-checkout-operations.sh demonstrates these improvements: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0% 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7% 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3% 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.34s to 0.05s improvement of 85%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14status: skip sparse-checkout percentage with sparse-indexDerrick Stolee1-0/+8
'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. This change is the reason we use 'git status --porcelain=v2' in t1092-sparse-checkout-compatibility.sh. We don't want to ensure that this message is equal across both modes, but instead just the important information about staged, modified, and untracked files are compared. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14t1092: add tests for status/add and sparse filesDerrick Stolee1-0/+38
Before moving to update 'git status' and 'git add' to work with sparse indexes, add an explicit test that ensures the sparse-index works the same as a normal sparse-checkout when the worktree contains directories and files outside of the sparse cone. Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is not in the sparse cone. When 'folder1/a' is modified, the file is not shown as modified and adding it will fail. This is new behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08). Before that change, these adds would be silently ignored. Untracked files are fine: adding new files both with 'git add .' and 'git add folder1/' works just as in a full checkout. This may not be entirely desirable, but we are not intending to change behavior at the moment, only document it. A future change could alter the behavior to be more sensible, and this test could be modified to satisfy the new expected behavior. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14t1092: expand repository data shapeDerrick Stolee1-3/+41
As more features integrate with the sparse-index feature, more and more special cases arise that require different data shapes within the tree structure of the repository in order to demonstrate those cases. Add several interesting special cases all at once instead of sprinkling them across several commits. The interesting cases being added here are: * Add sparse-directory entries on both sides of directories within the sparse-checkout definition. * Add directories outside the sparse-checkout definition who have only one entry and are the first entry of a directory with multiple entries. * Add filenames adjacent to a sparse directory entry that sort before and after the trailing slash. Later tests will take advantage of these shapes, but they also deepen the tests that already exist. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14t1092: replace incorrect 'echo' with 'cat'Derrick Stolee1-1/+1
This fixes the test data shape to be as expected, allowing rename detection to work properly now that the 'larger-content' file actually has meaningful lines. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14sparse-index: skip indexes with unmerged entriesDerrick Stolee1-0/+22
The sparse-index format is designed to be compatible with merge conflicts, even those outside the sparse-checkout definition. The reason is that when converting a full index to a sparse one, a cache entry with nonzero stage will not be collapsed into a sparse directory entry. However, this behavior was not tested, and a different behavior within convert_to_sparse() fails in this scenario. Specifically, cache_tree_update() will fail when unmerged entries exist. convert_to_sparse_rec() uses the cache-tree data to recursively walk the tree structure, but also to compute the OIDs used in the sparse-directory entries. Add an index scan to convert_to_sparse() that will detect if these merge conflict entries exist and skip the conversion before trying to update the cache-tree. This is marked as NEEDSWORK because this can be removed with a suitable update to cache_tree_update() or a similar method that can construct a cache-tree with invalid nodes, but still allow creating the nodes necessary for creating sparse directory entries. It is possible that in the future we will not need to make such an update, since if we do not expand a sparse-index into a full one, this conversion does not need to happen. Thus, this can be deferred until the merge machinery is made to integrate with the sparse-index. Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14load_ref_decorations(): fix decoration with tagsJeff King1-0/+14
Commit 88473c8bae ("load_ref_decorations(): avoid parsing non-tag objects", 2021-06-22) introduced a shortcut to `add_ref_decoration()`: Rather than calling `parse_object()`, we go for `oid_object_info()` and then `lookup_object_by_type()` using the type just discovered. As detailed in the commit message, this provides a significant time saving. Unfortunately, it also changes the behavior: We lose all annotated tags from the decoration. The reason this happens is in the loop where we try to peel the tags, we won't necessarily have parsed that first object. If we haven't, its `tagged` field will be NULL, so we won't actually add a decoration for the pointed-to object. Make sure to parse the tag object at the top of the peeling loop. This effectively restores the pre-88473c8bae parsing -- but only of tags, allowing us to keep most of the possible speedup from 88473c8bae. On my big ~220k ref test case (where it's mostly non-tags), the timings [using "git log -1 --decorate"] are: - before either patch: 2.945s - with my broken patch: 0.707s - with [this patch]: 0.788s The simplest way to do this is to just conditionally parse before the loop: if (obj->type == OBJ_TAG) parse_object(&obj->oid); But we can observe that our tag-peeling loop needs to peel already, to examine recursive tags-of-tags. So instead of introducing a new call to parse_object(), we can simply move the parsing higher in the loop: instead of parsing the new object before we loop, parse each tag object before we look at its "tagged" field. This has another beneficial side effect: if a tag points at a commit (or other non-tag type), we do not bother to parse the commit at all now. And we know it is a commit without calling oid_object_info(), because parsing the surrounding tag object will have created the correct in-core object based on the "type" field of the tag. Our test coverage for --decorate was obviously not good, since we missed this quite-basic regression. The new tests covers an annotated tag (showing the fix), but also that we correctly show annotations for lightweight tags and double-annotated tag-of-tags. Reported-by: Martin Ågren <martin.agren@gmail.com> Helped-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Jeff King <peff@peff.net> Reviewed-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-14t2400: clean up '"add" worktree with lock' testStephen Manz1-1/+1
- remove unneeded `git rev-parse` which must have come from a copy-paste of another test - unlock the worktree with test_when_finished Signed-off-by: Stephen Manz <smanz@alum.mit.edu> Reviewed-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-13Merge branch 'rs/grep-parser-fix'Junio C Hamano1-0/+9
"git grep --and -e foo" ought to have been diagnosed as an error but instead segfaulted, which has been corrected. * rs/grep-parser-fix: grep: report missing left operand of --and
2021-07-13Merge branch 'ab/pre-auto-gc-hook-test'Junio C Hamano1-0/+46
Test fix. * ab/pre-auto-gc-hook-test: gc tests: add a test for the "pre-auto-gc" hook
2021-07-13Merge branch 'jk/union-merge-binary'Junio C Hamano1-0/+18
The "union" conflict resolution variant misbehaved when used with binary merge driver. * jk/union-merge-binary: ll_union_merge(): rename path_unused parameter ll_union_merge(): pass name labels to ll_xdl_merge() ll_binary_merge(): handle XDL_MERGE_FAVOR_UNION
2021-07-13Merge branch 'ab/describe-tests-fix'Junio C Hamano1-62/+72
Various updates to tests around "git describe" * ab/describe-tests-fix: describe tests: support -C in "check_describe" describe tests: fix nested "test_expect_success" call describe tests: don't rely on err.actual from "check_describe" describe tests: refactor away from glob matching describe tests: improve test for --work-tree & --dirty
2021-07-13Merge branch 'ab/pickaxe-pcre2'Junio C Hamano3-18/+170
Rewrite the backend for "diff -G/-S" to use pcre2 engine when available. * ab/pickaxe-pcre2: (22 commits) xdiff-interface: replace discard_hunk_line() with a flag xdiff users: use designated initializers for out_line pickaxe -G: don't special-case create/delete pickaxe -G: terminate early on matching lines xdiff-interface: allow early return from xdiff_emit_line_fn xdiff-interface: prepare for allowing early return pickaxe -S: slightly optimize contains() pickaxe: rename variables in has_changes() for brevity pickaxe -S: support content with NULs under --pickaxe-regex pickaxe: assert that we must have a needle under -G or -S pickaxe: refactor function selection in diffcore-pickaxe() perf: add performance test for pickaxe pickaxe/style: consolidate declarations and assignments diff.h: move pickaxe fields together again pickaxe: die when --find-object and --pickaxe-all are combined pickaxe: die when -G and --pickaxe-regex are combined pickaxe tests: add missing test for --no-pickaxe-regex being an error pickaxe tests: test for -G, -S and --find-object incompatibility pickaxe tests: add test for "log -S" not being a regex pickaxe tests: add test for diffgrep_consume() internals ...
2021-07-13Merge branch 'hn/prep-tests-for-reftable'Junio C Hamano18-159/+170
Preliminary clean-up of tests before the main reftable changes hits the codebase. * hn/prep-tests-for-reftable: (22 commits) t1415: set REFFILES for test specific to storage format t4202: mark bogus head hash test with REFFILES t7003: check reflog existence only for REFFILES t7900: stop checking for loose refs t1404: mark tests that muck with .git directly as REFFILES. t2017: mark --orphan/logAllRefUpdates=false test as REFFILES t1414: mark corruption test with REFFILES t1407: require REFFILES for for_each_reflog test test-lib: provide test prereq REFFILES t5304: use "reflog expire --all" to clear the reflog t5304: restyle: trim empty lines, drop ':' before > t7003: use rev-parse rather than FS inspection t5000: inspect HEAD using git-rev-parse t5000: reformat indentation to the latest fashion t1301: fix typo in error message t1413: use tar to save and restore entire .git directory t1401-symbolic-ref: avoid direct filesystem access t1401: use tar to snapshot and restore repo state t5601: read HEAD using rev-parse t9300: check ref existence using test-helper rather than a file system check ...
2021-07-13Merge branch 'zh/cat-file-batch-fix'Junio C Hamano1-0/+22
"git cat-file --batch-all-objects"" misbehaved when "--batch" is in use and did not ask for certain object traits. * zh/cat-file-batch-fix: cat-file: merge two block into one cat-file: handle trivial --batch format with --batch-all-objects
2021-07-13advice.h: add missing __attribute__((format)) & fix usageÆvar Arnfjörð Bjarmason1-1/+1
Add the missing __attribute__((format)) checking to advise_if_enabled(). This revealed a trivial issue introduced in b3b18d16213 (advice: revamp advise API, 2020-03-02). We treated the argv[1] as a format string, but did not intend to do so. Let's use "%s" and pass argv[1] as an argument instead. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-12submodule: prefix die messages with 'fatal'Atharva Raykar2-7/+7
The standard `die()` function that is used in C code prefixes all the messages passed to it with 'fatal: '. This does not happen with the `die` used in 'git-submodule.sh'. Let's prefix each of the shell die messages with 'fatal: ' so that when they are converted to C code, the error messages stay the same as before the conversion. Note that the shell version of `die` exits with error code 1, while the C version exits with error code 128. In practice, this does not change any behaviour, as no functionality in 'submodule add' and 'submodule update' relies on the value of the exit code. Signed-off-by: Atharva Raykar <raykar.ath@gmail.com> Mentored-by: Christian Couder <christian.couder@gmail.com> Mentored-by: Shourya Shukla <periperidip@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-12rev-list: add option for --pretty=format without headerbrian m. carlson1-2/+78
In general, we encourage users to use plumbing commands, like git rev-list, over porcelain commands, like git log, when scripting. However, git rev-list has one glaring problem that prevents it from being used in certain cases: when --pretty is used with a custom format, it always prints out a line containing "commit" and the object ID. This makes it unsuitable for many scripting needs, and forces users to use git log instead. While we can't change this behavior for backwards compatibility, we can add an option to suppress this behavior, so let's do so, and call it "--no-commit-header". Additionally, add the corresponding positive option to switch it back on. Note that this option doesn't affect the built-in formats, only custom formats. This is exactly the same behavior as users already have from git log and is what most users will be used to. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-09commit: remove irrelavent prompt on `--allow-empty-message`Hu Jialun1-1/+1
Even when the `--allow-empty-message` option is given, "git commit" offers an interactive editor session with prefilled message that says the commit will be aborted if the buffer is emptied, which is wrong. Remove the "an empty message aborts" part from the message when the option is given to fix it. Helped-by: Junio C Hamano <gitster@pobox.com> Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Helped-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Hu Jialun <hujialun@comp.nus.edu.sg> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-08Merge branch 'ar/test-code-cleanup'Junio C Hamano5-5/+5
Test code clean-up. * ar/test-code-cleanup: t: fix whitespace around &&
2021-07-08Merge branch 'ab/cmd-foo-should-return'Junio C Hamano4-4/+4
Code clean-up. * ab/cmd-foo-should-return: builtins + test helpers: use return instead of exit() in cmd_*
2021-07-08Merge branch 'ab/fix-columns-to-80-during-tests'Junio C Hamano1-2/+3
Output from some of our tests were affected by the width of the terminal that they were run in, which has been corrected by exporting a fixed value in the COLUMNS environment. * ab/fix-columns-to-80-during-tests: test-lib.sh: set COLUMNS=80 for --verbose repeatability
2021-07-08Merge branch 'ar/more-typofix'Junio C Hamano2-2/+2
Typofixes. * ar/more-typofix: git-worktree.txt: fix typo in example path t: fix typos in test messages blame: correct name of config option in docs
2021-07-08Merge branch 'jk/test-without-readlink-1'Junio C Hamano3-3/+9
Some test scripts assumed that readlink(1) was universally installed and available, which is not the case. * jk/test-without-readlink-1: t: use portable wrapper for readlink(1)
2021-07-08Merge branch 'jx/sideband-cleanup'Junio C Hamano31-1013/+1013
The side-band demultiplexer that is used to display progress output from the remote end did not clear the line properly when the end of line hits at a packet boundary, which has been corrected. Also comes with test clean-ups. * jx/sideband-cleanup: test: refactor to use "get_abbrev_oid" to get abbrev oid test: refactor to use "test_commit" to create commits test: compare raw output, not mangle tabs and spaces sideband: don't lose clear-to-eol at packet boundary
2021-07-08Merge branch 'jk/test-avoid-globmatch-with-skip-patterns'Junio C Hamano1-12/+22
We broke "GIT_SKIP_TESTS=t?000" to skip certain tests in recent update, which got fixed. * jk/test-avoid-globmatch-with-skip-patterns: test-lib: avoid accidental globbing in match_pattern_list()
2021-07-08Merge branch 'ab/config-hooks-path-testfix'Junio C Hamano1-0/+1
Test fix. * ab/config-hooks-path-testfix: pre-commit hook tests: don't leave "actual" nonexisting on failure
2021-07-08Merge branch 'jx/t6020-with-older-bash'Junio C Hamano1-19/+31
Work around inefficient glob substitution in older versions of bash by rewriting parts of a test. * jx/t6020-with-older-bash: t6020: fix incompatible parameter expansion
2021-07-08Merge branch 'ar/typofix'Junio C Hamano2-2/+2
Typofixes. * ar/typofix: *: fix typos which duplicate a word
2021-07-08Merge branch 'dd/svn-test-wo-locale-a'Junio C Hamano4-22/+24
"git-svn" tests assumed that "locale -a", which is used to pick an available UTF-8 locale, is available everywhere. A knob has been introduced to allow testers to specify a suitable locale to use. * dd/svn-test-wo-locale-a: t: use user-specified utf-8 locale for testing svn
2021-07-08fetch: fix segfault in --negotiate-only without --negotiation-tip=*Ævar Arnfjörð Bjarmason1-0/+16
The recent --negotiate-only option would segfault in the call to oid_array_for_each() in negotiate_using_fetch() unless one or more --negotiation-tip=* options were provided. All of the other tests for the feature combine both, but nothing was checking this assumption, let's do that and add a test for it. Fixes a bug in 9c1e657a8fd (fetch: teach independent negotiation (no packfile), 2021-05-04). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-07oidtree: a crit-bit tree for odb_loose_cacheEric Wong4-0/+100
This saves 8K per `struct object_directory', meaning it saves around 800MB in my case involving 100K alternates (half or more of those alternates are unlikely to hold loose objects). This is implemented in two parts: a generic, allocation-free `cbtree' and the `oidtree' wrapper on top of it. The latter provides allocation using alloc_state as a memory pool to improve locality and reduce free(3) overhead. Unlike oid-array, the crit-bit tree does not require sorting. Performance is bound by the key length, for oidtree that is fixed at sizeof(struct object_id). There's no need to have 256 oidtrees to mitigate the O(n log n) overhead like we did with oid-array. Being a prefix trie, it is natively suited for expanding short object IDs via prefix-limited iteration in `find_short_object_filename'. On my busy workstation, p4205 performance seems to be roughly unchanged (+/-8%). Startup with 100K total alternates with no loose objects seems around 10-20% faster on a hot cache. (800MB in memory savings means more memory for the kernel FS cache). The generic cbtree implementation does impose some extra overhead for oidtree in that it uses memcmp(3) on "struct object_id" so it wastes cycles comparing 12 extra bytes on SHA-1 repositories. I've not yet explored reducing this overhead, but I expect there are many places in our code base where we'd want to investigate this. More information on crit-bit trees: https://cr.yp.to/critbit.html Signed-off-by: Eric Wong <e@80x24.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-07t7400: test failure to add submodule in tracked pathAtharva Raykar1-0/+11
Add a test to ensure failure on adding a submodule to a directory with tracked contents in the index. As we are going to refactor and port to C some parts of `git submodule add`, let's add a test to help ensure no regression is introduced. Signed-off-by: Atharva Raykar <raykar.ath@gmail.com> Mentored-by: Christian Couder <christian.couder@gmail.com> Based-on-patch-by: Shourya Shukla <periperidip@gmail.com> Mentored-by: Shourya Shukla <periperidip@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-06t7509: avoid direct file access for writing CHERRY_PICK_HEADHan-Wen Nienhuys1-2/+2
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-06t1415: avoid direct filesystem access for writing refsHan-Wen Nienhuys1-6/+4
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-06t6402: preserve git exit status codeĐoàn Trần Công Danh1-68/+64
In t6402, we're checking number of files in the index and the working tree by piping the output of Git's command to "wc -l", thus losing the exit status code of git. Let's use the new helper test_stdout_line_count in order to preserve Git's exit status code. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-06t6400: preserve git ls-files exit status codeĐoàn Trần Công Danh1-8/+8
In t6400, we're checking number of files in the index and the working tree by piping the output of "git ls-files" to "wc -l", thus losing the exit status code of git. Let's use the newly introduced test_stdout_line_count in order to check the exit status code of Git's command. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-06test-lib-functions: introduce test_stdout_line_countĐoàn Trần Công Danh1-0/+26
In some tests, we're checking the number of lines in output of some commands, including but not limited to Git's command. We're doing the check by running those commands in the left side of a pipe, thus losing the exit status code of those commands. Meanwhile, we really want to check the exit status code of Git's command. Let's write the output of those commands to a temporary file, and use test_line_count separately in order to check exit status code of those commands properly. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-02perf: fix when running with TEST_OUTPUT_DIRECTORYPatrick Steinhardt3-13/+24
When the TEST_OUTPUT_DIRECTORY is defined, then all test data will be written in that directory instead of the default directory located in "t/". While this works as expected for our normal tests, performance tests fail to locate and aggregate performance data because they don't know to handle TEST_OUTPUT_DIRECTORY correctly and always look at the default location. Fix the issue by adding a `--results-dir` parameter to "aggregate.perl" which identifies the directory where results are and by making the "run" script awake of the TEST_OUTPUT_DIRECTORY variable. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-07-01test-lib: avoid accidental globbing in match_pattern_list()Jeff King1-12/+22
We have a custom match_pattern_list() function which we use for matching test names (like "t1234") against glob-like patterns (like "t1???") for $GIT_SKIP_TESTS, --verbose-only, etc. Those patterns may have multiple whitespace-separated elements (e.g., "t0* t1234 t5?78"). The callers of match_pattern_list thus pass the strings unquoted, so that the shell does the usual field-splitting into separate arguments. But this also means the shell will do the usual globbing for each argument, which can result in us seeing an expansion based on what's in the filesystem, rather than the real pattern. For example, if I have the path "t5000" in the filesystem, and you feed the pattern "t?000", that _should_ match the string "t0000", but it won't after the shell has expanded it to "t5000". This has been a bug ever since that function was introduced. But it didn't usually trigger since we typically use the function inside the trash directory, which has a very limited set of files that are unlikely to match. It became a lot easier to trigger after edc23840b0 (test-lib: bring $remove_trash out of retirement, 2021-05-10), because now we match $GIT_SKIP_TESTS before even entering the trash directory. So the t5000 example above can be seen with: GIT_SKIP_TESTS=t?000 ./t0000-basic.sh which should skip all tests but doesn't. We can fix this by using "set -f" to ask the shell not to glob (which is in POSIX, so should hopefully be portable enough). We only want to do this in a subshell (to avoid polluting the rest of the script), which means we need to get the whole string intact into the match_pattern_list function by quoting it. Arguably this is a good idea anyway, since it makes it much more obvious that we intend to split, and it's not simply sloppy scripting. Diagnosed-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-30merge-recursive: handle rename-to-self caseElijah Newren1-3/+3
Directory rename detection can cause transitive renames, e.g. if the two different sides of history each do one half of: A/file -> B/file B/ -> C/ then directory rename detection transitively renames to give us A/file -> C/file However, when C/ == A/, note that this gives us A/file -> A/file. merge-recursive assumed that any rename D -> E would have D != E. While that is almost always true, the above is a special case where it is not. So we cannot do things like delete the rename source, we cannot assume that a file existing at path E implies a rename/add conflict and we have to be careful about what stages end up in the output. This change feels a bit hackish. It took me surprisingly many hours to find, and given merge-recursive's design causing it to attempt to enumerate all combinations of edge and corner cases with special code for each combination, I'm worried there are other similar fixes needed elsewhere if we can just come up with the right special testcase. Perhaps an audit would rule it out, but I have not the energy. merge-recursive deserves to die, and since it is on its way out anyway, fixing this particular bug narrowly will have to be good enough. Reported-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-30merge-ort: ensure we consult df_conflict and path_conflictsElijah Newren1-3/+3
Path conflicts (typically rename path conflicts, e.g. rename/rename(1to2) or rename/add/delete), and directory/file conflicts should obviously result in files not being marked as clean in the merge. We had a codepath where we missed consulting the path_conflict and df_conflict flags, based on match_mask. Granted, it requires an unusual setup to trigger this codepath (directory rename causing rename-to-self is the only case I can think of), but we still need to handle it. To make it clear that we have audited the other codepaths that do not explicitly mention these flags, add some assertions that the flags are not set. Reported-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-30t6423: test directory renames causing rename-to-selfElijah Newren1-0/+175
Directory rename detection can cause transitive renames, e.g. if the two different sides of history each do one half of: A/file -> B/file B/ -> C/ then directory rename detection transitively renames to give us C/file. Since the default for merge.directoryRenames is conflict, this results in an error message saying it is unclear whether the file should be placed at B/file or C/file. What if C/ is A/, though? In such a case, the transitive rename would give us A/file, the original name we started with. Logically, having an error message with B/file vs. A/file should be fine, as should leaving the file where it started. But the logic in both merge-recursive and merge-ort did not handle a case of a filename being renamed to itself correctly; merge-recursive had two bugs, and merge-ort had one. Add some testcases covering such a scenario. Based-on-testcase-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-30grep: report missing left operand of --andRené Scharfe1-0/+9
Git grep allows combining two patterns with --and. It checks and reports if the second pattern is missing when compiling the expression. A missing first pattern, however, is only reported later at match time. Thus no error is returned if no matching is done, e.g. because no file matches the also given pathspec. When that happens we get an expression tree with an GREP_NODE_AND node and a NULL pointer to the missing left child. free_pattern_expr() tries to dereference it during the cleanup at the end, which results in a segmentation fault. Fix this by verifying the presence of the left operand at expression compilation time. Reported-by: Matthew Hughes <matthewhughes934@gmail.com> Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-29test-lib.sh: set COLUMNS=80 for --verbose repeatabilityÆvar Arnfjörð Bjarmason1-2/+3
Some tests will fail under --verbose because while we've unset COLUMNS since b1d645b58ac (tests: unset COLUMNS inherited from environment, 2012-03-27), we also look for the columns with an ioctl(.., TIOCGWINSZ, ...) on some platforms. By setting COLUMNS again we preempt the TIOCGWINSZ lookup in pager.c's term_columns(), it'll take COLUMNS over TIOCGWINSZ, This fixes t0500-progress-display.sh., which broke because of a combination of the this issue and the progress output reacting to the column width since 545dc345ebd (progress: break too long progress bar lines, 2019-04-12). The t5324-split-commit-graph.sh fails in a similar manner due to progress output, see [1] for details. The issue is not specific to progress.c, the diff code also checks COLUMNS and some of its tests can be made to fail in a similar manner[2], anything that invokes a pager is potentially affected. See ea77e675e56 (Make "git help" react to window size correctly, 2005-12-18) and ad6c3739a33 (pager: find out the terminal width before spawning the pager, 2012-02-12) for how the TIOCGWINSZ code ended up in pager.c 1. http://lore.kernel.org/git/20210624051253.GG6312@szeder.dev 2. https://lore.kernel.org/git/20210627074419.GH6312@szeder.dev/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28midx: report checksum mismatches during 'verify'Taylor Blau1-0/+5
'git multi-pack-index verify' inspects the data in an existing MIDX for correctness by checking that the recorded object offsets are correct, and so on. But it does not check that the file's trailing checksum matches the data that it records. So, if an on-disk corruption happened to occur in the final few bytes (and all other data was recorded correctly), we would: - get a clean result from 'git multi-pack-index verify', but - be unable to reuse the existing MIDX when writing a new one (since we now check for checksum mismatches before reusing a MIDX) Teach the 'verify' sub-command to recognize corruption in the checksum by calling midx_checksum_valid(). Suggested-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28midx: don't reuse corrupt MIDXs when writingTaylor Blau1-0/+8
When writing a new multi-pack index, Git tries to reuse as much of the data from an existing MIDX as possible, like object offsets. This is done to avoid re-opening a bunch of *.idx files unnecessarily, but can lead to problems if the data we are reusing is corrupt. That's because we'll blindly reuse data from an existing MIDX without checking its trailing checksum for validity. So if there is memory corruption while writing a MIDX, or disk corruption in the intervening period between writing and reuse, we'll blindly propagate those bad values forward. Suppose we experience a memory corruption while writing a MIDX such that we write an incorrect object offset (or alternatively, the disk corrupts the data after being written, but before being reused). Then when we go to write a new MIDX, we'll reuse the bad object offset without checking its validity. This means that the MIDX we just wrote is broken, but its trailing checksum is in-tact, since we never bothered to look at the values before writing. In the above, a "git multi-pack-index verify" would have caught the problem before writing, but writing a new MIDX wouldn't have noticed anything wrong, blindly carrying forward the corrupt offset. Individual pack indexes check their validity by verifying the crc32 attached to each entry when carrying data forward during a repack. We could solve this problem for MIDXs in the same way, but individual crc32's don't make much sense, since their entries are so small. Likewise, checking the whole file on every read may be prohibitively expensive if a repository has a lot of objects, packs, or both. But we can check the trailing checksum when reusing an existing MIDX when writing a new one. And a corrupt MIDX need not stop us from writing a new one, since we can just avoid reusing the existing one at all and pretend as if we are writing a new MIDX from scratch. Suggested-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28t: fix typos in test messagesAndrei Rybak2-2/+2
Both in t4258 and in t9001, the code of the tests following shows the proper name for the configuration variables. So use the correct names in the test messages as well. Signed-off-by: Andrei Rybak <rybak.a.v@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28promisor-remote: teach lazy-fetch in any repoJonathan Tan4-0/+68
This is one step towards supporting partial clone submodules. Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper. Besides that, there is some code that uses the wrapper functions like has_promisor_remote(). Those will need to be checked to see if they could support the non-wrapper functions instead (and thus support any repository, not just the_repository). Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28show-branch tests: add missing testsÆvar Arnfjörð Bjarmason1-0/+61
Add missing tests for --remotes, --list and --merge-base. These are not exhaustive, but better than the nothing we have now. There were some tests for this command added in f76412ed6db ([PATCH] Add 'git show-branch'., 2005-08-21) has never been properly tested, namely for the --all option in t6432-merge-recursive-space-options.sh, and some of --merge-base and --independent in t6010-merge-base.sh. This fixes a few more blind spots, but there's still a lot of behavior that's not tested for. These new tests show the odd (and possibly unintentional) behavior of --merge-base with one argument, and how its output is the same as "git merge-base" with N bases in this particular case. See the test added in f621a8454d1 (git-merge-base/git-show-branch --merge-base: Documentation and test, 2009-08-05) for a case where the two aren't the same. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28show-branch: don't <COLOR></RESET> for space charactersÆvar Arnfjörð Bjarmason1-0/+30
Change the colored output introduced in ab07ba2a24 (show-branch: color the commit status signs, 2009-04-22) to not color and reset each individual space character we use for padding. The intent is to color just the "!", "+" etc. characters. This makes the output easier to test, so let's do that now. The test would be much more verbose without a color/reset for each space character. Since the coloring cycles through colors we previously had a "rainbow of space characters". In theory this breaks things for anyone who's relying on the exact colored output of show-branch, in practice I'd think anyone parsing it isn't actively turning on the colored output. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28mktag tests: test fast-exportÆvar Arnfjörð Bjarmason1-0/+8
Pass the bad tags we've created in the mktag tests through fast-export, it will die on the bad object or ref, let's make sure that happens. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28mktag tests: test for-each-refÆvar Arnfjörð Bjarmason1-0/+13
Add a "for-each-ref" for all the mktag tests. This test would have caught the segfault which was fixed in c6854508808 (ref-filter: fix NULL check for parse object failure, 2021-04-01). Let's make sure we test that code more exhaustively. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28mktag tests: test update-ref and reachable fsckÆvar Arnfjörð Bjarmason1-1/+20
Extend the mktag tests to pass the created bad tag through update-ref and fsck. The reason for passing it through update-ref is to guard against it having a segfault as for-each-ref did before c6854508808 (ref-filter: fix NULL check for parse object failure, 2021-04-01). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28mktag tests: test hash-object --literally and unreachable fsckÆvar Arnfjörð Bjarmason1-8/+38
Extend the mktag tests to pass the tag we've created through both hash-object --literally and fsck. This checks that fsck itself will not complain about certain invalid content if a reachable tip isn't involved. Due to how fsck works and walks the graph the failure will be different if the object is reachable, so we might succeed before we've created the ref. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28merge-ort: add prefetching for content mergesElijah Newren1-1/+1
Commit 7fbbcb21b1 ("diff: batch fetching of missing blobs", 2019-04-05) introduced batching of fetching missing blobs, so that the diff machinery would have one fetch subprocess grab N blobs instead of N processes each grabbing 1. However, the diff machinery is not the only thing in a merge that needs to work on blobs. The 3-way content merges need them as well. Rather than download all the blobs 1 at a time, prefetch all the blobs needed for regular content merges. This does not cover all possible paths in merge-ort that might need to download blobs. Others include: - The blob_unchanged() calls to avoid modify/delete conflicts (when blob renormalization results in an "unchanged" file) - Preliminary content merges needed for rename/add and rename/rename(2to1) style conflicts. (Both of these types of conflicts can result in nested conflict markers from the need to do two levels of content merging; the first happens before our new prefetch_for_content_merges() function.) The first of these wouldn't be an extreme amount of work to support, and even the second could be theoretically supported in batching, but all of these cases seem unusual to me, and this is a minor performance optimization anyway; in the worst case we only get some of the fetches batched and have a few additional one-off fetches. So for now, just handle the regular 3-way content merges in our prefetching. For the testcase from the previous commit, the number of downloaded objects remains at 63, but this drops the number of fetches needed from 32 down to 20, a sizeable reduction. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28diffcore-rename: use a different prefetch for basename comparisonsElijah Newren1-2/+2
merge-ort was designed to minimize the amount of data needed and used, and several changes were made to diffcore-rename to take advantage of extra metadata to enable this data minimization (particularly the relevant_sources variable for skipping "irrelevant" renames). This effort obviously succeeded in drastically reducing computation times, but should also theoretically allow partial clones to download much less information. Previously, though, the "prefetch" command used in diffcore-rename had never been modified and downloaded many blobs that were unnecessary for merge-ort. This commit corrects that. When doing basename comparisons, we want to fetch only the objects that will be used for basename comparisons. If after basename fetching this leaves us with no more relevant sources (or no more destinations), then we won't need to do the full inexact rename detection and can skip downloading additional source and destination files. Even if we have to do that later full inexact rename detection, irrelevant sources are culled after basename matching and before the full inexact rename detection, so we can still avoid downloading the blobs for irrelevant sources. Rename prefetch() to inexact_prefetch(), and introduce a new basename_prefetch() to take advantage of this. If we modify the testcase from commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2021-01-23) to pass --sparse --filter=blob:none to the clone command, and use the new trace2 "fetch_count" output from a few commits ago to track both the number of fetch subcommands invoked and the number of objects fetched across all those fetches, then for the mega-renames testcase we observe the following: BEFORE this commit, rebasing 35 patches: strategy # of fetches total # of objects fetched --------- ------------ -------------------------- recursive 62 11423 ort 30 11391 AFTER this commit, rebasing the same 35 patches: ort 32 63 This means that the new code only needs to download less than 2 blobs per patch being rebased. That is especially interesting given that the repository at the start only had approximately half a dozen TOTAL blobs downloaded to start with (because the default sparse-checkout of just the toplevel directory was in use). So, for this particular linux kernel testcase that involved ~26,000 renames on the upstream side (drivers/ -> pilots/) across which 35 patches were being rebased, this change reduces the number of blobs that need to be downloaded by a factor of ~180. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28t6421: add tests checking for excessive object downloads during mergeElijah Newren1-0/+440
Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-19t: use portable wrapper for readlink(1)Jeff King3-3/+9
Not all systems have a readlink program available for use by the shell. This causes t3210 to fail on at least AIX. Let's provide a perl one-liner to do the same thing, and use it there. I also updated calls in t9802. Nobody reported failure there, but it's the same issue. Presumably nobody actually tests with p4 on AIX in the first place (if it is even available there). I left the use of readlink in the "--valgrind" setup in test-lib.sh, as valgrind isn't available on exotic platforms anyway (and I didn't want to increase dependencies between test-lib.sh and test-lib-functions.sh). There's one other curious case. Commit d2addc3b96 (t7800: readlink may not be available, 2016-05-31) fixed a similar case. We can't use our wrapper function there, though, as it's inside a sub-script triggered by Git. It uses a slightly different technique ("ls" piped to "sed"). I chose not to use that here as it gives confusing "ls -l" output if the file is unexpectedly not a symlink (which is OK for its limited use, but potentially confusing for general use within the test suite). The perl version emits the empty string. Reported-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-17test: refactor to use "get_abbrev_oid" to get abbrev oidJiang Xin15-52/+72
Add new function "get_abbrev_oid" to get abbrev object ID. This function has a default value which helps to prepare a nonempty replace pattern for sed command. An empty replace pattern may cause sed fail to allocate memory. Refactor function "make_user_friendly_and_stable_output" to use "get_abbrev_oid" to get abbrev object ID. Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-17test: refactor to use "test_commit" to create commitsJiang Xin2-38/+12
Refactor function "create_commits_in" to use "test_commit" to create commit. Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-17test: compare raw output, not mangle tabs and spacesJiang Xin31-966/+972
Before comparing with the expect file, we used to call function "make_user_friendly_and_stable_output" to filter out trailing spaces in output. Ævar recommends using pattern "s/Z$//" to prepare expect file, and then compare it with raw output. Since we have fixed the issue of occasionally missing the clear-to-eol suffix when displaying sideband #2 messages, it is safe and stable to test against raw output. Suggested-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-17t6020: fix incompatible parameter expansionJiang Xin1-19/+31
Ævar reported that the function `make_user_friendly_and_stable_output()` failed on a i386 box (gcc45) in the gcc farm boxes with error: sed: couldn't re-allocate memory It turns out that older versions of bash (4.3) or dash (0.5.7) cannot evaluate expression like `${A%${A#???????}}` used to get the leading 7 characters of variable A. Replace the incompatible parameter expansion so that t6020 works on older version of bash or dash. Reported-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-16*: fix typosDerrick Stolee1-1/+0
These typos were found while searching the codebase for gendered pronouns. In the case of t9300-fast-import.sh, remove a confusing comment that is unnecessary to the understanding of the test. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-16gc tests: add a test for the "pre-auto-gc" hookÆvar Arnfjörð Bjarmason1-0/+46
Add a missing test for the behavior of the pre-auto-gc hook added in 0b85d92661e (Documentation/hooks: add pre-auto-gc hook, 2008-04-02). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-16pre-commit hook tests: don't leave "actual" nonexisting on failureÆvar Arnfjörð Bjarmason1-0/+1
Start by creating an "actual" file in a core.hooksPath test that has the hook echoing to the "actual" file. We later test_cmp that file to see what hooks were run. If we fail to run our hook(s) we'll have an empty list of hooks for the test_cmp instead of a nonexisting file. For the logic of this test that makes more sense. See 867ad08a261 (hooks: allow customizing where the hook directory is, 2016-05-04) for the commit that added these tests. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-15show-branch tests: modernize test codeÆvar Arnfjörð Bjarmason1-52/+40
Modernize test code added in ce567d1867a (Add test to show that show-branch misses out the 8th column, 2008-07-23) and 11ee57bc4c4 (sort_in_topological_order(): avoid setting a commit flag, 2008-07-23) to use test helpers. I'm renaming "out" to "actual" for consistency with other tests, and introducing a "branches.sorted" file in the setup, to make it clear that it's important that the list be sorted in this particular way. The "show-branch" output is indented with spaces, which would cause complaints under "git show --check" with an indented here-doc block. Let's prefix the lines with "> " to work around that, and to make it clear that the leading whitespace is important. We can also get rid of the hardcoding of "main" added here in 334afbc76fb (tests: mark tests relying on the current default for `init.defaultBranch`, 2020-11-18). For this test we're setting up an "initial" commit anyway, and now that we've moved over to test_commit we can reference that instead. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-15show-branch tests: rename the one "show-branch" test fileÆvar Arnfjörð Bjarmason1-1/+1
Rename the only *show-branch* test file to indicate that more tests belong it in than just the one-off octopus test it now contains. The test was initially added in ce567d1867a (Add test to show that show-branch misses out the 8th column, 2008-07-23) and 11ee57bc4c4 (sort_in_topological_order(): avoid setting a commit flag, 2008-07-23). Those two add almost the same content, one with a test_expect_success and the other a test_expect_failure (a bug being tested for was fixed on one of the branches). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-15mktag tests: invert --no-strict testÆvar Arnfjörð Bjarmason1-0/+2
Change the mktag --no-strict test to actually test success under --no-strict, that test was added in 06ce79152be (mktag: add a --[no-]strict option, 2021-01-06). It doesn't make sense to check that we have the same failure except when we want --no-strict, by doing that we're assuming that the behavior will be different under --no-strict, bun nothing was testing for that. We should instead assert that --strict is the same as --no-strict, except in the cases where we've declared that it's not. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-15mktag tests: parse out options in helperÆvar Arnfjörð Bjarmason1-12/+31
Change check_verify_failure() helper to parse out options from $@. This makes it easier to add new options in the future. See 06ce79152be (mktag: add a --[no-]strict option, 2021-01-06) for the initial implementation. Let's also replace "" quotes with '' for the test body, the varables we need are eval'd into the body, so there's no need for the quoting confusion. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-14Merge branch 'ab/test-lib-updates'Junio C Hamano14-119/+89
Test clean-up. * ab/test-lib-updates: test-lib: split up and deprecate test_create_repo() test-lib: do not show advice about init.defaultBranch under --verbose test-lib: reformat argument list in test_create_repo() submodule tests: use symbolic-ref --short to discover branch name test-lib functions: add --printf option to test_commit describe tests: convert setup to use test_commit test-lib functions: add an --annotated option to "test_commit" test-lib-functions: document test_commit --no-tag test-lib-functions: reword "test_commit --append" docs test-lib tests: remove dead GIT_TEST_FRAMEWORK_SELFTEST variable test-lib: bring $remove_trash out of retirement
2021-06-14Merge branch 'dd/honor-users-tar-in-tests'Junio C Hamano2-4/+4
Test portability fix. * dd/honor-users-tar-in-tests: t: use configured TAR instead of tar
2021-06-14Merge branch 'so/log-m-implies-p'Junio C Hamano5-1/+163
The "-m" option in "git log -m" that does not specify which format, if any, of diff is desired did not have any visible effect; it now implies some form of diff (by default "--patch") is produced. * so/log-m-implies-p: diff-merges: let "-m" imply "-p" diff-merges: rename "combined_imply_patch" to "merges_imply_patch" stash list: stop passing "-m" to "git log" git-svn: stop passing "-m" to "git rev-list" diff-merges: move specific diff-index "-m" handling to diff-index t4013: test "git diff-index -m" t4013: test "git diff-tree -m" t4013: test "git log -m --stat" t4013: test "git log -m --raw" t4013: test that "-m" alone has no effect in "git log"
2021-06-14Merge branch 'en/ort-perf-batch-11'Junio C Hamano3-20/+792
Optimize out repeated rename detection in a sequence of mergy operations. * en/ort-perf-batch-11: merge-ort, diffcore-rename: employ cached renames when possible merge-ort: handle interactions of caching and rename/rename(1to1) cases merge-ort: add helper functions for using cached renames merge-ort: preserve cached renames for the appropriate side merge-ort: avoid accidental API mis-use merge-ort: add code to check for whether cached renames can be reused merge-ort: populate caches of rename detection results merge-ort: add data structures for in-memory caching of rename detection t6429: testcases for remembering renames fast-rebase: write conflict state to working tree, index, and HEAD fast-rebase: change assert() to BUG() Documentation/technical: describe remembering renames optimization t6423: rename file within directory that other side renamed
2021-06-14Merge branch 'jk/clone-clean-upon-transport-error'Junio C Hamano1-0/+7
Recent "git clone" left a temporary directory behind when the transport layer returned an failure. * jk/clone-clean-upon-transport-error: clone: clean up directory after transport_fetch_refs() failure
2021-06-14Merge branch 'ga/send-email-sendmail-cmd'Junio C Hamano1-0/+31
"git send-email" learned the "--sendmail-cmd" command line option and the "sendemail.sendmailCmd" configuration variable, which is a more sensible approach than the current way of repurposing the "smtp-server" that is meant to name the server to instead name the command to talk to the server. * ga/send-email-sendmail-cmd: git-send-email: add option to specify sendmail command
2021-06-14*: fix typos which duplicate a wordAndrei Rybak2-2/+2
Fix typos in documentation, code comments, and RelNotes which repeat various words. In trivial cases, just delete the duplicated word and rewrap text, if needed. Reword the affected sentence in Documentation/RelNotes/1.8.4.txt for it to make sense. Signed-off-by: Andrei Rybak <rybak.a.v@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-11ll_union_merge(): pass name labels to ll_xdl_merge()Jeff King1-1/+2
Since cd1d61c44f (make union merge an xdl merge favor, 2010-03-01), we pass NULL to ll_xdl_merge() for the "name" labels of the ancestor, ours and theirs buffers. We usually use these for annotating conflict markers left in a file. For a union merge, these shouldn't matter; the point of it is that we'd never leave conflict markers in the first place. But there is one code path where we may dereference them: if the file contents appear to be binary, ll_binary_merge() will give up and pass them to warning() to generate a message for the user (that was true even when cd1d61c44f was written, though the warning was in ll_xdl_merge() back then). That can result in a segfault, though on many systems (including glibc), the printf routines will helpfully just say "(null)" instead. We can extend our binary-union test in t6406 to check stderr, which catches the problem on all systems. This also fixes a warning from "gcc -O3". Unlike lower optimization levels, it inlines enough to see that the NULL can make it to warning() and complains: In function ‘ll_binary_merge’, inlined from ‘ll_xdl_merge’ at ll-merge.c:115:10, inlined from ‘ll_union_merge’ at ll-merge.c:151:9: ll-merge.c:74:4: warning: ‘%s’ directive argument is null [-Wformat-overflow=] 74 | warning("Cannot merge binary files: %s (%s vs. %s)", | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 75 | path, name1, name2); | ~~~~~~~~~~~~~~~~~~~ Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-11ll_binary_merge(): handle XDL_MERGE_FAVOR_UNIONJeff King1-0/+17
Prior to commit a944af1d86 (merge: teach -Xours/-Xtheirs to binary ll-merge driver, 2012-09-08), we always reported a conflict from ll_binary_merge() by returning "1" (in the xdl_merge and ll_merge code, this value is the number of conflict hunks). After that commit, we report zero conflicts if the "variant" flag is set, under the assumption that it is one of XDL_MERGE_FAVOR_OURS or XDL_MERGE_FAVOR_THEIRS. But this gets confused by XDL_MERGE_FAVOR_UNION. We do not know how to do a binary union merge, but erroneously report no conflicts anyway (and just blindly use the "ours" content as the result). Let's tighten our check to just the cases that a944af1d86 meant to cover. This fixes the union case (which existed already back when that commit was made), as well as future-proofing us against any other variants that get added later. Note that you can't trigger this from "git merge-file --union", as that bails on binary files before even calling into the ll-merge machinery. The test here uses the "union" merge attribute, which does erroneously report a successful merge. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-09Fix various issues found in commentsElijah Newren1-1/+1
A random hodge-podge of incorrect or out-of-date comments that I found: * t6423 had a comment that has referred to the wrong test for years; fix it to refer to the right one. * diffcore-rename had a FIXME comment meant to remind myself to investigate if I could make another code change. I later investigated and removed the FIXME, but while cherry-picking the patch to submit upstream I missed the later update. Remove the comment now. * merge-ort had the early part of a comment for a function; I had meant to include the more involved description when I updated the function. Update the comment now. Signed-off-by: Elijah Newren <newren@gmail.com> Reviewed-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-09builtins + test helpers: use return instead of exit() in cmd_*Ævar Arnfjörð Bjarmason4-4/+4
Change various cmd_* functions that claim to return an "int" to use "return" instead of exit() to indicate an exit code. These were not marked with NORETURN, and by directly exit()-ing we'll skip the cleanup git.c would otherwise do (e.g. closing fd's, erroring if we can't). See run_builtin() in git.c. In the case of shell.c and sh-i18n--envsubst.c this was the result of an incomplete migration to using a cmd_main() in 3f2e2297b9 (add an extra level of indirection to main(), 2016-07-01). This was spotted by SunCC 12.5 on Solaris 10 (gcc210 on the gccfarm). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-08t: use user-specified utf-8 locale for testing svnĐoàn Trần Công Danh4-22/+24
In some test-cases, UTF-8 locale is required. To find such locale, we're using the first available UTF-8 locale that returned by "locale -a". However, the locale(1) utility is unavailable on some systems, e.g. Linux with musl libc. However, without "locale -a", we can't guess provided UTF-8 locale. Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for linux-musl in our CI system. Rename t/lib-git-svn.sh:prepare_a_utf8_locale to prepare_utf8_locale, since we no longer prepare the variable named "a_utf8_locale", but set up a fallback value for GIT_TEST_UTF8_LOCALE instead. The fallback will be LC_ALL, LANG environment variable, or the first UTF-8 locale from output of "locale -a", in that order. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-08t: fix whitespace around &&Andrei Rybak5-5/+5
Add missing spaces before '&&' and switch tabs around '&&' to spaces. These issues were found using `git grep '[^ ]&&$'` and `git grep -P '&&\t'`. Signed-off-by: Andrei Rybak <rybak.a.v@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-06Merge branch 'rs/parallel-checkout-test-fix'Junio C Hamano1-1/+1
Test fix. * rs/parallel-checkout-test-fix: parallel-checkout: avoid dash local bug in tests
2021-06-06parallel-checkout: avoid dash local bug in testsRené Scharfe1-1/+1
Dash bug https://bugs.launchpad.net/ubuntu/+source/dash/+bug/139097 lets the shell erroneously perform field splitting on the expansion of a command substitution during declaration of a local variable. It causes the parallel-checkout tests to fail e.g. when running them with /bin/dash on MacOS 11.4, where they error out like this: ./t2080-parallel-checkout-basics.sh: 33: local: 0: bad variable name That's because the output of wc -l contains leading spaces and the returned number of lines is treated as another variable to declare, i.e. as in "local workers= 0". Work around it by enclosing the command substitution in quotes. Helped-by: Matheus Tavares Bernardino <matheus.bernardino@usp.br> Helped-by: SZEDER Gábor <szeder.dev@gmail.com> Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-04cat-file: handle trivial --batch format with --batch-all-objectsZheNing Hu1-0/+22
The --batch code to print an object assumes we found out the type of the object from calling oid_object_info_extended(). This is true for the default format, but even in a custom format, we manually modify the object_info struct to ask for the type. This assumption was broken by 845de33a5b (cat-file: avoid noop calls to sha1_object_info_extended, 2016-05-18). That commit skips the call to oid_object_info_extended() entirely when --batch-all-objects is in use, and the custom format does not include any placeholders that require calling it. Or when the custom format only include placeholders like %(objectname) or %(rest), oid_object_info_extended() will not get the type of the object. This results in an error when we try to confirm that the type didn't change: $ git cat-file --batch=batman --batch-all-objects batman fatal: object 000023961a0c02d6e21dc51ea3484ff71abf1c74 changed type!? and also has other subtle effects (e.g., we'd fail to stream a blob, since we don't realize it's a blob in the first place). We can fix this by flipping the order of the setup. The check for "do we need to get the object info" must come _after_ we've decided whether we need to look up the type. Helped-by: Jeff King <peff@peff.net> Signed-off-by: ZheNing Hu <adlternative@gmail.com> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t1415: set REFFILES for test specific to storage formatHan-Wen Nienhuys1-1/+4
Packing refs (and therefore checking that certain refs are not packed) is a property of the packed/loose ref storage. Add a comment to explain what the test checks. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t4202: mark bogus head hash test with REFFILESHan-Wen Nienhuys1-3/+9
In reftable, hashes are correctly formed by design. Split off test for git-log in empty repo. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t7003: check reflog existence only for REFFILESHan-Wen Nienhuys1-1/+4
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t7900: stop checking for loose refsHan-Wen Nienhuys1-2/+0
Given that git-maintenance simply calls out git-pack-refs, it seems superfluous to test the functionality of pack-refs itself, as that is covered by t3210-pack-refs.sh. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t1404: mark tests that muck with .git directly as REFFILES.Han-Wen Nienhuys1-15/+15
The packed/loose ref storage is an overlay combination of packed-refs (refs and tags in a single file) and one-file-per-ref. This creates all kinds of edge cases related to directory/file conflicts, (non-)empty directories, and the locking scheme, none of which applies to reftable. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t2017: mark --orphan/logAllRefUpdates=false test as REFFILESHan-Wen Nienhuys1-1/+1
In reftable, there is no notion of a per-ref 'existence' of a reflog. Each reflog entry has its own key, so it is not possible to distinguish between {reflog doesn't exist,reflog exists but is empty}. This makes the logic in log_ref_setup() (file refs/files-backend.c), which depends on the existence of the reflog file infeasible. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t1414: mark corruption test with REFFILESHan-Wen Nienhuys1-1/+3
The test checks what happens if reflog and ref database disagree on the state of the latest commit. This seems to require accessing reflog storage directly. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t1407: require REFFILES for for_each_reflog testHan-Wen Nienhuys1-1/+8
Add extensive comment why this test needs a REFFILES annotation. I tried forcing universal reflog creation with core.logAllRefUpdates=true, but that apparently also doesn't cause reflogs to be created for pseudorefs Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02test-lib: provide test prereq REFFILESHan-Wen Nienhuys2-0/+8
REFFILES can be used to mark tests that are specific to the packed/loose ref storage format and its limitations. Marking such tests is a preparation for introducing the reftable storage backend. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t5304: use "reflog expire --all" to clear the reflogHan-Wen Nienhuys1-5/+4
This test checks that unreachable objects are really removed. For the test to work, it has to ensure that no reflog retain any reachable objects. Previously, it did this by manipulating the file system to remove reflog in the first test, and relying on git not updating the reflog if the relevant logfile doesn't exist in follow-up tests. Now, explicitly clear the reflog using 'reflog expire'. This reduces the dependency between test functions. It also is more amenable to use with reftable, which has no concept of (non)-existence of a reflog Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t5304: restyle: trim empty lines, drop ':' before >Han-Wen Nienhuys1-54/+20
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t7003: use rev-parse rather than FS inspectionHan-Wen Nienhuys1-1/+1
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t5000: inspect HEAD using git-rev-parseHan-Wen Nienhuys1-4/+5
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t5000: reformat indentation to the latest fashionHan-Wen Nienhuys1-51/+62
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t1301: fix typo in error messageHan-Wen Nienhuys1-1/+1
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t1413: use tar to save and restore entire .git directoryHan-Wen Nienhuys1-3/+2
This makes the test independent of the particulars of the storage formats. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t1401-symbolic-ref: avoid direct filesystem accessHan-Wen Nienhuys1-13/+9
Use symbolic-ref and rev-parse to inspect refs. Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t1401: use tar to snapshot and restore repo stateHan-Wen Nienhuys1-1/+8
This is agnostic to the ref storage format Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t5601: read HEAD using rev-parseHan-Wen Nienhuys1-1/+2
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t9300: check ref existence using test-helper rather than a file system checkHan-Wen Nienhuys1-1/+1
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t/helper/ref-store: initialize oid in resolve-refHan-Wen Nienhuys1-1/+1
This will print $ZERO_OID when asking for a non-existent ref from the test-helper. Since resolve-ref provides direct access to refs_resolve_ref_unsafe(), it provides a reliable mechanism for accessing REFNAME, while avoiding the implicit resolution to refs/heads/REFNAME. Reviewed-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-02t4202: split testcase for invalid HEAD symref and HEAD hashHan-Wen Nienhuys1-3/+7
Reftable will prohibit invalid hashes at the storage level, but git-symbolic-ref can still create branches ending in ".lock". Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-28send-email tests: test for boolean variables without a valueÆvar Arnfjörð Bjarmason1-0/+29
The Git.pm code does its own Perl-ifying of boolean variables, let's ensure that empty values = true for boolean variables, as in the C code. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-27Merge branch 'mt/t2080-cp-symlink-fix'Junio C Hamano1-1/+1
Test portability fix. * mt/t2080-cp-symlink-fix: t2080: fix cp invocation to copy symlinks instead of following them
2021-05-27Merge branch 'ab/send-email-inline-hooks-path'Junio C Hamano1-3/+4
Code simplification. * ab/send-email-inline-hooks-path: send-email: move "hooks_path" invocation to git-send-email.perl send-email: don't needlessly abs_path() the core.hooksPath
2021-05-27Merge branch 'ds/t1092-fix-flake-from-progress'Junio C Hamano1-3/+3
Workaround flaky tests introduced recently. * ds/t1092-fix-flake-from-progress: t1092: revert the "-1" hack for emulating "no progress meter" t1092: use GIT_PROGRESS_DELAY for consistent results
2021-05-27t2080: fix cp invocation to copy symlinks instead of following themMatheus Tavares1-1/+1
t2080 makes a few copies of a test repository and later performs a branch switch on each one of the copies to verify that parallel checkout and sequential checkout produce the same results. However, the repository is copied with `cp -R` which, on some systems, defaults to following symlinks on the directory hierarchy and copying their target files instead of copying the symlinks themselves. AIX is one example of system where this happens. Because the symlinks are not preserved, the copied repositories have paths that do not match what is in the index, causing git to abort the checkout operation that we want to test. This makes the test fail on these systems. Fix this by copying the repository with the POSIX flag '-P', which forces cp to copy the symlinks instead of following them. Note that we already use this flag for other cp invocations in our test suite (see t7001). With this change, t2080 now passes on AIX. Reported-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-27send-email: don't needlessly abs_path() the core.hooksPathÆvar Arnfjörð Bjarmason1-3/+4
In c8243933c74 (git-send-email: Respect core.hooksPath setting, 2021-03-23) we started supporting core.hooksPath in "send-email". It's been reported that on Windows[1] doing this by calling abs_path() results in different canonicalizations of the absolute path. This wasn't an issue in c8243933c74 itself, but was revealed by my ea7811b37e0 (git-send-email: improve --validate error output, 2021-04-06) when we started emitting the path to the hook, which was previously only internal to git-send-email.perl. The just-landed 53753a37d09 (t9001-send-email.sh: fix expected absolute paths on Windows, 2021-05-24) narrowly fixed this issue, but I believe we can do better here. We should not be relying on whatever changes Perl's abs_path() makes to the path "rev-parse --git-path hooks" hands to us. Let's instead trust it, and hand it to Perl's system() in git-send-email.perl. It will handle either a relative or absolute path. So let's revert most of 53753a37d09 and just have "hooks_path" return what we get from "rev-parse" directly without modification. This has the added benefit of making the error message friendlier in the common case, we'll no longer print an absolute path for repository-local hook errors. 1. http://lore.kernel.org/git/bb30fe2b-cd75-4782-24a6-08bb002a0367@kdbg.org Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-26t1092: revert the "-1" hack for emulating "no progress meter"Junio C Hamano1-3/+3
This looked like a good idea, but it seems to break tests on 32-bit builds rather badly. Revert to just use "100 thousands must be big enough" for now. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-25Merge branch 'mt/init-template-userpath-fix'Junio C Hamano1-8/+20
Regression fix. * mt/init-template-userpath-fix: init: fix bug regarding ~/ expansion in init.templateDir
2021-05-25Merge branch 'jt/send-email-validate-errors-fix'Junio C Hamano1-4/+3
Fix a test breakage. * jt/send-email-validate-errors-fix: t9001-send-email.sh: fix expected absolute paths on Windows
2021-05-25Merge branch 'ab/send-email-validate-errors-fix'Junio C Hamano1-2/+21
* ab/send-email-validate-errors-fix: send-email: fix missing error message regression
2021-05-25t1092: use GIT_PROGRESS_DELAY for consistent resultsDerrick Stolee1-3/+3
The t1092-sparse-checkout-compatibility.sh tests compare the stdout and stderr for several Git commands across both full checkouts, sparse checkouts with a full index, and sparse checkouts with a sparse index. Since these are direct comparisons, sometimes a progress indicator can flush at unpredictable points, especially on slower machines. This causes the tests to be flaky. One standard way to avoid this is to add GIT_PROGRESS_DELAY=0 to the Git commands that are run, as this will force every progress indicator created with start_progress_delay() to be created immediately. However, there are some progress indicators that are created in the case of a full index that are not created with a sparse index. Moreover, their values may be different as those indexes have a different number of entries. Instead, use GIT_PROGRESS_DELAY=-1 (which will turn into UINT_MAX) to ensure that any reasonable machine running these tests would never display delayed progress indicators. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-25init: fix bug regarding ~/ expansion in init.templateDirMatheus Tavares1-8/+20
We used to read the init.templateDir setting at builtin/init-db.c using a git_config() callback that, in turn, called git_config_pathname(). To simplify the config reading logic at this file and plug a memory leak, this was replaced by a direct call to git_config_get_value() at e4de4502e6 ("init: remove git_init_db_config() while fixing leaks", 2021-03-14). However, this function doesn't provide path expanding semantics, like git_config_pathname() does, so paths with '~/' and '~user/' are treated literally. This makes 'git init' fail to handle init.templateDir paths using these constructs: $ git config init.templateDir '~/templates_dir' $ git init 'warning: templates not found in ~/templates_dir' Replace the git_config_get_value() call by git_config_get_pathname(), which does the '~/' and '~user/' expansions. Also add a regression test. Note that unlike git_config_get_value(), the config cache does not own the memory for the path returned by git_config_get_pathname(), so we must free() it. Reported on IRC by rkta. Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>