aboutsummaryrefslogtreecommitdiffstats
path: root/t
AgeCommit message (Collapse)AuthorFilesLines
2021-07-02perf: fix when running with TEST_OUTPUT_DIRECTORYPatrick Steinhardt3-13/+24
When the TEST_OUTPUT_DIRECTORY is defined, then all test data will be written in that directory instead of the default directory located in "t/". While this works as expected for our normal tests, performance tests fail to locate and aggregate performance data because they don't know to handle TEST_OUTPUT_DIRECTORY correctly and always look at the default location. Fix the issue by adding a `--results-dir` parameter to "aggregate.perl" which identifies the directory where results are and by making the "run" script awake of the TEST_OUTPUT_DIRECTORY variable. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-14Merge branch 'ab/test-lib-updates'Junio C Hamano14-119/+89
Test clean-up. * ab/test-lib-updates: test-lib: split up and deprecate test_create_repo() test-lib: do not show advice about init.defaultBranch under --verbose test-lib: reformat argument list in test_create_repo() submodule tests: use symbolic-ref --short to discover branch name test-lib functions: add --printf option to test_commit describe tests: convert setup to use test_commit test-lib functions: add an --annotated option to "test_commit" test-lib-functions: document test_commit --no-tag test-lib-functions: reword "test_commit --append" docs test-lib tests: remove dead GIT_TEST_FRAMEWORK_SELFTEST variable test-lib: bring $remove_trash out of retirement
2021-06-14Merge branch 'dd/honor-users-tar-in-tests'Junio C Hamano2-4/+4
Test portability fix. * dd/honor-users-tar-in-tests: t: use configured TAR instead of tar
2021-06-14Merge branch 'so/log-m-implies-p'Junio C Hamano5-1/+163
The "-m" option in "git log -m" that does not specify which format, if any, of diff is desired did not have any visible effect; it now implies some form of diff (by default "--patch") is produced. * so/log-m-implies-p: diff-merges: let "-m" imply "-p" diff-merges: rename "combined_imply_patch" to "merges_imply_patch" stash list: stop passing "-m" to "git log" git-svn: stop passing "-m" to "git rev-list" diff-merges: move specific diff-index "-m" handling to diff-index t4013: test "git diff-index -m" t4013: test "git diff-tree -m" t4013: test "git log -m --stat" t4013: test "git log -m --raw" t4013: test that "-m" alone has no effect in "git log"
2021-06-14Merge branch 'en/ort-perf-batch-11'Junio C Hamano3-20/+792
Optimize out repeated rename detection in a sequence of mergy operations. * en/ort-perf-batch-11: merge-ort, diffcore-rename: employ cached renames when possible merge-ort: handle interactions of caching and rename/rename(1to1) cases merge-ort: add helper functions for using cached renames merge-ort: preserve cached renames for the appropriate side merge-ort: avoid accidental API mis-use merge-ort: add code to check for whether cached renames can be reused merge-ort: populate caches of rename detection results merge-ort: add data structures for in-memory caching of rename detection t6429: testcases for remembering renames fast-rebase: write conflict state to working tree, index, and HEAD fast-rebase: change assert() to BUG() Documentation/technical: describe remembering renames optimization t6423: rename file within directory that other side renamed
2021-06-14Merge branch 'jk/clone-clean-upon-transport-error'Junio C Hamano1-0/+7
Recent "git clone" left a temporary directory behind when the transport layer returned an failure. * jk/clone-clean-upon-transport-error: clone: clean up directory after transport_fetch_refs() failure
2021-06-14Merge branch 'ga/send-email-sendmail-cmd'Junio C Hamano1-0/+31
"git send-email" learned the "--sendmail-cmd" command line option and the "sendemail.sendmailCmd" configuration variable, which is a more sensible approach than the current way of repurposing the "smtp-server" that is meant to name the server to instead name the command to talk to the server. * ga/send-email-sendmail-cmd: git-send-email: add option to specify sendmail command
2021-06-06Merge branch 'rs/parallel-checkout-test-fix'Junio C Hamano1-1/+1
Test fix. * rs/parallel-checkout-test-fix: parallel-checkout: avoid dash local bug in tests
2021-06-06parallel-checkout: avoid dash local bug in testsRené Scharfe1-1/+1
Dash bug https://bugs.launchpad.net/ubuntu/+source/dash/+bug/139097 lets the shell erroneously perform field splitting on the expansion of a command substitution during declaration of a local variable. It causes the parallel-checkout tests to fail e.g. when running them with /bin/dash on MacOS 11.4, where they error out like this: ./t2080-parallel-checkout-basics.sh: 33: local: 0: bad variable name That's because the output of wc -l contains leading spaces and the returned number of lines is treated as another variable to declare, i.e. as in "local workers= 0". Work around it by enclosing the command substitution in quotes. Helped-by: Matheus Tavares Bernardino <matheus.bernardino@usp.br> Helped-by: SZEDER Gábor <szeder.dev@gmail.com> Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-27Merge branch 'mt/t2080-cp-symlink-fix'Junio C Hamano1-1/+1
Test portability fix. * mt/t2080-cp-symlink-fix: t2080: fix cp invocation to copy symlinks instead of following them
2021-05-27Merge branch 'ab/send-email-inline-hooks-path'Junio C Hamano1-3/+4
Code simplification. * ab/send-email-inline-hooks-path: send-email: move "hooks_path" invocation to git-send-email.perl send-email: don't needlessly abs_path() the core.hooksPath
2021-05-27Merge branch 'ds/t1092-fix-flake-from-progress'Junio C Hamano1-3/+3
Workaround flaky tests introduced recently. * ds/t1092-fix-flake-from-progress: t1092: revert the "-1" hack for emulating "no progress meter" t1092: use GIT_PROGRESS_DELAY for consistent results
2021-05-27t2080: fix cp invocation to copy symlinks instead of following themMatheus Tavares1-1/+1
t2080 makes a few copies of a test repository and later performs a branch switch on each one of the copies to verify that parallel checkout and sequential checkout produce the same results. However, the repository is copied with `cp -R` which, on some systems, defaults to following symlinks on the directory hierarchy and copying their target files instead of copying the symlinks themselves. AIX is one example of system where this happens. Because the symlinks are not preserved, the copied repositories have paths that do not match what is in the index, causing git to abort the checkout operation that we want to test. This makes the test fail on these systems. Fix this by copying the repository with the POSIX flag '-P', which forces cp to copy the symlinks instead of following them. Note that we already use this flag for other cp invocations in our test suite (see t7001). With this change, t2080 now passes on AIX. Reported-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-27send-email: don't needlessly abs_path() the core.hooksPathÆvar Arnfjörð Bjarmason1-3/+4
In c8243933c74 (git-send-email: Respect core.hooksPath setting, 2021-03-23) we started supporting core.hooksPath in "send-email". It's been reported that on Windows[1] doing this by calling abs_path() results in different canonicalizations of the absolute path. This wasn't an issue in c8243933c74 itself, but was revealed by my ea7811b37e0 (git-send-email: improve --validate error output, 2021-04-06) when we started emitting the path to the hook, which was previously only internal to git-send-email.perl. The just-landed 53753a37d09 (t9001-send-email.sh: fix expected absolute paths on Windows, 2021-05-24) narrowly fixed this issue, but I believe we can do better here. We should not be relying on whatever changes Perl's abs_path() makes to the path "rev-parse --git-path hooks" hands to us. Let's instead trust it, and hand it to Perl's system() in git-send-email.perl. It will handle either a relative or absolute path. So let's revert most of 53753a37d09 and just have "hooks_path" return what we get from "rev-parse" directly without modification. This has the added benefit of making the error message friendlier in the common case, we'll no longer print an absolute path for repository-local hook errors. 1. http://lore.kernel.org/git/bb30fe2b-cd75-4782-24a6-08bb002a0367@kdbg.org Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-26t1092: revert the "-1" hack for emulating "no progress meter"Junio C Hamano1-3/+3
This looked like a good idea, but it seems to break tests on 32-bit builds rather badly. Revert to just use "100 thousands must be big enough" for now. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-25Merge branch 'mt/init-template-userpath-fix'Junio C Hamano1-8/+20
Regression fix. * mt/init-template-userpath-fix: init: fix bug regarding ~/ expansion in init.templateDir
2021-05-25Merge branch 'jt/send-email-validate-errors-fix'Junio C Hamano1-4/+3
Fix a test breakage. * jt/send-email-validate-errors-fix: t9001-send-email.sh: fix expected absolute paths on Windows
2021-05-25Merge branch 'ab/send-email-validate-errors-fix'Junio C Hamano1-2/+21
* ab/send-email-validate-errors-fix: send-email: fix missing error message regression
2021-05-25t1092: use GIT_PROGRESS_DELAY for consistent resultsDerrick Stolee1-3/+3
The t1092-sparse-checkout-compatibility.sh tests compare the stdout and stderr for several Git commands across both full checkouts, sparse checkouts with a full index, and sparse checkouts with a sparse index. Since these are direct comparisons, sometimes a progress indicator can flush at unpredictable points, especially on slower machines. This causes the tests to be flaky. One standard way to avoid this is to add GIT_PROGRESS_DELAY=0 to the Git commands that are run, as this will force every progress indicator created with start_progress_delay() to be created immediately. However, there are some progress indicators that are created in the case of a full index that are not created with a sparse index. Moreover, their values may be different as those indexes have a different number of entries. Instead, use GIT_PROGRESS_DELAY=-1 (which will turn into UINT_MAX) to ensure that any reasonable machine running these tests would never display delayed progress indicators. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-25init: fix bug regarding ~/ expansion in init.templateDirMatheus Tavares1-8/+20
We used to read the init.templateDir setting at builtin/init-db.c using a git_config() callback that, in turn, called git_config_pathname(). To simplify the config reading logic at this file and plug a memory leak, this was replaced by a direct call to git_config_get_value() at e4de4502e6 ("init: remove git_init_db_config() while fixing leaks", 2021-03-14). However, this function doesn't provide path expanding semantics, like git_config_pathname() does, so paths with '~/' and '~user/' are treated literally. This makes 'git init' fail to handle init.templateDir paths using these constructs: $ git config init.templateDir '~/templates_dir' $ git init 'warning: templates not found in ~/templates_dir' Replace the git_config_get_value() call by git_config_get_pathname(), which does the '~/' and '~user/' expansions. Also add a regression test. Note that unlike git_config_get_value(), the config cache does not own the memory for the path returned by git_config_get_pathname(), so we must free() it. Reported on IRC by rkta. Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-25send-email: fix missing error message regressionÆvar Arnfjörð Bjarmason1-2/+21
Fix a regression with the "the editor exited uncleanly, aborting everything" error message going missing after my d21616c0394 (git-send-email: refactor duplicate $? checks into a function, 2021-04-06). I introduced a $msg variable, but did not actually use it. This caused us to miss the optional error message supplied by the "do_edit" codepath. Fix that, and add tests to check that this works. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-25t9001-send-email.sh: fix expected absolute paths on WindowsJohannes Sixt1-4/+3
Git for Windows is a native Windows program that works with native absolute paths in the drive letter style C:\dir. The auxiliary infrastructure is based on MSYS2, which uses POSIX style /C/dir. When we test for output of absolute paths produced by git.exe, we usally have to expect C:\dir style paths. To produce such expected paths, we have to use $(pwd) in the test scripts; the alternative, $PWD, produces a POSIX style path. ($PWD is a shell variable, and the shell is bash, an MSYS2 program, and operates in the POSIX realm.) There are two recently added tests that were written to expect C:\dir paths. The output that is tested is produced by `git send-email`, but behind the scenes, this is a Perl script, which also works in the POSIX realm and produces /C/dir style output. In the first test case that is changed here, replace $(pwd) by $PWD so that the expected path is constructed using /C/dir style. The second test case sets core.hooksPath to an absolute path. Since the test script talks to native git.exe, it is supposed to place a C:/dir style path into the configuration; therefore, keep $(pwd). When this configuration value is consumed by the Perl script, it is transformed to /C/dir style by the MSYS2 layer and echoed back in this form in the error message. Hence, do use $PWD for the expected value. Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-22Merge branch 'dl/stash-show-untracked-fixup'Junio C Hamano1-0/+2
Another brown paper bag inconsistency fix for a new feature introduced during this cycle. * dl/stash-show-untracked-fixup: stash show: use stash.showIncludeUntracked even when diff options given
2021-05-22Merge branch 'wm/rev-parse-path-format-wo-arg'Junio C Hamano1-0/+4
The "rev-parse" command did not diagnose the lack of argument to "--path-format" option, which was introduced in v2.31 era, which has been corrected. * wm/rev-parse-path-format-wo-arg: rev-parse: fix segfault with missing --path-format argument
2021-05-22t: use configured TAR instead of tarĐoàn Trần Công Danh2-4/+4
Despite that tar is available everywhere, it's not required by POSIX. In our build system, users are allowed to specify which tar to be used in Makefile knobs. Furthermore, GNU tar (gtar) is prefered when autotools is being used. In our testsuite, 7 out of 9 tar-required-tests use "$TAR", the other two use "tar". Let's change the remaining two tests to "$TAR". Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-22stash show: use stash.showIncludeUntracked even when diff options givenDenton Liu1-0/+2
If options pertaining to how the diff is displayed is provided to `git stash show`, the command will ignore the stash.showIncludeUntracked configuration variable, defaulting to not showing any untracked files. This is unintuitive behaviour since the format of the diff output and whether or not to display untracked files are orthogonal. Use stash.showIncludeUntracked even when diff options are given. Of course, this is still overridable via the command-line options. Update the documentation to explicitly say which configuration variables will be overridden when a diff options are given. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-21diff-merges: let "-m" imply "-p"Sergey Organov1-2/+2
Fix long standing inconsistency between -c/--cc that do imply -p on one side, and -m that did not imply -p on the other side. Change corresponding test accordingly, as "log -m" output should now match one from "log -m -p", rather than from just "log". Change documentation accordingly. NOTES: After this patch git log -m produces diffs without need to provide -p as well, that improves both consistency and usability. It gets even more useful if one sets "log.diffMerges" configuration variable to "first-parent" to force -m produce usual diff with respect to first parent only. This patch, however, does not change behavior when specific diff format is explicitly provided on the command-line, so that commands like git log -m --raw git log -m --stat are not affected, nor does it change commands where specific diff format is active by default, such as: git diff-tree -m It's also worth to be noticed that exact historical semantics of -m is still provided by --diff-merges=separate. Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-21stash list: stop passing "-m" to "git log"Sergey Organov1-1/+1
Passing "-m" in "git log --first-parent -m" is not needed as --first-parent implies --diff-merges=first-parent anyway. OTOH, it will stop being harmless once we let "-m" imply "-p". While we are at it, fix corresponding test description in t3903-stash to match what it actually tests. Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-21t4013: test "git diff-index -m"Sergey Organov1-0/+13
-m in "git diff-index" means "match missing", that differs from its meaning in "git diff". Let's check it in diff-index. Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-21t4013: test "git diff-tree -m"Sergey Organov2-0/+12
We want to ensure we don't affect plumbing commands with our changes of "-m" semantics, so add corresponding test. Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-21t4013: test "git log -m --stat"Sergey Organov2-0/+67
This is to ensure we won't break different diff formats when we start to imply "-p" by "-m". Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-21t4013: test "git log -m --raw"Sergey Organov2-0/+62
This is to ensure we won't break different diff formats when we start to imply "-p" by "-m". Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-21t4013: test that "-m" alone has no effect in "git log"Sergey Organov1-0/+8
This is to notice current behavior that we are going to change when we start to imply "-p" by "-m". Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-21Merge branch 'tz/c-locale-output-is-no-more'Junio C Hamano1-1/+1
Test update. * tz/c-locale-output-is-no-more: t7500: remove non-existant C_LOCALE_OUTPUT prereq
2021-05-21Merge branch 'cs/http-use-basic-after-failed-negotiate'Junio C Hamano1-0/+41
Regression fix for a change made during this cycle. * cs/http-use-basic-after-failed-negotiate: Revert "remote-curl: fall back to basic auth if Negotiate fails" t5551: test http interaction with credential helpers
2021-05-20merge-ort, diffcore-rename: employ cached renames when possibleElijah Newren1-20/+28
When there are many renames between the old base of a series of commits and the new base, the way sequencer.c, merge-recursive.c, and diffcore-rename.c have traditionally split the work resulted in redetecting the same renames with each and every commit being transplanted. To address this, the last several commits have been creating a cache of rename detection results, determining when it was safe to use such a cache in subsequent merge operations, adding helper functions, and so on. See the previous half dozen commit messages for additional discussion of this optimization, particularly the message a few commits ago entitled "add code to check for whether cached renames can be reused". This commit finally ties all of that work together, modifying the merge algorithm to make use of these cached renames. For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 5.665 s ± 0.129 s 5.622 s ± 0.059 s mega-renames: 11.435 s ± 0.158 s 10.127 s ± 0.073 s just-one-mega: 494.2 ms ± 6.1 ms 500.3 ms ± 3.8 ms That's a fairly small improvement, but mostly because the previous optimizations were so effective for these particular testcases; this optimization only kicks in when the others don't. If we undid the basename-guided rename detection and skip-irrelevant-renames optimizations, then we'd see that this series by itself improved performance as follows: Before Basename Series After Just This Series no-renames: 13.815 s ± 0.062 s 5.697 s ± 0.080 s mega-renames: 1799.937 s ± 0.493 s 205.709 s ± 0.457 s Since this optimization kicks in to help accelerate cases where the previous optimizations do not apply, this last comparison shows that this cached-renames optimization has the potential to help signficantly in cases that don't meet the requirements for the other optimizations to be effective. The changes made in this optimization also lay some important groundwork for a future optimization around having collect_merge_info() avoid recursing into subtrees in more cases. However, for this optimization to be effective, merge_switch_to_result() should only be called when the rebase or cherry-pick operation has either completed or hit a case where the user needs to resolve a conflict or edit the result. If it is called after every commit, as sequencer.c does, then the working tree and index are needlessly updated with every commit and the cached metadata is tossed, defeating this optimization. Refactoring sequencer.c to only call merge_switch_to_result() at the end of the operation is a bigger undertaking, and the practical benefits of this optimization will not be realized until that work is performed. Since `test-tool fast-rebase` only updates at the end of the operation, it was used to obtain the timings above. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-20t6429: testcases for remembering renamesElijah Newren1-0/+692
We will soon be adding an optimization that caches (in memory only, never written to disk) upstream renames during a sequence of merges such as occurs during a cherry-pick or rebase operation. Add several tests meant to stress such an implementation to ensure it does the right thing, and include a test whose outcome we will later change due to this optimization as well. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-20fast-rebase: write conflict state to working tree, index, and HEADElijah Newren1-19/+32
Previously, when fast-rebase hit a conflict, it simply aborted and left HEAD, the index, and the working tree where they were before the operation started. While fast-rebase does not support restarting from a conflicted state, write the conflicted state out anyway as it gives us a way to see what the conflicts are and write tests that check for them. This will be important in the upcoming commits, because sequencer.c is only superficially integrated with merge-ort.c; in particular, it calls merge_switch_to_result() after EACH merge instead of only calling it at the end of all the sequence of merges (or when a conflict is hit). This not only causes needless updates to the working copy and index, but also causes all intermediate data to be freed and tossed, preventing caching information from one merge to the next. However, integrating sequencer.c more deeply with merge-ort.c is a big task, and making this small extension to fast-rebase.c provides us with a simple way to test the edge and corner cases that we want to make sure continue working. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-20fast-rebase: change assert() to BUG()Elijah Newren1-1/+2
assert() can succinctly document expectations for the code, and do so in a way that may be useful to future folks trying to refactor the code and change basic assumptions; it allows them to more quickly find some places where their violations of previous assumptions trips things up. Unfortunately, assert() can surround a function call with important side-effects, which is a huge mistake since some users will compile with assertions disabled. I've had to debug such mistakes before in other codebases, so I should know better. Luckily, this was only in test code, but it's still very embarrassing. Change an assert() to an if (...) BUG (...). Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-20Merge branch 'jk/test-chainlint-softer'Junio C Hamano4-3/+21
The "chainlint" feature in the test framework is a handy way to catch common mistakes in writing new tests, but tends to get expensive. An knob to selectively disable it has been introduced to help running tests that the developer has not modified. * jk/test-chainlint-softer: t: avoid sed-based chain-linting in some expensive cases
2021-05-20Merge branch 'zh/ref-filter-push-remote-fix'Junio C Hamano1-0/+19
The handling of "%(push)" formatting element of "for-each-ref" and friends was broken when the same codepath started handling "%(push:<what>)", which has been corrected. * zh/ref-filter-push-remote-fix: ref-filter: fix read invalid union member bug
2021-05-20Merge branch 'en/dir-traversal'Junio C Hamano6-91/+176
"git clean" and "git ls-files -i" had confusion around working on or showing ignored paths inside an ignored directory, which has been corrected. * en/dir-traversal: dir: introduce readdir_skip_dot_and_dotdot() helper dir: update stale description of treat_directory() dir: traverse into untracked directories if they may have ignored subfiles dir: avoid unnecessary traversal into ignored directory t3001, t7300: add testcase showcasing missed directory traversal t7300: add testcase showing unnecessary traversal into ignored directory ls-files: error out on -i unless -o or -c are specified dir: report number of visited directories and paths with trace2 dir: convert trace calls to trace2 equivalents
2021-05-19clone: clean up directory after transport_fetch_refs() failureJeff King1-0/+7
git-clone started respecting errors from the transport subsystem in aab179d937 (builtin/clone.c: don't ignore transport_fetch_refs() errors, 2020-12-03). However, that commit didn't handle the cleanup of the filesystem quite right. The cleanup of the directory that cmd_clone() creates is done by an atexit() handler, which we control with a flag. It starts as JUNK_LEAVE_NONE ("clean up everything"), then progresses to JUNK_LEAVE_REPO when we know we have a valid repo but not working tree, and then finally JUNK_LEAVE_ALL when we have a successful checkout. Most errors cause us to die(), which then triggers the handler to do the right thing based on how far into cmd_clone() we got. But the checks added by aab179d937 instead set the "err" variable and then jump to a new "cleanup" label, which then returns our non-zero status. However, the code after the cleanup label includes setting the flag to JUNK_LEAVE_ALL, and so we accidentally leave the repository and working tree in place. One obvious option to fix this is to reorder the end of the function to set the flag first, before cleanup code, and put the label between them. But we can observe another small bug: the error return from transport_fetch_refs() is generally "-1", and we propagate that to the return value of cmd_clone(), which ultimately becomes the exit code of the process. And we try to avoid transmitting negative values via exit codes (only the low 8 bits are passed along as an unsigned value, though in practice for "-1" this at least retains the property that it's non-zero). Instead, let's just die(). That makes us consistent with rest of the code in the function. It does add a new "fatal:" line to the output, but I'd argue that's a good thing: - in the rare case that the transport code didn't say anything, now the user gets _some_ error message - even if the transport code said something like "error: ssh died of signal 9", it's nice to also say "fatal" to indicate that we considered that to be a show-stopper. Triggering this in the test suite turns out to be surprisingly difficult. Almost every error we'd encounter, including ones deep inside the transport code, cause us to just die() right there! However, one way is to put a fake wrapper around git-upload-pack that sends the complete packfile but exits with a failure code. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-19Revert "remote-curl: fall back to basic auth if Negotiate fails"Jeff King1-1/+1
This reverts commit 1b0d9545bb85912a16b367229d414f55d140d3be. That commit does fix the situation it intended to (avoiding Negotiate even when the credentials were provided in the URL), but it creates a more serious regression: we now never hit the conditional for "we had a username and password, tried them, but the server still gave us a 401". That has two bad effects: 1. we never call credential_reject(), and thus a bogus credential stored by a helper will live on forever 2. we never return HTTP_NOAUTH, so the error message the user gets is "The requested URL returned error: 401", instead of "Authentication failed". Doing this correctly seems non-trivial, as we don't know whether the Negotiate auth was a problem. Since this is a regression in the upcoming v2.23.0 release (for which we're in -rc0), let's revert for now and work on a fix separately. (Note that this isn't a pure revert; the previous commit added a test showing the regression, so we can now flip it to expect_success). Reported-by: Ben Humphreys <behumphreys@atlassian.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-19t5551: test http interaction with credential helpersJeff King1-0/+41
We test authentication with http, and we independently test that credential helpers work, but we don't have any tests that cover the two features working together. Let's add two: 1. Make sure that a successful request asks the helper to save the credential. This works as expected. 2. Make sure that a failed request asks the helper to forget the credential. This is marked as expect_failure, as it was recently regressed by 1b0d9545bb (remote-curl: fall back to basic auth if Negotiate fails, 2021-03-22). The symptom here is that the second request should prompt the user, but doesn't. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-18t7500: remove non-existant C_LOCALE_OUTPUT prereqTodd Zullinger1-1/+1
The C_LOCALE_OUTPUT prerequisite was removed in b1e079807b (tests: remove last uses of C_LOCALE_OUTPUT, 2021-02-11), where Ævar noted: I'm not leaving the prerequisite itself in place for in-flight changes as there currently are none that introduce new tests that rely on it, and because C_LOCALE_OUTPUT is currently a noop on the master branch we likely won't have any new submissions that use it. One more use of C_LOCALE_OUTPUT did creep in with 3d1bda6b5b (t7500: add tests for --fixup=[amend|reword] options, 2021-03-15). This causes a number of the tests to be skipped by default: ok 35 # SKIP --fixup=reword: incompatible with --all (missing C_LOCALE_OUTPUT) ok 36 # SKIP --fixup=reword: incompatible with --include (missing C_LOCALE_OUTPUT) ok 37 # SKIP --fixup=reword: incompatible with --only (missing C_LOCALE_OUTPUT) ok 38 # SKIP --fixup=reword: incompatible with --interactive (missing C_LOCALE_OUTPUT) ok 39 # SKIP --fixup=reword: incompatible with --patch (missing C_LOCALE_OUTPUT) Remove the C_LOCALE_OUTPUT prerequisite from these tests so they are not skipped. Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-17rev-parse: fix segfault with missing --path-format argumentWolfgang Müller1-0/+4
Calling "git rev-parse --path-format" without an argument segfaults instead of giving an error message. Commit fac60b8925 (rev-parse: add option for absolute or relative path formatting, 2020-12-13) added the argument parsing code but forgot to handle NULL. Returning an error makes sense here because there is no default value we could use. Add a test case to verify. Signed-off-by: Wolfgang Müller <wolf@oriole.systems> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-17git-send-email: add option to specify sendmail commandGregory Anders1-0/+31
The sendemail.smtpServer configuration option and --smtp-server command line option both support using a sendmail-like program to send emails by specifying an absolute file path. However, this is not ideal for the following reasons: 1. It overloads the meaning of smtpServer (now a program is being used for the server?) 2. It doesn't allow for non-absolute paths, arguments, or arbitrary scripting Requiring an absolute path is bad for portability, as the same program may be in different locations on different systems. If a user wishes to pass arguments to their program, they have to use the smtpServerOption option, which is cumbersome (as it must be repeated for each option) and doesn't adhere to normal git conventions. Introduce a new configuration option sendemail.sendmailCmd as well as a command line option --sendmail-cmd that can be used to specify a command (with or without arguments) or shell expression to run to send email. The name of this option is consistent with --to-cmd and --cc-cmd. This invocation honors the user's $PATH so that absolute paths are not necessary. Arbitrary shell expressions are also supported, allowing users to do basic scripting. Give this option a higher precedence over --smtp-server and sendemail.smtpServer, as the new interface is more flexible. For backward compatibility, continue to support absolute paths in --smtp-server and sendemail.smtpServer. Signed-off-by: Gregory Anders <greg@gpanders.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-16Merge branch 'dl/stash-show-untracked-fixup'Junio C Hamano1-1/+16
The code to handle options recently added to "git stash show" around untracked part of the stash segfaulted when these options were used on a stash entry that does not record untracked part. * dl/stash-show-untracked-fixup: stash show: fix segfault with --{include,only}-untracked t3905: correct test title
2021-05-16Merge branch 'wc/packed-ref-removal-cleanup'Junio C Hamano1-0/+9
When "git update-ref -d" removes a ref that is packed, it left empty directories under $GIT_DIR/refs/ for * wc/packed-ref-removal-cleanup: refs: cleanup directories when deleting packed ref
2021-05-16Merge branch 'dd/mailinfo-quoted-cr'Junio C Hamano7-0/+165
"git mailinfo" (hence "git am") learned the "--quoted-cr" option to control how lines ending with CRLF wrapped in base64 or qp are handled. * dd/mailinfo-quoted-cr: am: learn to process quoted lines that ends with CRLF mailinfo: allow stripping quoted CR without warning mailinfo: allow squelching quoted CRLF warning mailinfo: warn if CRLF found in decoded base64/QP email mailinfo: stop parsing options manually mailinfo: load default metainfo_charset lazily
2021-05-16Merge branch 'mt/parallel-checkout-part-3'Junio C Hamano7-24/+660
The final part of "parallel checkout". * mt/parallel-checkout-part-3: ci: run test round with parallel-checkout enabled parallel-checkout: add tests related to .gitattributes t0028: extract encoding helpers to lib-encoding.sh parallel-checkout: add tests related to path collisions parallel-checkout: add tests for basic operations checkout-index: add parallel checkout support builtin/checkout.c: complete parallel checkout support make_transient_cache_entry(): optionally alloc from mem_pool
2021-05-16Merge branch 'jt/push-negotiation'Junio C Hamano3-1/+125
"git push" learns to discover common ancestor with the receiving end over protocol v2. * jt/push-negotiation: send-pack: support push negotiation fetch: teach independent negotiation (no packfile) fetch-pack: refactor command and capability write fetch-pack: refactor add_haves() fetch-pack: refactor process_acks()
2021-05-14Merge branch 'ow/no-dryrun-in-add-i'Junio C Hamano1-0/+4
"git add -i --dry-run" does not dry-run, which was surprising. The combination of options has taught to error out. * ow/no-dryrun-in-add-i: add: die if both --dry-run and --interactive are given
2021-05-14Merge branch 'jk/p4-locate-branch-point-optim'Junio C Hamano1-0/+2
"git p4" learned to find branch points more efficiently. * jk/p4-locate-branch-point-optim: git-p4: speed up search for branch parent git-p4: ensure complex branches are cloned correctly
2021-05-14Merge branch 'ba/object-info'Junio C Hamano1-0/+26
Over-the-wire protocol learns a new request type to ask for object sizes given a list of object names. * ba/object-info: object-info: support for retrieving object info
2021-05-14Merge branch 'pw/word-diff-zero-width-matches'Junio C Hamano1-0/+5
The word-diff mode has been taught to work better with a word regexp that can match an empty string. * pw/word-diff-zero-width-matches: word diff: handle zero length matches
2021-05-13t: avoid sed-based chain-linting in some expensive casesJeff King4-3/+21
Commit 878f988350 (t/test-lib: teach --chain-lint to detect broken &&-chains in subshells, 2018-07-11) introduced additional chain-lint tests which add an extra "sed" pipeline to each test we run. This has a measurable impact on runtime. Here are timings with and without a new environment variable (added by this patch) that lets you disable just the additional sed-based chain-lint tests: Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER=1 make test Time (mean ± σ): 64.202 s ± 1.030 s [User: 622.469 s, System: 301.402 s] Range (min … max): 61.571 s … 65.662 s 10 runs Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER=0 make test Time (mean ± σ): 57.591 s ± 0.333 s [User: 529.368 s, System: 270.618 s] Range (min … max): 57.143 s … 58.309 s 10 runs Summary 'GIT_TEST_CHAIN_LINT_HARDER=0 make test' ran 1.11 ± 0.02 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=1 make test' Of course those extra lint checks are doing something useful, so paying a few extra seconds (at least on Linux) isn't so bad (though note the CPU time; we're bounded in our parallel run here by the slowest test, so it really is ~120s of CPU improvement). But we can observe that there are some test scripts where they produce a much stronger effect, and provide less value. In t0027 and t3070 we run a very large number of small tests, all driven by a series of functions/loops which are filling in the test bodies. There we get much less bang for our buck in terms of bug-finding versus CPU cost. This patch introduces a mechanism for controlling when those extra lint checks are run, at two levels: - a user can ask to disable or to force-enable the checks by setting GIT_TEST_CHAIN_LINT_HARDER - if the user hasn't specified a preference, individual scripts can disable the checks by setting GIT_TEST_CHAIN_LINT_HARDER_DEFAULT; scripts which don't set that get the current behavior of enabling them. In addition, this patch flips the default for t0027 and t3070's mass-generated sections to disable the extra checks. Here are the timing results for t0027: Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER=1 ./t0027-auto-crlf.sh Time (mean ± σ): 17.078 s ± 0.848 s [User: 14.878 s, System: 7.075 s] Range (min … max): 15.952 s … 18.421 s 10 runs Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER=0 ./t0027-auto-crlf.sh Time (mean ± σ): 9.063 s ± 0.759 s [User: 7.890 s, System: 3.362 s] Range (min … max): 7.747 s … 10.619 s 10 runs Benchmark #3: ./t0027-auto-crlf.sh Time (mean ± σ): 9.186 s ± 0.881 s [User: 7.957 s, System: 3.427 s] Range (min … max): 7.796 s … 10.498 s 10 runs Summary 'GIT_TEST_CHAIN_LINT_HARDER=0 ./t0027-auto-crlf.sh' ran 1.01 ± 0.13 times faster than './t0027-auto-crlf.sh' 1.88 ± 0.18 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=1 ./t0027-auto-crlf.sh' We can see that disabling the checks for the whole script buys us an almost 2x speedup. But the new default behavior, disabling them only for the mass-generated part, gets us most of that speedup (but still leaves the checks on for further manual tests people might write). As a side note, I'd caution about comparing runtimes and CPU seconds between this timing and the earlier "make test" one. In "make test", we're running a lot of scripts in parallel, so the CPU is throttling down (and thus a CPU second saved here would count for more during a parallel run; the same work takes more CPU seconds there). We get similar results for t3070: Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER=1 ./t3070-wildmatch.sh Time (mean ± σ): 20.054 s ± 3.967 s [User: 16.003 s, System: 8.286 s] Range (min … max): 11.891 s … 23.671 s 10 runs Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER=0 ./t3070-wildmatch.sh Time (mean ± σ): 12.399 s ± 2.256 s [User: 7.542 s, System: 5.342 s] Range (min … max): 9.606 s … 15.727 s 10 runs Benchmark #3: ./t3070-wildmatch.sh Time (mean ± σ): 10.726 s ± 3.476 s [User: 6.790 s, System: 4.365 s] Range (min … max): 5.444 s … 15.376 s 10 runs Summary './t3070-wildmatch.sh' ran 1.16 ± 0.43 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=0 ./t3070-wildmatch.sh' 1.87 ± 0.71 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=1 ./t3070-wildmatch.sh' Again, we get almost a 2x speedup disabling these. In this case, there are no tests not covered by the script's "default to disable" behavior, so the second two benchmarks should be the same (and while they do differ, you can see the variance is quite high but they're within one standard deviation). So it seems like for these two scripts, at least, disabling the extra checks is a reasonable tradeoff. Sadly, the overall runtime of "make test" on my system doesn't get much faster. But that's because we're mostly limited by the cost of the single biggest test. Here are the top-5 tests by wall-clock time from a parallel run, before my patch: 57.9192368984222 t9001-send-email.sh 45.6329638957977 t0027-auto-crlf.sh 32.5278220176697 t3070-wildmatch.sh 22.2701289653778 t7610-mergetool.sh 20.8635759353638 t1701-racy-split-index.sh And after: 57.1476998329163 t9001-send-email.sh 33.776211977005 t0027-auto-crlf.sh 21.3116669654846 t7610-mergetool.sh 20.7748689651489 t1701-racy-split-index.sh 19.6957249641418 t7112-reset-submodule.sh We dropped 12s from t0027, and t3070 dropped off our list entirely at around 16s. In both cases we're bound by t9001, but its slowness is due to the actual tests, so we'll have to deal with it in a different way. But this reduces overall CPU, and means that dealing with t9001 (by improving the speed of send-email or splitting it apart) will let us reduce our overall runtime even on multi-core machines. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13stash show: fix segfault with --{include,only}-untrackedDenton Liu1-0/+15
When `git stash show --include-untracked` or `git stash show --only-untracked` is run on a stash that doesn't include an untracked entry, a segfault occurs. This happens because we do not check whether the untracked entry is actually present and just attempt to blindly dereference it. Ensure that the untracked entry is present before actually attempting to dereference it. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13t3905: correct test titleDenton Liu1-1/+1
We reference the non-existent option `git stash show --show-untracked` when we really meant `--only-untracked`. Correct the test title accordingly. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13dir: traverse into untracked directories if they may have ignored subfilesElijah Newren2-2/+2
A directory that is untracked does not imply that all files under it should be categorized as untracked; in particular, if the caller is interested in ignored files, many files or directories underneath the untracked directory may be ignored. We previously partially handled this right with DIR_SHOW_IGNORED_TOO, but missed DIR_SHOW_IGNORED. It was not obvious, though, because the logic for untracked and excluded files had been fused together making it harder to reason about. The previous commit split that logic out, making it easier to notice that DIR_SHOW_IGNORED was missing. Add it. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13dir: avoid unnecessary traversal into ignored directoryElijah Newren1-1/+1
The show_other_directories case in treat_directory() tried to handle both excludes and untracked files with the same logic, and mishandled both the excludes and the untracked files in the process, in different ways. Split that logic apart, and then focus on the logic for the excludes; a subsequent commit will address the logic for untracked files. For show_other_directories, an excluded directory means that every path underneath that directory will also be excluded. Given that the calling code requested to just show directories when everything under a directory had the same state (that's what the "DIR_SHOW_OTHER_DIRECTORIES" flag means), we generally do not need to traverse into such directories and can just immediately mark them as ignored (i.e. as path_excluded). The only reason we cannot just immediately return path_excluded is the DIR_HIDE_EMPTY_DIRECTORIES flag and the possibility that the ignored directory is an empty directory. The code previously treated DIR_SHOW_IGNORED_TOO in most cases as an exception as well, which was wrong. It can sometimes reduce the number of cases where we need to recurse (namely if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set), but should not be able to increase the number of cases where we need to recurse. Fix the logic accordingly. Some sidenotes about possible confusion with dir.c: * "ignored" often refers to an untracked ignore", i.e. a file which is not tracked which matches one of the ignore/exclusion rules. But you can also have a "tracked ignore", a tracked file that happens to match one of the ignore/exclusion rules and which dir.c has to worry about since "git ls-files -c -i" is supposed to list them. * The dir code often uses "ignored" and "excluded" interchangeably, which you need to keep in mind while reading the code. * "exclude" is used multiple ways in the code: * As noted above, "exclude" is often a synonym for "ignored". * The logic for parsing .gitignore files was re-used in .git/info/sparse-checkout, except there it is used to mark paths that the user wants to *keep*. This was mostly addressed by commit 65edd96aec ("treewide: rename 'exclude' methods to 'pattern'", 2019-09-03), but every once in a while you'll find a comment about "exclude" referring to these patterns that might in fact be in use by the sparse-checkout machinery for inclusion rules. * The word "EXCLUDE" is also used for pathspec negation, as in (pathspec->items[3].magic & PATHSPEC_EXCLUDE) Thus if a user had a .gitignore file containing *~ *.log !settings.log And then ran git add -- 'settings.*' ':^settings.log' Then :^settings.log is a pathspec negation making settings.log not be requested to be added even though all other settings.* files are being added. Also, !settings.log in the gitignore file is a negative exclude pattern meaning that settings.log is normally a file we want to track even though all other *.log files are ignored. Sometimes it feels like dir.c needs its own glossary with its many definitions, including the multiply-defined terms. Reported-by: Jason Gore <Jason.Gore@microsoft.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13t3001, t7300: add testcase showcasing missed directory traversalElijah Newren2-0/+24
In the last commit, we added a testcase showing that the directory traversal machinery sometimes traverses into directories unnecessarily. Here we show that there are cases where it does the opposite: it does not traverse into directories, despite those directories having important files that need to be flagged. Add a testcase showing that `git ls-files -o -i --directory` can omit some of the files it should be listing, and another showing that `git clean -fX` can fail to clean out some of the expected files. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13t7300: add testcase showing unnecessary traversal into ignored directoryElijah Newren1-0/+23
The PNPM package manager is apparently creating deeply nested (but ignored) directory structures; traversing them is costly performance-wise, unnecessary, and in some cases is even throwing warnings/errors because the paths are too long to handle on various platforms. Add a testcase that checks for such unnecessary directory traversal. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13ls-files: error out on -i unless -o or -c are specifiedElijah Newren2-3/+3
ls-files --ignored can be used together with either --others or --cached. After being perplexed for a bit and digging in to the code, I assumed that ls-files -i was just broken and not printing anything and I had a nice patch ready to submit when I finally realized that -i can be used with --cached to find tracked ignores. While that was a mistake on my part, and a careful reading of the documentation could have made this more clear, I suspect this is an error others are likely to make as well. In fact, of two uses in our testsuite, I believe one of the two did make this error. In t1306.13, there are NO tracked files, and all the excludes built up and used in that test and in previous tests thus have to be about untracked files. However, since they were looking for an empty result, the mistake went unnoticed as their erroneous command also just happened to give an empty answer. -i will most the time be used with -o, which would suggest we could just make -i imply -o in the absence of either a -o or -c, but that would be a backward incompatible break. Instead, let's just flag -i without either a -o or -c as an error, and update the two relevant testcases to specify their intent. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13dir: report number of visited directories and paths with trace2Elijah Newren1-1/+2
Provide more statistics in trace2 output that include the number of directories and total paths visited by the directory traversal logic. Subsequent patches will take advantage of this to ensure we do not unnecessarily traverse into ignored directories. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13dir: convert trace calls to trace2 equivalentsElijah Newren2-88/+125
Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-12ref-filter: fix read invalid union member bugZheNing Hu1-0/+19
used_atom.u is an union, and it has different members depending on what atom the auxiliary data the union part of the "struct used_atom" wants to record. At most only one of the members can be valid at any one time. Since the code checks u.remote_ref without even making sure if the atom is "push" or "push:" (which are only two cases that u.remote_ref.push becomes valid), but u.remote_ref shares the same storage for other members of the union, the check was reading from an invalid member, which was the bug. Modify the condition here to check whether the atom name equals to "push" or starts with "push:", to avoid reading the value of invalid member of the union. Signed-off-by: ZheNing Hu <adlternative@gmail.com> [jc: further test fixes] Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11Merge branch 'zh/trailer-cmd'Junio C Hamano1-0/+84
The way the command line specified by the trailer.<token>.command configuration variable receives the end-user supplied value was both error prone and misleading. An alternative to achieve the same goal in a safer and more intuitive way has been added, as the trailer.<token>.cmd configuration variable, to replace it. * zh/trailer-cmd: trailer: add new .cmd config option docs: correct descript of trailer.<token>.command
2021-05-11Merge branch 'jk/symlinked-dotgitx-cleanup'Junio C Hamano3-41/+151
Various test and documentation updates about .gitsomething paths that are symlinks. * jk/symlinked-dotgitx-cleanup: docs: document symlink restrictions for dot-files fsck: warn about symlinked dotfiles we'll open with O_NOFOLLOW t0060: test ntfs/hfs-obscured dotfiles t7450: test .gitmodules symlink matching against obscured names t7450: test verify_path() handling of gitmodules t7415: rename to expand scope fsck_tree(): wrap some long lines fsck_tree(): fix shadowed variable t7415: remove out-dated comment about translation
2021-05-11Merge branch 'jk/pack-objects-negative-options-fix'Junio C Hamano2-158/+122
Options to "git pack-objects" that take numeric values like --window and --depth should not accept negative values; the input validation has been tightened. * jk/pack-objects-negative-options-fix: pack-objects: clamp negative depth to 0 t5316: check behavior of pack-objects --depth=0 pack-objects: clamp negative window size to 0 t5300: check that we produced expected number of deltas t5300: modernize basic tests
2021-05-11Merge branch 'nc/submodule-update-quiet'Junio C Hamano1-0/+24
"git submodule update --quiet" did not propagate the quiet option down to underlying "git fetch", which has been corrected. * nc/submodule-update-quiet: submodule update: silence underlying fetch with "--quiet"
2021-05-11Merge branch 'rj/bisect-skip-honor-terms'Junio C Hamano1-0/+11
"git bisect skip" when custom words are used for new/old did not work, which has been corrected. * rj/bisect-skip-honor-terms: bisect--helper: use BISECT_TERMS in 'bisect skip' command
2021-05-11refs: cleanup directories when deleting packed refWill Chandler1-0/+9
When deleting a packed ref via 'update-ref -d', a lockfile is made in the directory that would contain the loose copy of that ref, creating any directories in the ref's path that do not exist. When the transaction completes, the lockfile is deleted, but any empty parent directories made when creating the lockfile are left in place. These empty directories are not removed by 'pack-refs' or other housekeeping tasks and will accumulate over time. When deleting a loose ref, we remove all empty parent directories at the end of the transaction. This commit applies the parent directory cleanup logic used when deleting loose refs to packed refs as well. Signed-off-by: Will Chandler <wfc@wfchandler.org> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11test-lib: split up and deprecate test_create_repo()Ævar Arnfjörð Bjarmason5-18/+4
Remove various redundant or obsolete code from the test_create_repo() function, and split up its use in test-lib.sh from what tests need from it. This leave us with a pass-through wrapper for "git init" in test-lib-functions.sh, in test-lib.sh we have the same, except for needing to redirect stdout/stderr, and emitting an error ourselves if it fails. We don't need to error() ourselves when test_create_repo() is invoked, as the invocation will be a part of a test's "&&"-chain. Everything below this paragraph is a detailed summary of the history of test_create_repo() explaining why it's safe to remove the various things it was doing: 1. "mkdir -p" isn't needed because "git init" itself will create leading directories if needed. 2. Since we're now a simple wrapper for "git init" we don't need to check that we have only one argument. If someone wants to run "test_create_repo --bare x" that's OK. 3. We won't ever hit that "Cannot setup test environment" error. Checking the test environment sanity when doing "git init" dates back to eea420693be (t0000: catch trivial pilot errors., 2005-12-10) and 2ccd2027b01 (trivial: check, if t/trash directory was successfully created, 2006-01-05). We can also see it in another form a bit later in my own 0d314ce834d (test-lib: use subshell instead of cd $new && .. && cd $old, 2010-08-30). But since 2006f0adaee (t/test-lib: make sure Git has already been built, 2012-09-17) we already check if we have a built git earlier. The one thing this was testing after that 2012 change was that we'd just built "git", but not "git-init", but since 3af4c7156c4 (tests: respect GIT_TEST_INSTALLED when initializing repositories, 2018-11-12) we invoke "git", not "git-init". So all of that's been checked already, and we don't need to re-check it here. 4. We don't need to move .git/hooks out of the way. That dates back to c09a69a83e3 (Disable hooks during tests., 2005-10-16), since then hooks became disabled by default in f98f8cbac01 (Ship sample hooks with .sample suffix, 2008-06-24). So the hooks were already disabled by default, but as can be seen from "mkdir .git/hooks" changes various tests needed to re-setup that directory. Now they no longer do. This makes us implicitly depend on the default hooks being disabled, which is a good thing. If and when we'd have any on-by-default hooks (I see no reason we ever would) we'd want to see the subtle and not so subtle ways that would break the test suite. 5. We don't need to "cd" to the "$repo" directory at all anymore. In the code being removed here we both "cd"'d to the repository before calling "init", and did so in a subshell. It's not important to do either, so both of those can be removed. We cd'd because this code grew from test-lib.sh code where we'd have done so already, see eedf8f97e58 (Abstract test_create_repo out for use in tests., 2006-02-17), and later "cd"'d inside a subshell since 0d314ce834d to avoid having to keep track of an "old pwd" variable to cd back after the setup. Being in the repository directory made moving the hooks around easier (we wouldn't have to fully qualify the path). Since we're not moving the hooks per #4 above we don't need to "cd" for that reason either. 6. We can drop the --template argument and instead rely on the GIT_TEMPLATE_DIR set to the same path earlier in test-lib.sh. See 8683a45d669 (Introduce GIT_TEMPLATE_DIR, 2006-12-19) 7. We only needed that ">&3 2>&4" redirection when invoked from test-lib.sh. We could still invoke test_create_repo() there, but as the invocation is now trivial and we don't have a good reason to use test_create_repo() elsewhere let's call "git init" there ourselves. 8. We didn't need to resolve "git" as "${GIT_TEST_INSTALLED:-$GIT_EXEC_PATH}/git$X" in test_create_repo(), even for the use of test-lib.sh PATH is already set up in test-lib.sh to start with GIT_TEST_INSTALLED and/or GIT_EXEC_PATH before test_create_repo() (now "git init") is called.. So we can simply run "git" and rely on the PATH lookup choosing the right executable. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11test-lib: do not show advice about init.defaultBranch under --verboseÆvar Arnfjörð Bjarmason2-1/+5
Arrange for the advice about naming the initial branch not to be shown in the --verbose output of the test suite. Since 675704c74dd (init: provide useful advice about init.defaultBranch, 2020-12-11) some tests have been very chatty with repeated occurrences of this multi-line advice. Having it be this verbose isn't helpful for anyone in the context of git's own test suite, and it makes debugging tests that use their own "git init" invocations needlessly distracting. By setting the GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME variable early in test-lib.sh itself we'll squash the warning not only for test_create_repo(), as 675704c74dd explicitly intended, but also for other "git init" invocations. And once we'd like to have this configuration set for all "git init" invocations in the test suite we can get rid of the init.defaultBranch configuration setting in test_create_repo(), as repo_default_branch_name() in refs.c will take the GIT_TEST_* variable over it being set. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11test-lib: reformat argument list in test_create_repo()Ævar Arnfjörð Bjarmason1-2/+2
Reformat an argument list changed in 675704c74dd (init: provide useful advice about init.defaultBranch, 2020-12-11) to have the "-c" on the same line as the argument it sets. This whitespace-only change makes it easier to review a subsequent commit. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11submodule tests: use symbolic-ref --short to discover branch nameÆvar Arnfjörð Bjarmason1-1/+2
Change a use of $GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME added in 704fed9ea22 (tests: start moving to a different default main branch name, 2020-10-23) to simply discover the initial branch name of a repository set up in this function with "symbolic-ref --short". That's something done in another test in 704fed9ea22, so doing it like this seems like an omission, or rather an overly eager search/replacement instead of fixing the test logic. There are only three uses of the GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME variable in the test suite, this gets rid of one of those. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11test-lib functions: add --printf option to test_commitÆvar Arnfjörð Bjarmason6-25/+20
Add a --printf option to test_commit to allow writing to the file with "printf" instead of "echo". This is useful for writing "\n", "\0" etc., in particular in combination with the --append option added in 3373518cc8 (test-lib functions: add an --append option to test_commit, 2021-01-12). I'm converting a few tests to use the new option rather than a manual printf/add/commit combination to demonstrate its usefulness. While I'm at it use "test_create_repo" where appropriate, and give the first/second commit a meaningful/more conventional log message in cases where no test cared about that message. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11describe tests: convert setup to use test_commitÆvar Arnfjörð Bjarmason1-45/+13
Convert the setup of the describe tests to use test_commit when possible. This makes use of the new --annotate option to test_commit. Some of the setup here could simply be removed since the data being created wasn't important to any of the subsequent tests, so I've done so. E.g. assigning to the "one" variable was always useless, and just checking that we can describe HEAD after the first commit wasn't useful. In the case of the "two" variable we could instead use the tag we just created. See 5312ab11fbf (Add describe test., 2007-01-13) for the initial version of this code. There's other cases here like redundant "test_tick" invocations, or the simplification of not echoing "X" to a file we're about to tag as "x", now we just use "x" in both cases. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11test-lib functions: add an --annotated option to "test_commit"Ævar Arnfjörð Bjarmason2-9/+24
Add an --annotated option to test_commit to create annotated tags. The tag will share the same message as the commit, and we'll call test_tick before creating it (unless --notick) is provided. There's quite a few tests that could be simplified with this construct. I've picked one to convert in this change as a demonstration. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11test-lib-functions: document test_commit --no-tagÆvar Arnfjörð Bjarmason1-0/+2
In 76b8b8d05c (test-lib functions: document arguments to test_commit, 2021-01-12) I added missing documentation to test_commit, but in less than a month later in 3803a3a099 (t: add --no-tag option to test_commit, 2021-02-09) we got another undocumented option. Let's fix that. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11test-lib-functions: reword "test_commit --append" docsÆvar Arnfjörð Bjarmason1-2/+1
Reword the documentation for "test_commit --append" added in my 3373518cc8 (test-lib functions: add an --append option to test_commit, 2021-01-12). A follow-up commit will make the "echo" part of this configurable, and in any case saying "echo >>" rather than ">>" was redundant. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11test-lib tests: remove dead GIT_TEST_FRAMEWORK_SELFTEST variableÆvar Arnfjörð Bjarmason1-4/+0
Stop setting the GIT_TEST_FRAMEWORK_SELFTEST variable. This was originally needed back in 4231d1ba99 (t0000: do not get self-test disrupted by environment warnings, 2018-09-20). It hasn't been needed since I deleted the relevant code in test-lib.sh in c0eedbc009 (test-lib: remove check_var_migration, 2021-02-09), I just didn't notice that it was set here. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-11test-lib: bring $remove_trash out of retirementÆvar Arnfjörð Bjarmason1-14/+18
There's no point in creating a repository or directory only to decide right afterwards that we're skipping all the tests. We can save ourselves the redundant "git init" or "mkdir" and "rm -rf" in this case. We carry around the "$remove_trash" variable because if the directory is unexpectedly gone at test_done time we'll still want to hit the "trash directory already removed" error, but not if we never created the trash directory. See df4c0d1a792 (test-lib: abort when can't remove trash directory, 2017-04-20) for the addition of that error. So let's partially revert 06478dab4c (test-lib: retire $remove_trash variable, 2017-04-23) and move the decision about whether to skip all tests earlier. Let's also fix a bug that was with us since abc5d372ec (Enable parallel tests, 2008-08-08): we would leak $remove_trash from the environment. We don't want this to error out, so let's reset it to the empty string first: remove_trash=t GIT_SKIP_TESTS=t0001 ./t0001-init.sh I tested this with --debug, see 4d0912a206 (test-lib.sh: do not barf under --debug at the end of the test, 2017-04-24) for a bug we don't want to re-introduce. While I'm at it, let's move the HOME assignment to just before test_create_repo, it could be lower, but it seems better to set it before calling anything in test-lib-functions.sh Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-10Merge branch 'rs/repack-without-loosening-promised-objects'Junio C Hamano2-0/+12
"git repack -A -d" in a partial clone unnecessarily loosened objects in promisor pack. * rs/repack-without-loosening-promised-objects: repack: avoid loosening promisor objects in partial clones
2021-05-10Merge branch 'bc/hash-transition-interop-part-1'Junio C Hamano1-1/+1
SHA-256 transition. * bc/hash-transition-interop-part-1: hex: print objects using the hash algorithm member hex: default to the_hash_algo on zero algorithm value builtin/pack-objects: avoid using struct object_id for pack hash commit-graph: don't store file hashes as struct object_id builtin/show-index: set the algorithm for object IDs hash: provide per-algorithm null OIDs hash: set, copy, and use algo field in struct object_id builtin/pack-redundant: avoid casting buffers to struct object_id Use the final_oid_fn to finalize hashing of object IDs hash: add a function to finalize object IDs http-push: set algorithm when reading object ID Always use oidread to read into struct object_id hash: add an algo member to struct object_id
2021-05-10am: learn to process quoted lines that ends with CRLFĐoàn Trần Công Danh2-0/+49
In previous changes, mailinfo has learnt to process lines that decoded from base64 or quoted-printable, and ends with CRLF. Let's teach "am" that new trick, too. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-10mailinfo: allow stripping quoted CR without warningĐoàn Trần Công Danh1-0/+6
In previous changes, we've turned on warning for quoted CR in base64 or quoted-printable email messages. Some projects see those quoted CR a lot, they know that it happens most of the time, and they find it's desirable to always strip those CR. Those projects in question usually fall back to use other tools to handle patches when receive such patches. Let's help those projects handle those patches by stripping those excessive CR. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-10mailinfo: allow squelching quoted CRLF warningĐoàn Trần Công Danh1-1/+5
In previous change, Git starts to warn for quoted CRLF in decoded base64/QP email. Despite those warnings are usually helpful, quoted CRLF could be part of some users' workflow. Let's give them an option to turn off the warning completely. Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-10mailinfo: warn if CRLF found in decoded base64/QP emailĐoàn Trần Công Danh5-0/+106
When SMTP servers receive 8-bit email messages, possibly with only LF as line ending, some of them decide to change said LF to CRLF. Some mailing list softwares, when receive 8-bit email messages, decide to encode those messages in base64 or quoted-printable. If an email is transfered through above mail servers, then distributed by such mailing list softwares, the recipients will receive an email contains a patch mungled with CRLF encoded inside another encoding. Thus, such CR (in CRLF) couldn't be dropped by "mailsplit". Hence, the mailed patch couldn't be applied cleanly. Such accidents have been observed in the wild [1]. Instead of silently rejecting those messages, let's give our users some warnings if such CR (as part of CRLF) is found. [1]: https://nmbug.notmuchmail.org/nmweb/show/m2lf9ejegj.fsf%40guru.guru-group.fi Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-07Merge branch 'll/clone-reject-shallow'Junio C Hamano1-1/+1
Fix tests when forced to use v0 protocol. * ll/clone-reject-shallow: t5601: mark protocol v2-only test
2021-05-07Merge branch 'dl/complete-stash-updates'Junio C Hamano1-0/+19
Further update the command line completion (in contrib/) for "git stash". * dl/complete-stash-updates: git-completion.bash: consolidate cases in _git_stash() git-completion.bash: use $__git_cmd_idx in more places git-completion.bash: rename to $__git_cmd_idx git-completion.bash: separate some commands onto their own line
2021-05-07Merge branch 'ps/rev-list-object-type-filter'Junio C Hamano2-1/+139
"git rev-list" learns the "--filter=object:type=<type>" option, which can be used to exclude objects of the given kind from the packfile generated by pack-objects. * ps/rev-list-object-type-filter: rev-list: allow filtering of provided items pack-bitmap: implement combined filter pack-bitmap: implement object type filter list-objects: implement object type filter list-objects: support filtering by tag and commit list-objects: move tag processing into its own function revision: mark commit parents as NOT_USER_GIVEN uploadpack.txt: document implication of `uploadpackfilter.allow`
2021-05-07Merge branch 'ab/svn-tests-set-e-fix'Junio C Hamano2-18/+15
Test clean-up. * ab/svn-tests-set-e-fix: svn tests: refactor away a "set -e" in test body svn tests: remove legacy re-setup from init-clone test
2021-05-07Merge branch 'ab/rebase-no-reschedule-failed-exec'Junio C Hamano1-2/+25
"git rebase --[no-]reschedule-failed-exec" did not work well with its configuration variable, which has been corrected. * ab/rebase-no-reschedule-failed-exec: rebase: don't override --no-reschedule-failed-exec with config rebase tests: camel-case rebase.rescheduleFailedExec consistently
2021-05-07Merge branch 'mt/add-rm-in-sparse-checkout'Junio C Hamano5-24/+239
"git add" and "git rm" learned not to touch those paths that are outside of sparse checkout. * mt/add-rm-in-sparse-checkout: rm: honor sparse checkout patterns add: warn when asked to update SKIP_WORKTREE entries refresh_index(): add flag to ignore SKIP_WORKTREE entries pathspec: allow to ignore SKIP_WORKTREE entries on index matching add: make --chmod and --renormalize honor sparse checkouts t3705: add tests for `git add` in sparse checkouts add: include magic part of pathspec on --refresh error
2021-05-07Merge branch 'ps/config-global-override'Junio C Hamano1-0/+85
Replace GIT_CONFIG_NOSYSTEM mechanism to decline from reading the system-wide configuration file with GIT_CONFIG_SYSTEM that lets users specify from which file to read the system-wide configuration (setting it to an empty file would essentially be the same as setting NOSYSTEM), and introduce GIT_CONFIG_GLOBAL to override the per-user configuration in $HOME/.gitconfig. * ps/config-global-override: t1300: fix unset of GIT_CONFIG_NOSYSTEM leaking into subsequent tests config: allow overriding of global and system configuration config: unify code paths to get global config paths config: rename `git_etc_config()`
2021-05-07Merge branch 'zh/pretty-date-human'Junio C Hamano1-0/+6
"git log --format=..." placeholders learned %ah/%ch placeholders to request the --date=human output. * zh/pretty-date-human: pretty: provide human date format
2021-05-07Merge branch 'ad/cygwin-no-backslashes-in-paths'Junio C Hamano2-5/+10
Cygwin pathname handling fix. * ad/cygwin-no-backslashes-in-paths: cygwin: disallow backslashes in file names
2021-05-07Merge branch 'jk/prune-with-bitmap-fix'Junio C Hamano2-6/+32
When the reachability bitmap is in effect, the "do not lose recently created objects and those that are reachable from them" safety to protect us from races were disabled by mistake, which has been corrected. * jk/prune-with-bitmap-fix: prune: save reachable-from-recent objects with bitmaps pack-bitmap: clean up include_check after use
2021-05-07Merge branch 'ab/pretty-date-format-tests'Junio C Hamano1-5/+4
Tweak a few tests for "log --format=..." that show timestamps in various formats. * ab/pretty-date-format-tests: pretty tests: give --date/format tests a better description pretty tests: simplify %aI/%cI date format test
2021-05-07Merge branch 'ps/config-env-option-with-separate-value'Junio C Hamano1-1/+14
"git --config-env var=val cmd" weren't accepted (only --config-env=var=val was). * ps/config-env-option-with-separate-value: git: support separate arg for `--config-env`'s value git.txt: fix synopsis of `--config-env` missing the equals sign
2021-05-07add: die if both --dry-run and --interactive are givenØystein Walle1-0/+4
The interactive machinery does not obey --dry-run. Die appropriately if both flags are passed. Signed-off-by: Øystein Walle <oystwa@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-06git-p4: ensure complex branches are cloned correctlyJoachim Kuebart1-0/+2
When importing a branch from p4, git-p4 searches the history of the parent branch for the branch point. The test for the complex branch structure ensures all files have the expected contents, but doesn't examine the branch structure. Check for the correct branch structure by making sure that the initial commit on each branch is empty. This ensures that the initial commit's parent is indeed the correct branch-off point. Signed-off-by: Joachim Kuebart <joachim.kuebart@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-05word diff: handle zero length matchesPhillip Wood1-0/+5
If find_word_boundaries() encounters a zero length match (which can be caused by matching a newline or using '*' instead of '+' in the regex) we stop splitting the input into words which generates an inaccurate diff. To fix this increment the start point when there is a zero length match and try a new match. This is safe as posix regular expressions always return the longest available match so a zero length match means there are no longer matches available from the current position. Commit bf82940dbf1 (color-words: enable REG_NEWLINE to help user, 2009-01-17) prevented matching newlines in negated character classes but it is still possible for the user to have an explicit newline match in the regex which could cause a zero length match. One could argue that having explicit newline matches or using '*' rather than '+' are user errors but it seems to be better to work round them than produce inaccurate diffs. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-05ci: run test round with parallel-checkout enabledMatheus Tavares2-0/+7
We already have tests for the basic parallel-checkout operations. But this code can also run be executed by other commands, such as git-read-tree and git-sparse-checkout, which are currently not tested with multiple workers. To promote a wider test coverage without duplicating tests: 1. Add the GIT_TEST_CHECKOUT_WORKERS environment variable, to optionally force parallel-checkout execution during the whole test suite. 2. Set this variable (with a value of 2) in the second test round of our linux-gcc CI job. This round runs `make test` again with some optional GIT_TEST_* variables enabled, so there is no additional overhead in exercising the parallel-checkout code here. Note that tests checking out less than two parallel-eligible entries will fall back to the sequential mode. Nevertheless, it's still a good exercise for the parallel-checkout framework as the fallback codepath also writes the queued entries using the parallel-checkout functions (only without spawning any worker). Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-05parallel-checkout: add tests for basic operationsMatheus Tavares2-0/+271
Add tests to populate the working tree during clone and checkout using sequential and parallel mode, to confirm that they produce identical results. Also test basic checkout mechanics, such as checking for symlinks in the leading directories and the abidance to --force. Note: some helper functions are added to a common lib file which is only included by t2080 for now. But they will also be used by other parallel-checkout tests in the following patches. Co-authored-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-05parallel-checkout: add tests related to .gitattributesMatheus Tavares1-0/+194
Add tests to confirm that the `struct conv_attrs` data is correctly passed from the main process to the workers, and that they can properly convert the blobs before writing them to the working tree. Also check that parallel-ineligible entries, such as regular files that require external filters, are correctly smudge and written when parallel-checkout is enabled. Co-authored-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-05t0028: extract encoding helpers to lib-encoding.shMatheus Tavares2-24/+26
The following patch will add tests outside t0028 which will also need to re-encode some strings. Extract the auxiliary encoding functions from t0028 to a common lib file so that they can be reused. Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-05parallel-checkout: add tests related to path collisionsMatheus Tavares2-2/+164
Add tests to confirm that path collisions are properly detected by checkout workers, both to avoid race conditions and to report colliding entries on clone. Co-authored-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-05t5601: mark protocol v2-only testJonathan Tan1-1/+1
A HTTP-clone test introduced in 4fe788b1b0 ("builtin/clone.c: add --reject-shallow option", 2021-04-01) only works in protocol v2, but is not marked as such. The aforementioned patch implements --reject-shallow for a variety of situations, but usage of a protocol that requires a remote helper is not one of them. (Such an implementation would require extending the remote helper protocol to support the passing of a "reject shallow" option, and then teaching it to both protocol-speaking ends.) For now, to make it pass when GIT_TEST_PROTOCOL_VERSION=0 is passed, add "-c protocol.version=2". A more complete solution would be either to augment the remote helper protocol to support this feature or to return a fatal error when using --reject-shallow with a protocol that uses a remote helper. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-05send-pack: support push negotiationJonathan Tan1-0/+35
Teach Git the push.negotiate config variable. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-05fetch: teach independent negotiation (no packfile)Jonathan Tan2-1/+90
Currently, the packfile negotiation step within a Git fetch cannot be done independent of sending the packfile, even though there is at least one application wherein this is useful. Therefore, make it possible for this negotiation step to be done independently. A subsequent commit will use this for one such application - push negotiation. This feature is for protocol v2 only. (An implementation for protocol v0 would require a separate implementation in the fetch, transport, and transport helper code.) In the protocol, the main hindrance towards independent negotiation is that the server can unilaterally decide to send the packfile. This is solved by a "wait-for-done" argument: the server will then wait for the client to say "done". In practice, the client will never say it; instead it will cease requests once it is satisfied. In the client, the main change lies in the transport and transport helper code. fetch_refs_via_pack() performs everything needed - protocol version and capability checks, and the negotiation itself. There are 2 code paths that do not go through fetch_refs_via_pack() that needed to be individually excluded: the bundle transport (excluded through requiring smart_options, which the bundle transport doesn't support) and transport helpers that do not support takeover. If or when we support independent negotiation for protocol v0, we will need to modify these 2 code paths to support it. But for now, report failure if independent negotiation is requested in these cases. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-04t6423: rename file within directory that other side renamedElijah Newren1-0/+58
Add a new testcase where one side of history renames: olddir/ -> newdir/ and the other side of history renames: olddir/a -> olddir/alpha When using merge.directoryRenames=true, it seems logical to expect the file to end up at newdir/alpha. Unfortunately, both merge-recursive and merge-ort currently see this as a rename/rename conflict: olddir/a -> newdir/a vs. olddir/a -> newdir/alpha Suggesting that there's some extra logic we probably want to add somewhere to allow this case to run without triggering a conflict. For now simply document this known issue. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-04trailer: add new .cmd config optionZheNing Hu1-0/+84
The `trailer.<token>.command` configuration variable specifies a command (run via the shell, so it does not have to be a single name or path to the command, but can be a shell script), and the first occurrence of substring $ARG is replaced with the value given to the `interpret-trailer` command for the token in a '--trailer <token>=<value>' argument. This has three downsides: * The use of $ARG in the mechanism misleads the users that the value is passed in the shell variable, and tempt them to use $ARG more than once, but that would not work, as the second and subsequent $ARG are not replaced. * Because $ARG is textually replaced without regard to the shell language syntax, even '$ARG' (inside a single-quote pair), which a user would expect to stay intact, would be replaced, and worse, if the value had an unmatched single quote (imagine a name like "O'Connor", substituted into NAME='$ARG' to make it NAME='O'Connor'), it would result in a broken command that is not syntactically correct (or worse). * The first occurrence of substring `$ARG` will be replaced with the empty string, in the command when the command is first called to add a trailer with the specified <token>. This is a bad design, the nature of automatic execution causes it to add a trailer that we don't expect. Introduce a new `trailer.<token>.cmd` configuration that takes higher precedence to deprecate and eventually remove `trailer.<token>.command`, which passes the value as an argument to the command. Instead of "$ARG", users can refer to the value as positional argument, $1, in their scripts. At the same time, in order to allow `git interpret-trailers` to better simulate the behavior of `git command -s`, 'trailer.<token>.cmd' will not automatically execute. Helped-by: Junio C Hamano <gitster@pobox.com> Helped-by: Christian Couder <christian.couder@gmail.com> Signed-off-by: ZheNing Hu <adlternative@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-04fsck: warn about symlinked dotfiles we'll open with O_NOFOLLOWJeff King1-2/+27
In the commits merged in via 204333b015 (Merge branch 'jk/open-dotgitx-with-nofollow', 2021-03-22), we stopped following symbolic links for .gitattributes, .gitignore, and .mailmap files. Let's teach fsck to warn that these symlinks are not going to do anything. Note that this is just a warning, and won't block the objects via transfer.fsckObjects, since there are reported to be cases of this in the wild (and even once fixed, they will continue to exist in the commit history of those projects, but are not particularly dangerous). Note that we won't add these to the existing gitmodules block in the fsck code. The logic for gitmodules is a bit more complicated, as we also check the content of non-symlink instances we find. But for these new files, there is no content check; we're just looking at the name and mode of the tree entry (and we can avoid even the complicated name checks in the common case that the mode doesn't indicate a symlink). We can reuse the test helper function we defined for .gitmodules, though (it needs some slight adjustments for the fsck error code, and because we don't block these symlinks via verify_path()). Note that I didn't explicitly test the transfer.fsckObjects case here (nor does the existing .gitmodules test that it blocks a push). The translation of fsck severities to outcomes is covered in general in t5504. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-04t0060: test ntfs/hfs-obscured dotfilesJeff King2-13/+63
We have tests that cover various filesystem-specific spellings of ".gitmodules", because we need to reliably identify that path for some security checks. These are from dc2d9ba318 (is_{hfs,ntfs}_dotgitmodules: add tests, 2018-05-12), with the actual code coming from e7cb0b4455 (is_ntfs_dotgit: match other .git files, 2018-05-11) and 0fc333ba20 (is_hfs_dotgit: match other .git files, 2018-05-02). Those latter two commits also added similar matching functions for .gitattributes and .gitignore. These ended up not being used in the final series, and are currently dead code. But in preparation for them being used in some fsck checks, let's make sure they actually work by throwing a few basic tests at them. Likewise, let's cover .mailmap (which does need matching code added). I didn't bother with the whole battery of tests that we cover for .gitmodules. These functions are all based on the same generic matcher, so it's sufficient to test most of the corner cases just once. Note that the ntfs magic prefix names in the tests come from the algorithm described in e7cb0b4455 (and are different for each file). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-04t7450: test .gitmodules symlink matching against obscured namesJeff King1-38/+53
In t7450 we check that both verify_path() and fsck catch malformed .gitmodules entries in trees. However, we don't check that we catch filesystem-equivalent forms of these (e.g., ".GITMOD~1" on Windows). Our name-matching functions are exercised well in t0060, but there's nothing to test that we correctly call the matching functions from the actual fsck and verify_path() code. So instead of testing just .gitmodules, let's repeat our tests for a few basic cases. We don't need to be exhaustive here (t0060 handles that), but just make sure we hit one name of each type. Besides pushing the tests into a function that takes the path as a parameter, we'll need to do a few things: - adjust the directory name to accommodate the tests running multiple times - set core.protecthfs for index checks. Fsck always protects all types by default, but we want to be able to exercise the HFS routines on every system. Note that core.protectntfs is already the default these days, but it doesn't hurt to explicitly label our need for it. - we'll also take the filename ("gitmodules") as a parameter. All calls use the same name for now, but a future patch will extend this to handle other .gitfoo files. Note that our fake-content symlink destination is somewhat .gitmodules specific. But it isn't necessary for other files (which don't do a content check). And it happens to be a valid attribute and ignore file anyway. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-04t7450: test verify_path() handling of gitmodulesJeff King1-3/+16
Commit 10ecfa7649 (verify_path: disallow symlinks in .gitmodules, 2018-05-04) made it impossible to load a symlink .gitmodules file into the index. However, there are no tests of this behavior. Let's make sure this case is covered. We can easily reuse the test setup created by the matching b7b1fca175 (fsck: complain when .gitmodules is a symlink, 2018-05-04). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03t7415: rename to expand scopeJeff King1-3/+10
This script has already expanded beyond its original intent of ".. in submodule names" to include other malicious submodule bits. Let's update the name and description to reflect that, as well as the fact that we'll soon be adding similar tests for other dotfiles (.gitattributes, etc). We'll also renumber it to move it out of the group of submodule-specific tests. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03fsck_tree(): fix shadowed variableJeff King1-2/+3
Commit b2f2039c2b (fsck: accept an oid instead of a "struct tree" for fsck_tree(), 2019-10-18) introduced a new "oid" parameter to fsck_tree(), and we pass it to the report() function when we find problems. However, that is shadowed within the tree-walking loop by the existing "oid" variable which we use to store the oid of each tree entry. As a result, we may report the wrong oid for some problems we detect within the loop (the entry oid, instead of the tree oid). Our tests didn't catch this because they checked only that we found the expected fsck problem, not that it was attached to the correct object. Let's rename both variables in the function to avoid confusion. This makes the diff a little noisy (e.g., all of the report() calls outside the loop were already correct but need to be touched), but makes sure we catch all cases and will avoid similar confusion in the future. And we can update the test to be a bit more specific and catch this problem. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03t7415: remove out-dated comment about translationJeff King1-3/+2
Since GETTEXT_POISON does not exist anymore, there is no point warning people about whether we should use test_i18ngrep. This is doubly confusing because the comment was describing why it was OK to use grep, but it got caught up in the mass conversion of 674ba34038 (fsck: mark strings for translation, 2018-11-10). Note there are other uses of test_i18ngrep in this script which are now obsolete; I'll save those for a mass-cleanup. My goal here was just to fix the confusing comment in code I'm about to refactor. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03pack-objects: clamp negative depth to 0Jeff King1-0/+7
A negative delta depth makes no sense, and the code is not prepared to handle it. If passed "--depth=-1" on the command line, then this line from break_delta_chains(): cur->depth = (total_depth--) % (depth + 1); triggers a divide-by-zero. This is undefined behavior according to the C standard, but on POSIX systems results in SIGFPE killing the process. This is certainly one way to inform the use that the command was invalid, but it's a bit friendlier to just treat it as "don't allow any deltas", which we already do for --depth=0. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03t5316: check behavior of pack-objects --depth=0Jeff King1-0/+8
We'd expect this to cleanly produce no deltas at all (as opposed to getting confused by an out-of-bounds value), and it does. Note we have to adjust our max_chain test helper, which expected to find at least one delta. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03pack-objects: clamp negative window size to 0Jeff King1-0/+5
A negative window size makes no sense, and the code in find_deltas() is not prepared to handle it. If you pass "-1", for example, we end up generate a 0-length array of "struct unpacked", but our loop assumes it has at least one entry in it (and we end up reading garbage memory). We could complain to the user about this, but it's more forgiving to just clamp it to 0, which means "do not find any deltas at all". The 0-case is already tested earlier in the script, so we'll make sure this does the same thing. Reported-by: Yiyuan guo <yguoaz@gmail.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03t5300: check that we produced expected number of deltasJeff King1-3/+20
We pack a set of objects both with and without --window=0, assuming that the 0-length window will cause us not to produce any deltas. Let's confirm that this is the case. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03t5300: modernize basic testsJeff King1-158/+85
The first set of tests in t5300 goes back to 2005, and doesn't use some of our customary style and tools these days. In preparation for touching them, let's modernize a few things: - titles go on the line with test_expect_success, with a hanging open-quote to start the test body - test bodies should be indented with tabs - opening braces for shell blocks in &&-chains go on their own line - no space between redirect operators and files (">foo", not "> foo") - avoid doing work outside of test blocks; in this case, we can stick the setup of ".git2" into the appropriate blocks - avoid modifying and then cleaning up the environment or current directory by using subshells and "git -C" - this test does a curious thing when testing the unpacking: it sets GIT_OBJECT_DIRECTORY, and then does a "git init" in the _original_ directory, creating a weird mixed situation. Instead, it's much simpler to just "git init --bare" a new repository to unpack into, and check the results there. I renamed this "git2" instead of ".git2" to make it more clear it's a separate repo. - we can observe that the bodies of the no-delta, ref_delta, and ofs_delta cases are all virtually identical except for the pack creation, and factor out shared helper functions. I collapsed "do the unpack" and "check the results of the unpack" into a single test, since that makes the expected lifetime of the "git2" temporary directory more clear (that also lets us use test_when_finished to clean it up). This does make the "-v" output slightly less useful, but the improvement in reading the actual test code makes it worth it. - I dropped the "pwd" calls from some tests. These don't do anything functional, and I suspect may have been an aid for debugging when the script was more cavalier about leaving the working directory changed between tests. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03submodule update: silence underlying fetch with "--quiet"Nicholas Clark1-0/+24
Commands such as $ git submodule update --quiet --init --depth=1 involving shallow clones, call the shell function fetch_in_submodule, which in turn invokes git fetch. Pass the --quiet option onward there. Signed-off-by: Nicholas Clark <nick@ccl4.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-30Merge branch 'so/log-diff-merge'Junio C Hamano2-0/+34
"git log" learned "--diff-merges=<style>" option, with an associated configuration variable log.diffMerges. * so/log-diff-merge: doc/diff-options: document new --diff-merges features diff-merges: introduce log.diffMerges config variable diff-merges: adapt -m to enable default diff format diff-merges: refactor set_diff_merges() diff-merges: introduce --diff-merges=on
2021-04-30Merge branch 'ds/sparse-index-protections'Junio C Hamano5-24/+302
Builds on top of the sparse-index infrastructure to mark operations that are not ready to mark with the sparse index, causing them to fall back on fully-populated index that they always have worked with. * ds/sparse-index-protections: (47 commits) name-hash: use expand_to_path() sparse-index: expand_to_path() name-hash: don't add directories to name_hash revision: ensure full index resolve-undo: ensure full index read-cache: ensure full index pathspec: ensure full index merge-recursive: ensure full index entry: ensure full index dir: ensure full index update-index: ensure full index stash: ensure full index rm: ensure full index merge-index: ensure full index ls-files: ensure full index grep: ensure full index fsck: ensure full index difftool: ensure full index commit: ensure full index checkout: ensure full index ...
2021-04-30Merge branch 'ds/maintenance-prefetch-fix'Junio C Hamano2-8/+57
The prefetch task in "git maintenance" assumed that "git fetch" from any remote would fetch all its local branches, which would fetch too much if the user is interested in only a subset of branches there. * ds/maintenance-prefetch-fix: maintenance: respect remote.*.skipFetchAll maintenance: use 'git fetch --prefetch' fetch: add --prefetch option maintenance: simplify prefetch logic
2021-04-30Merge branch 'ow/push-quiet-set-upstream'Junio C Hamano1-0/+7
"git push --quiet --set-upstream" was not quiet when setting the upstream branch configuration, which has been corrected. * ow/push-quiet-set-upstream: transport: respect verbosity when setting upstream
2021-04-30Merge branch 'jk/promisor-optim'Junio C Hamano2-3/+15
Handling of "promisor packs" that allows certain objects to be missing and lazily retrievable has been optimized (a bit). * jk/promisor-optim: revision: avoid parsing with --exclude-promisor-objects lookup_unknown_object(): take a repository argument is_promisor_object(): free tree buffer after parsing
2021-04-30bisect--helper: use BISECT_TERMS in 'bisect skip' commandRamsay Jones1-0/+11
Commit e4c7b33747 ("bisect--helper: reimplement `bisect_skip` shell function in C", 2021-02-03), as part of the shell-to-C conversion, forgot to read the 'terms' file (.git/BISECT_TERMS) during the new 'bisect skip' command implementation. As a result, the 'bisect skip' command will use the default 'bad'/'good' terms. If the bisection terms have been set to non-default values (for example by the 'bisect start' command), then the 'bisect skip' command will fail. In order to correct this problem, we insert a call to the get_terms() function, which reads the non-default terms from that file (if set), in the '--bisect-skip' command implementation of 'bisect--helper'. Also, add a test[1] to protect against potential future regression. [1] https://lore.kernel.org/git/xmqqim45h585.fsf@gitster.g/T/#m207791568054b0f8cf1a3942878ea36293273c7d Reported-by: Trygve Aaberge <trygveaa@gmail.com> Signed-off-by: Bagas Sanjaya <bagasdotme@gmail.com> Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-30cygwin: disallow backslashes in file namesAdam Dinwoodie2-5/+10
The backslash character is not a valid part of a file name on Windows. If, in Windows, Git attempts to write a file that has a backslash character in the filename, it will be incorrectly interpreted as a directory separator. This caused CVE-2019-1354 in MinGW, as this behaviour can be manipulated to cause the checkout to write to files it ought not write to, such as adding code to the .git/hooks directory. This was fixed by e1d911dd4c (mingw: disallow backslash characters in tree objects' file names, 2019-09-12). However, the vulnerability also exists in Cygwin: while Cygwin mostly provides a POSIX-like path system, it will still interpret a backslash as a directory separator. To avoid this vulnerability, CVE-2021-29468, extend the previous fix to also apply to Cygwin. Similarly, extend the test case added by the previous version of the commit. The test suite doesn't have an easy way to say "run this test if in MinGW or Cygwin", so add a new test prerequisite that covers both. As well as checking behaviour in the presence of paths containing backslashes, the existing test also checks behaviour in the presence of paths that differ only by the presence of a trailing ".". MinGW follows normal Windows application behaviour and treats them as the same path, but Cygwin more closely emulates *nix systems (at the expense of compatibility with native Windows applications) and will create and distinguish between such paths. Gate the relevant bit of that test accordingly. Reported-by: RyotaK <security@ryotak.me> Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Adam Dinwoodie <adam@dinwoodie.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-30git: support separate arg for `--config-env`'s valuePatrick Steinhardt1-1/+14
While not documented as such, many of the top-level options like `--git-dir` and `--work-tree` support two syntaxes: they accept both an equals sign between option and its value, and they do support option and value as two separate arguments. The recently added `--config-env` option only supports the syntax with an equals sign. Mitigate this inconsistency by accepting both syntaxes and add tests to verify both work. Signed-off-by: Patrick Steinhardt <ps@pks.im> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-29prune: save reachable-from-recent objects with bitmapsJeff King2-6/+32
We pass our prune expiration to mark_reachable_objects(), which will traverse not only the reachable objects, but consider any recent ones as tips for reachability; see d3038d22f9 (prune: keep objects reachable from recent objects, 2014-10-15) for details. However, this interacts badly with the bitmap code path added in fde67d6896 (prune: use bitmaps for reachability traversal, 2019-02-13). If we hit the bitmap-optimized path, we return immediately to avoid the regular traversal, accidentally skipping the "also traverse recent" code. Instead, we should do an if-else for the bitmap versus regular traversal, and then follow up with the "recent" traversal in either case. This reuses the "rev_info" for a bitmap and then a regular traversal, but that should work OK (the bitmap code clears the pending array in the usual way, just like a regular traversal would). Note that I dropped the comment above the regular traversal here. It has little explanatory value, and makes the if-else logic much harder to read. Here are a few variants that I rejected: - it seems like both the reachability and recent traversals could be done in a single traversal. This was rejected by d3038d22f9 (prune: keep objects reachable from recent objects, 2014-10-15), though the balance may be different when using bitmaps. However, there's a subtle correctness issue, too: we use revs->ignore_missing_links for the recent traversal, but not the reachability one. - we could try using bitmaps for the recent traversal, too, which could possibly improve performance. But it would require some fixes in the bitmap code, which uses ignore_missing_links for its own purposes. Plus it would probably not help all that much in practice. We use the reachable tips to generate bitmaps, so those objects are likely not covered by bitmaps (unless they just became unreachable). And in general, we expect the set of unreachable objects to be much smaller anyway, so there's less to gain. The test in t5304 detects the bug and confirms the fix. I also beefed up the tests in t6501, which covers the mtime-checking code more thoroughly, to handle the bitmap case (in addition to just "loose" and "packed" cases). Interestingly, this test doesn't actually detect the bug, because it is running "git gc", and not "prune" directly. And "gc" will call "repack" first, which does not suffer the same bug. So the old-but-reachable-from-recent objects get scooped up into the new pack along with the actually-recent objects, which gives both a recent mtime. But it seemed prudent to get more coverage of the bitmap case for related code. Reported-by: David Emett <dave@sp4m.net> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-28repack: avoid loosening promisor objects in partial clonesRafael Silva2-0/+12
When `git repack -A -d` is run in a partial clone, `pack-objects` is invoked twice: once to repack all promisor objects, and once to repack all non-promisor objects. The latter `pack-objects` invocation is with --exclude-promisor-objects and --unpack-unreachable, which loosens all objects unused during this invocation. Unfortunately, this includes promisor objects. Because the -d argument to `git repack` subsequently deletes all loose objects also in packs, these just-loosened promisor objects will be immediately deleted. However, this extra disk churn is unnecessary in the first place. For example, in a newly-cloned partial repo that filters all blob objects (e.g. `--filter=blob:none`), `repack` ends up unpacking all trees and commits into the filesystem because every object, in this particular case, is a promisor object. Depending on the repo size, this increases the disk usage considerably: In my copy of the linux.git, the object directory peaked 26GB of more disk usage. In order to avoid this extra disk churn, pass the names of the promisor packfiles as --keep-pack arguments to the second invocation of `pack-objects`. This informs `pack-objects` that the promisor objects are already in a safe packfile and, therefore, do not need to be loosened. For testing, we need to validate whether any object was loosened. However, the "evidence" (loosened objects) is deleted during the process which prevents us from inspecting the object directory. Instead, let's teach `pack-objects` to count loosened objects and emit via trace2 thus allowing inspecting the debug events after the process is finished. This new event is used on the added regression test. Lastly, add a new perf test to evaluate the performance impact made by this changes (tested on git.git): Test HEAD^ HEAD ---------------------------------------------------------- 5600.3: gc 134.38(41.93+90.95) 7.80(6.72+1.35) -94.2% For a bigger repository, such as linux.git, the improvement is even bigger: Test HEAD^ HEAD ------------------------------------------------------------------- 5600.3: gc 6833.00(918.07+3162.74) 268.79(227.02+39.18) -96.1% These improvements are particular big because every object in the newly-cloned partial repository is a promisor object. Reported-by: SZEDER Gábor <szeder.dev@gmail.com> Helped-by: Jeff King <peff@peff.net> Helped-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Rafael Silva <rafaeloliveira.cs@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-27hash: provide per-algorithm null OIDsbrian m. carlson1-1/+1
Up until recently, object IDs did not have an algorithm member, only a hash. Consequently, it was possible to share one null (all-zeros) object ID among all hash algorithms. Now that we're going to be handling objects from multiple hash algorithms, it's important to make sure that all object IDs have a correct algorithm field. Introduce a per-algorithm null OID, and add it to struct hash_algo. Introduce a wrapper function as well, and use it everywhere we used to use the null_oid constant. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-27pretty: provide human date formatZheNing Hu1-0/+6
Add the placeholders %ah and %ch to format author date and committer date, like --date=human does, which provides more humanity date output. Signed-off-by: ZheNing Hu <adlternative@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-27pretty tests: give --date/format tests a better descriptionÆvar Arnfjörð Bjarmason1-3/+3
Change the description for the --date/format equivalency tests added in 466fb6742d7 (pretty: provide a strict ISO 8601 date format, 2014-08-29) and 0df621172d8 (pretty: provide short date format, 2019-11-19) to be more meaningful. This allows us to reword the comment added in the former commit to refer to both tests, and any other future test, such as the in-flight --date=human format being proposed in [1]. 1. http://lore.kernel.org/git/pull.939.v2.git.1619275340051.gitgitgadget@gmail.com Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-27pretty tests: simplify %aI/%cI date format testÆvar Arnfjörð Bjarmason1-2/+1
Change a needlessly complex test for the %aI/%cI date formats (iso-strict) added in 466fb6742d7 (pretty: provide a strict ISO 8601 date format, 2014-08-29) to instead use the same pattern used to test %as/%cs since 0df621172d8 (pretty: provide short date format, 2019-11-19). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-27git-completion.bash: use $__git_cmd_idx in more placesDenton Liu1-0/+19
With the introduction of the $__git_cmd_idx variable in e94fb44042 (git-completion.bash: pass $__git_subcommand_idx from __git_main(), 2021-03-24), completion functions were able to know the index at which the git command is listed, allowing them to skip options that are given to the underlying git itself, not the corresponding command (e.g. `-C asdf` in `git -C asdf branch`). While most of the changes here are self-explanatory, some bear further explanation. For the __git_find_on_cmdline() and __git_find_last_on_cmdline() pair of functions, these functions are only ever called in the context of a git command completion function. These functions will only care about words after the command so we can safely ignore the words before this. For _git_worktree(), this change is technically a no-op (once the __git_find_last_on_cmdline change is also applied). It was in poor style to have hard-coded on the index right after `worktree`. In case `git worktree` were to ever learn to accept options, the current situation would be inflexible. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-27t1300: fix unset of GIT_CONFIG_NOSYSTEM leaking into subsequent testsPatrick Steinhardt1-4/+3
In order to test whether the new GIT_CONFIG_SYSTEM environment variable behaves as expected, we unset GIT_CONFIG_NOSYSTEM in one of our tests in t1300. But because tests are not executed in a subshell, this unset leaks into all subsequent tests and may thus cause them to fail in some environments. These failures are easily reproducable with `make prefix=/root test`. Fix the issue by not using `sane_unset GIT_CONFIG_NOSYSTEM`, but instead just manually add it to the environment of the two command invocations which need it. Reported-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-20object-info: support for retrieving object infoBruno Albuquerque1-0/+26
Sometimes it is useful to get information of an object without having to download it completely. Add the "object-info" capability that lets the client ask for object-related information with their full hexadecimal object names. Only sizes are returned for now. Signed-off-by: Bruno Albuquerque <bga@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-20Merge branch 'ab/detox-gettext-tests'Junio C Hamano3-13/+6
Test clean-up. * ab/detox-gettext-tests: tests: remove all uses of test_i18cmp
2021-04-20Merge branch 'jk/pack-objects-bitmap-progress-fix'Junio C Hamano1-0/+23
When "git pack-objects" makes a literal copy of a part of existing packfile using the reachability bitmaps, its update to the progress meter was broken. * jk/pack-objects-bitmap-progress-fix: pack-objects: update "nr_seen" progress based on pack-reused count
2021-04-20Merge branch 'ab/userdiff-tests'Junio C Hamano6-52/+87
A bit of code clean-up and a lot of test clean-up around userdiff area. * ab/userdiff-tests: blame tests: simplify userdiff driver test blame tests: don't rely on t/t4018/ directory userdiff: remove support for "broken" tests userdiff tests: list builtin drivers via test-tool userdiff tests: explicitly test "default" pattern userdiff: add and use for_each_userdiff_driver() userdiff style: normalize pascal regex declaration userdiff style: declare patterns with consistent style userdiff style: re-order drivers in alphabetical order
2021-04-20Merge branch 'ar/userdiff-scheme'Junio C Hamano16-0/+90
Userdiff patterns for "Scheme" has been added. * ar/userdiff-scheme: userdiff: add support for Scheme
2021-04-19config: allow overriding of global and system configurationPatrick Steinhardt1-0/+86
In order to have git run in a fully controlled environment without any misconfiguration, it may be desirable for users or scripts to override global- and system-level configuration files. We already have a way of doing this, which is to unset both HOME and XDG_CONFIG_HOME environment variables and to set `GIT_CONFIG_NOGLOBAL=true`. This is quite kludgy, and unsetting the first two variables likely has an impact on other executables spawned by such a script. The obvious way to fix this would be to introduce `GIT_CONFIG_NOGLOBAL` as an equivalent to `GIT_CONFIG_NOSYSTEM`. But in the past, it has turned out that this design is inflexible: we cannot test system-level parsing of the git configuration in our test harness because there is no way to change its location, so all tests run with `GIT_CONFIG_NOSYSTEM` set. Instead of doing the same mistake with `GIT_CONFIG_NOGLOBAL`, introduce two new variables `GIT_CONFIG_GLOBAL` and `GIT_CONFIG_SYSTEM`: - If unset, git continues to use the usual locations. - If set to a specific path, we skip reading the normal configuration files and instead take the path. By setting the path to `/dev/null`, no configuration will be loaded for the respective level. This implements the usecase where we want to execute code in a sanitized environment without any potential misconfigurations via `/dev/null`, but is more flexible and allows for more usecases than simply adding `GIT_CONFIG_NOGLOBAL`. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-19rev-list: allow filtering of provided itemsPatrick Steinhardt2-0/+64
When providing an object filter, it is currently impossible to also filter provided items. E.g. when executing `git rev-list HEAD` , the commit this reference points to will be treated as user-provided and is thus excluded from the filtering mechanism. This makes it harder than necessary to properly use the new `--filter=object:type` filter given that even if the user wants to only see blobs, he'll still see commits of provided references. Improve this by introducing a new `--filter-provided-objects` option to the git-rev-parse(1) command. If given, then all user-provided references will be subject to filtering. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-19pack-bitmap: implement combined filterPatrick Steinhardt1-0/+7
When the user has multiple objects filters specified, then this is internally represented by having a "combined" filter. These combined filters aren't yet supported by bitmap indices and can thus not be accelerated. Fix this by implementing support for these combined filters. The implementation is quite trivial: when there's a combined filter, we simply recurse into `filter_bitmap()` for all of the sub-filters. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-19pack-bitmap: implement object type filterPatrick Steinhardt1-1/+24
The preceding commit has added a new object filter for git-rev-list(1) which allows to filter objects by type. Implement the equivalent filter for packfile bitmaps so that we can answer these queries fast. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-19list-objects: implement object type filterPatrick Steinhardt1-0/+44
While it already is possible to filter objects by some criteria in git-rev-list(1), it is not yet possible to filter out only a specific type of objects. This makes some filters less useful. The `blob:limit` filter for example filters blobs such that only those which are smaller than the given limit are returned. But it is unfit to ask only for these smallish blobs, given that git-rev-list(1) will continue to print tags, commits and trees. Now that we have the infrastructure in place to also filter tags and commits, we can improve this situation by implementing a new filter which selects objects based on their type. Above query can thus trivially be implemented with the following command: $ git rev-list --objects --filter=object:type=blob \ --filter=blob:limit=200 Furthermore, this filter allows to optimize for certain other cases: if for example only tags or commits have been selected, there is no need to walk down trees. The new filter is not yet supported in bitmaps. This is going to be implemented in a subsequent commit. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-16diff-merges: introduce log.diffMerges config variableSergey Organov2-0/+26
New log.diffMerges configuration variable sets the format that --diff-merges=on will be using. The default is "separate". t4013: add the following tests for log.diffMerges config: * Test that wrong values are denied. * Test that the value of log.diffMerges properly affects both --diff-merges=on and -m. t9902: fix completion tests for log.d* to match log.diffMerges. Added documentation for log.diffMerges. Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-16diff-merges: introduce --diff-merges=onSergey Organov1-0/+8
Introduce the notion of default diff format for merges, and the option "on" to select it. The default format is "separate" and can't yet be changed, so effectively "on" is just a synonym for "separate" for now. Add corresponding test to t4013. This is in preparation for introducing log.diffMerges configuration option that will let --diff-merges=on to be configured to any supported format. Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-16Merge branch 'en/ort-readiness'Junio C Hamano8-10/+183
Plug the ort merge backend throughout the rest of the system, and start testing it as a replacement for the recursive backend. * en/ort-readiness: Add testing with merge-ort merge strategy t6423: mark remaining expected failure under merge-ort as such Revert "merge-ort: ignore the directory rename split conflict for now" merge-recursive: add a bunch of FIXME comments documenting known bugs merge-ort: write $GIT_DIR/AUTO_MERGE whenever we hit a conflict t: mark several submodule merging tests as fixed under merge-ort merge-ort: implement CE_SKIP_WORKTREE handling with conflicted entries t6428: new test for SKIP_WORKTREE handling and conflicts merge-ort: support subtree shifting merge-ort: let renormalization change modify/delete into clean delete merge-ort: have ll_merge() use a special attr_index for renormalization merge-ort: add a special minimal index just for renormalization merge-ort: use STABLE_QSORT instead of QSORT where required
2021-04-16maintenance: respect remote.*.skipFetchAllDerrick Stolee1-1/+7
If a remote has the skipFetchAll setting enabled, then that remote is not intended for frequent fetching. It makes sense to not fetch that data during the 'prefetch' maintenance task. Skip that remote in the iteration without error. The skip_default_update member is initialized in remote.c:handle_config() as part of initializing the 'struct remote'. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-16maintenance: use 'git fetch --prefetch'Derrick Stolee1-7/+7
The 'prefetch' maintenance task previously forced the following refspec for each remote: +refs/heads/*:refs/prefetch/<remote>/* If a user has specified a more strict refspec for the remote, then this prefetch task downloads more objects than necessary. The previous change introduced the '--prefetch' option to 'git fetch' which manipulates the remote's refspec to place all resulting refs into refs/prefetch/, with further partitioning based on the destinations of those refspecs. Update the documentation to be more generic about the destination refs. Do not mention custom refspecs explicitly, as that does not need to be highlighted in this documentation. The important part of placing refs in refs/prefetch/ remains. Reported-by: Tom Saeger <tom.saeger@oracle.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-16fetch: add --prefetch optionDerrick Stolee1-0/+43
The --prefetch option will be used by the 'prefetch' maintenance task instead of sending refspecs explicitly across the command-line. The intention is to modify the refspec to place all results in refs/prefetch/ instead of anywhere else. Create helper method filter_prefetch_refspec() to modify a given refspec to fit the rules expected of the prefetch task: * Negative refspecs are preserved. * Refspecs without a destination are removed. * Refspecs whose source starts with "refs/tags/" are removed. * Other refspecs are placed within "refs/prefetch/". Finally, we add the 'force' option to ensure that prefetch refs are replaced as necessary. There are some interesting cases that are worth testing. An earlier version of this change dropped the "i--" from the loop that deletes a refspec item and shifts the remaining entries down. This allowed some refspecs to not be modified. The subtle part about the first --prefetch test is that the "refs/tags/*" refspec appears directly before the "refs/heads/bogus/*" refspec. Without that "i--", this ordering would remove the "refs/tags/*" refspec and leave the last one unmodified, placing the result in "refs/heads/*". It is possible to have an empty refspec. This is typically the case for remotes other than the origin, where users want to fetch a specific tag or branch. To correctly test this case, we need to further remove the upstream remote for the local branch. Thus, we are testing a refspec that will be deleted, leaving nothing to fetch. Helped-by: Tom Saeger <tom.saeger@oracle.com> Helped-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-15Merge branch 'jz/apply-3way-cached'Junio C Hamano1-0/+50
"git apply" now takes "--3way" and "--cached" at the same time, and work and record results only in the index. * jz/apply-3way-cached: git-apply: allow simultaneous --cached and --3way options
2021-04-15Merge branch 'jz/apply-run-3way-first'Junio C Hamano1-0/+20
"git apply --3way" has always been "to fall back to 3-way merge only when straight application fails". Swap the order of falling back so that 3-way is always attempted first (only when the option is given, of course) and then straight patch application is used as a fallback when it fails. * jz/apply-run-3way-first: git-apply: try threeway first when "--3way" is used
2021-04-15transport: respect verbosity when setting upstreamØystein Walle1-0/+7
A command such as `git push -qu origin feature` will print "Branch 'feature' set up to track remote branch 'feature' from 'origin'." even when --quiet is passed. In this case it's because install_branch_config() is always called with BRANCH_CONFIG_VERBOSE. struct transport keeps track of the desired verbosity. Fix the above issue by passing BRANCH_CONFIG_VERBOSE conditionally based on that. Signed-off-by: Øystein Walle <oystwa@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-13Merge branch 'cc/test-helper-bloom-usage-fix'Junio C Hamano1-1/+1
Usage message fix for a test helper. * cc/test-helper-bloom-usage-fix: test-bloom: fix missing 'bloom' from usage string
2021-04-13Merge branch 'ab/send-email-validate-errors'Junio C Hamano1-8/+27
Clean-up codepaths that implements "git send-email --validate" option and improves the message from it. * ab/send-email-validate-errors: git-send-email: improve --validate error output git-send-email: refactor duplicate $? checks into a function git-send-email: test full --validate output
2021-04-13Merge branch 'tb/pack-preferred-tips-to-give-bitmap'Junio C Hamano4-0/+64
A configuration variable has been added to force tips of certain refs to be given a reachability bitmap. * tb/pack-preferred-tips-to-give-bitmap: builtin/pack-objects.c: respect 'pack.preferBitmapTips' t/helper/test-bitmap.c: initial commit pack-bitmap: add 'test_bitmap_commits()' helper
2021-04-13Merge branch 'jk/ref-filter-segfault-fix'Junio C Hamano1-0/+10
A NULL-dereference bug has been corrected in an error codepath in "git for-each-ref", "git branch --list" etc. * jk/ref-filter-segfault-fix: ref-filter: fix NULL check for parse object failure
2021-04-13tests: remove all uses of test_i18cmpÆvar Arnfjörð Bjarmason3-13/+6
Finish the removal I started in 1108cea7f8e (tests: remove most uses of test_i18ncmp, 2021-02-11). At that time the function wasn't removed due to disruption with in-flight changes, remove the occurrences that have landed since then. As of writing this there are no test_i18ncmp uses between "master" and "seen", so let's also remove the function to finally put it to rest. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-13revision: avoid parsing with --exclude-promisor-objectsJeff King1-0/+8
When --exclude-promisor-objects is given, before traversing any objects we iterate over all of the objects in any promisor packs, marking them as UNINTERESTING and SEEN. We turn the oid we get from iterating the pack into an object with parse_object(), but this has two problems: - it's slow; we are zlib inflating (and reconstructing from deltas) every byte of every object in the packfile - it leaves the tree buffers attached to their structs, which means our heap usage will grow to store every uncompressed tree simultaneously. This can be gigabytes. We can obviously fix the second by freeing the tree buffers after we've parsed them. But we can observe that the function doesn't look at the object contents at all! The only reason we call parse_object() is that we need a "struct object" on which to set the flags. There are two options here: - we can look up just the object type via oid_object_info(), and then call the appropriate lookup_foo() function - we can call lookup_unknown_object(), which gives us an OBJ_NONE struct (which will get auto-converted later by object_as_type() via calls to lookup_commit(), etc). The first one is closer to the current code, but we do pay the price to look up the type for each object. The latter should be more efficient in CPU, though it wastes a little bit of memory (the "unknown" object structs are a union of all object types, so some of the structs are bigger than they need to be). It also runs the risk of triggering a latent bug in code that calls lookup_object() directly but isn't ready to handle OBJ_NONE (such code would already be buggy, but we use lookup_unknown_object() infrequently enough that it might be hiding). I went with the second option here. I don't think the risk is high (and we'd want to find and fix any such bugs anyway), and it should be more efficient overall. The new tests in p5600 show off the improvement (this is on git.git): Test HEAD^ HEAD ------------------------------------------------------------------------------- 5600.5: count commits 0.37(0.37+0.00) 0.38(0.38+0.00) +2.7% 5600.6: count non-promisor commits 11.74(11.37+0.37) 0.04(0.03+0.00) -99.7% The improvement is particularly big in this script because _every_ object in the newly-cloned partial repo is a promisor object. So after marking them all, there's nothing left to traverse. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-13lookup_unknown_object(): take a repository argumentJeff King1-3/+3
All of the other lookup_foo() functions take a repository argument, but lookup_unknown_object() was never converted, and it uses the_repository internally. Let's fix that. We could leave a wrapper that uses the_repository, but there aren't that many calls, so we'll just convert them all. I looked briefly at each site to see if we had a repository struct (besides the_repository) we could pass, but none of them do (so this conversion to pass the_repository is a pure noop in each case, though it does take us one step closer to eventually getting rid of the_repository). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-13is_promisor_object(): free tree buffer after parsingJeff King1-0/+4
To get the list of all promisor objects, we not only include all objects in promisor packs, but also parse each of those objects to see which objects they reference. After parsing a tree object, the tree->buffer field will remain populated until we explicitly free it. So in a partial clone of blob:none, for example, we are essentially reading every tree in the repository (since they're all in the initial promisor pack), and keeping all of their uncompressed contents in memory at once. This patch frees the tree buffers after we've finished marking all of their reachable objects. We shouldn't need to do this for any other object type. While we are using some extra memory to store the structs, no other object type stores the whole contents in its parsed form (we do sometimes hold on to commit buffers, but less so these days due to commit graphs, plus most commands which care about promisor objects turn off the save_commit_buffer global). Even for a moderate-sized repository like git.git, this patch drops the peak heap (as measured by massif) for git-fsck from ~1.7GB to ~138MB. Fsck is a good candidate for measuring here because it doesn't interact with the promisor code except to call is_promisor_object(), so we can isolate just this problem. The added perf test shows only a tiny improvement on my machine for git.git, since 1.7GB isn't enough to cause any real memory pressure: Test HEAD^ HEAD -------------------------------------------------------------------------------- 5600.4: fsck 21.26(20.90+0.35) 20.84(20.79+0.04) -2.0% With linux.git the absolute change is a bit bigger, though still a small percentage: Test HEAD^ HEAD ----------------------------------------------------------------------------- 5600.4: fsck 262.26(259.13+3.12) 254.92(254.62+0.29) -2.8% I didn't have the patience to run it under massif with linux.git, but it's probably on the order of about 14GB improvement, since that's the sum of the sizes of all of the uncompressed trees (but still isn't enough to create memory pressure on this particular machine, which has 64GB of RAM). Smaller machines would probably see a bigger effect on runtime (and sadly our perf suite does not measure peak heap). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-12svn tests: refactor away a "set -e" in test bodyÆvar Arnfjörð Bjarmason1-12/+15
Refactor a test added in 83c9433e67 (git-svn: support for git-svn propset, 2014-12-07) to avoid using "set -e" in the test body. Let's move this into a setup test using "test_expect_success" instead. While I'm at it refactor: * Repeated "mkdir" to "mkdir -p" * Uses of "touch" to creating the files with ">" instead * The "rm -rf" at the end to happen in a "test_when_finished" Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-12svn tests: remove legacy re-setup from init-clone testÆvar Arnfjörð Bjarmason1-6/+0
Remove the immediate "rm -rf .git" from the start of this test. This was added back in 41337e22f0 (git-svn: add tests for command-line usage of init and clone commands, 2007-11-17) when there was a "trash" directory shared by all the tests, but ever since abc5d372ec (Enable parallel tests, 2008-08-08) we've had per-test trash directories. So this setup can simply be removed. We could use TEST_NO_CREATE_REPO=true, but I don't think it's worth the effort to go out of our way to be different. It doesn't matter that we now have a redundant .git at the top-level. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-12pack-objects: update "nr_seen" progress based on pack-reused countJeff King1-0/+23
When serving a clone or fetch with bitmaps, after deciding which objects need to be sent our "pack reuse" mechanism kicks in: we try to send more-or-less verbatim a bunch of objects from the beginning of the bitmapped packfile without even adding them to the to_pack.objects array. After deciding which objects will be in the "reused" portion, we update nr_result to account for those, and then trigger display_progress() to show the user (who is undoubtedly dazzled that we managed to enumerate so many objects so quickly). But then something confusing happens: the "Enumerating objects" progress meter jumps _backwards_, counting up from zero the number of objects we actually add into to_pack.objects. This worked correctly once upon a time, but was broken in 5af050437a (pack-objects: show some progress when counting kept objects, 2018-04-15), when the latter half of that progress meter switched to using a separate nr_seen counter, rather than nr_result. Nobody noticed for two reasons: - prior to the pack-reuse fixes from a14aebeac3 (Merge branch 'jk/packfile-reuse-cleanup', 2020-02-14), the reuse code almost never kicked in anyway - the output looks _kind of_ correct. The "backwards" moment is hard to catch, because we overwrite the old progress number with the new one, and the larger number is displayed only for a second. So unless you look at that exact second, you just see the much smaller value, counting up to the number of non-reused objects (though of course if you catch it in stderr, or look at GIT_TRACE_PACKET from a server with bitmaps, you can see both values). This smaller output isn't wrong per se, but isn't counting what we ever intended to. We should give the user the whole number of objects we considered (which, as per 5af050437a's original purpose, is already _not_ a count of what goes into to_pack.objects). The follow-on "Counting objects" meter shows the actual number of objects we feed into that array. We can easily fix this by bumping (and showing) nr_seen for the pack-reused objects. When the included test is run without this patch, the second pack-objects invocation produces "Enumerating objects: 1" to show the one loose object, even though the resulting pack has hundreds of objects in it. With it, we jump to "Enumerating objects: 674" after deciding on reuse, and then "675" when we add in the loose object. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-10rebase: don't override --no-reschedule-failed-exec with configÆvar Arnfjörð Bjarmason1-0/+23
Fix a bug in how --no-reschedule-failed-exec interacts with rebase.rescheduleFailedExec=true being set in the config. Before this change the --no-reschedule-failed-exec config option would be overridden by the config. This bug happened because of the particulars of how "rebase" works v.s. most other git commands when it comes to parsing options and config: When we read the config and parse the CLI options we correctly prefer the --no-reschedule-failed-exec option over rebase.rescheduleFailedExec=true in the config. So far so good. However the --reschedule-failed-exec option doesn't take effect when the rebase starts (we'd just create a ".git/rebase-merge/reschedule-failed-exec" file if it was true). It only takes effect when the exec command fails, at which point we'll reschedule the failed "exec" command. Since we only wrote out the positive ".git/rebase-merge/reschedule-failed-exec" under --reschedule-failed-exec, but nothing with --no-reschedule-failed-exec we'll forget that we asked not to reschedule failed "exec", and would happily re-read the config and see that rebase.rescheduleFailedExec=true is set. So the config will effectively override the user having explicitly disabled the option on the command-line. Even more confusingly: Since rebase accepts different options based on its state there wasn't even a way to get around this with "rebase --continue --no-reschedule-failed-exec" (but you could of course set the config with "rebase -c ..."). I think the least bad way out of this is to declare that for such options and config whatever we decide at the beginning of the rebase goes. So we'll now always create either a "reschedule-failed-exec" or a "no-reschedule-failed-exec file at the start, not just the former if we decided we wanted the feature. With this new worldview you can no longer change the setting once a rebase has started except by manually removing the state files discussed above. I think making it work like that is the the least confusing thing we can do. In the future we might want to learn to change the setting in the middle by combining "--edit-todo" with "--[no-]reschedule-failed-exec", we currently don't support combining those options, or any other way to change the state in the middle of the rebase short of manually editing the files in ".git/rebase-merge/*". The bug being fixed here originally came about because of a combination of the behavior of the code added in d421afa0c66 (rebase: introduce --reschedule-failed-exec, 2018-12-10) and the addition of the config variable in 969de3ff0e0 (rebase: add a config option to default to --reschedule-failed-exec, 2018-12-10). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-10rebase tests: camel-case rebase.rescheduleFailedExec consistentlyÆvar Arnfjörð Bjarmason1-2/+2
Fix a test added in 906b63942ac (rebase --am: ignore rebase.rescheduleFailedExec, 2019-07-01) to camel-case the configuration variable. This doesn't change the behavior of the test, it's merely to help its human readers. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08rm: honor sparse checkout patternsMatheus Tavares2-5/+78
`git add` refrains from adding or updating index entries that are outside the current sparse checkout, but `git rm` doesn't follow the same restriction. This is somewhat counter-intuitive and inconsistent. So make `rm` honor the sparsity rules and advise on how to remove SKIP_WORKTREE entries just like `add` does. Also add some tests for the new behavior. Suggested-by: Elijah Newren <newren@gmail.com> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08add: warn when asked to update SKIP_WORKTREE entriesMatheus Tavares1-7/+66
`git add` already refrains from updating SKIP_WORKTREE entries, but it silently exits with zero code when it is asked to do so. Instead, let's warn the user and display a hint on how to update these entries. Note that we only warn the user whey they give a pathspec item that matches no eligible path for updating, but it does match one or more SKIP_WORKTREE entries. A warning was chosen over erroring out right away to reproduce the same behavior `add` already exhibits with ignored files. This also allow users to continue their workflow without having to invoke `add` again with only the eligible paths (as those will have already been added). Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08add: make --chmod and --renormalize honor sparse checkoutsMatheus Tavares1-2/+2
Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08t3705: add tests for `git add` in sparse checkoutsMatheus Tavares2-19/+96
We already have a couple tests for `add` with SKIP_WORKTREE entries in t7012, but these only cover the most basic scenarios. As we will be changing how `add` deals with sparse paths in the subsequent commits, let's move these two tests to their own file and add more test cases for different `add` options and situations. This also demonstrates two options that don't currently respect SKIP_WORKTREE entries: `--chmod` and `--renormalize`. Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08add: include magic part of pathspec on --refresh errorMatheus Tavares1-0/+6
When `git add --refresh <pathspec>` doesn't find any matches for the given pathspec, it prints an error message using the `match` field of the `struct pathspec_item`. However, this field doesn't contain the magic part of the pathspec. Instead, let's use the `original` field. Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08userdiff: add support for SchemeAtharva Raykar16-0/+90
Add a diff driver for Scheme-like languages which recognizes top level and local `define` forms, whether it is a function definition, binding, syntax definition or a user-defined `define-xyzzy` form. Also supports R6RS `library` forms, `module` forms along with class and struct declarations used in Racket (PLT Scheme). Alternate "def" syntax such as those in Gerbil Scheme are also supported, like defstruct, defsyntax and so on. The rationale for picking `define` forms for the hunk headers is because it is usually the only significant form for defining the structure of the program, and it is a common pattern for schemers to have local function definitions to hide their visibility, so it is not only the top level `define`'s that are of interest. Schemers also extend the language with macros to provide their own define forms (for example, something like a `define-test-suite`) which is also captured in the hunk header. Since it is common practice to extend syntax with variants of a form like `module+`, `class*` etc, those have been supported as well. The word regex is a best-effort attempt to conform to R7RS[1] valid identifiers, symbols and numbers. [1] https://small.r7rs.org/attachment/r7rs.pdf (section 2.1) Signed-off-by: Atharva Raykar <raykar.ath@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08Merge branch 'en/ort-perf-batch-9'Junio C Hamano1-0/+71
The ort merge backend has been optimized by skipping irrelevant renames. * en/ort-perf-batch-9: diffcore-rename: avoid doing basename comparisons for irrelevant sources merge-ort: skip rename detection entirely if possible merge-ort: use relevant_sources to filter possible rename sources merge-ort: precompute whether directory rename detection is needed merge-ort: introduce wrappers for alternate tree traversal merge-ort: add data structures for an alternate tree traversal merge-ort: precompute subset of sources for which we need rename detection diffcore-rename: enable filtering possible rename sources
2021-04-08Merge branch 'en/sequencer-edit-upon-conflict-fix'Junio C Hamano1-1/+31
"git cherry-pick/revert" with or without "--[no-]edit" did not spawn the editor as expected (e.g. "revert --no-edit" after a conflict still asked to edit the message), which has been corrected. * en/sequencer-edit-upon-conflict-fix: sequencer: fix edit handling for cherry-pick and revert messages
2021-04-08Merge branch 'll/clone-reject-shallow'Junio C Hamano3-1/+60
"git clone --reject-shallow" option fails the clone as soon as we notice that we are cloning from a shallow repository. * ll/clone-reject-shallow: builtin/clone.c: add --reject-shallow option
2021-04-08Merge branch 'tb/reverse-midx'Junio C Hamano2-4/+62
An on-disk reverse-index to map the in-pack location of an object back to its object name across multiple packfiles is introduced. * tb/reverse-midx: midx.c: improve cache locality in midx_pack_order_cmp() pack-revindex: write multi-pack reverse indexes pack-write.c: extract 'write_rev_file_order' pack-revindex: read multi-pack reverse indexes Documentation/technical: describe multi-pack reverse indexes midx: make some functions non-static midx: keep track of the checksum midx: don't free midx_name early midx: allow marking a pack as preferred t/helper/test-read-midx.c: add '--show-objects' builtin/multi-pack-index.c: display usage on unrecognized command builtin/multi-pack-index.c: don't enter bogus cmd_mode builtin/multi-pack-index.c: split sub-commands builtin/multi-pack-index.c: define common usage with a macro builtin/multi-pack-index.c: don't handle 'progress' separately builtin/multi-pack-index.c: inline 'flags' with options
2021-04-08blame tests: simplify userdiff driver testÆvar Arnfjörð Bjarmason1-21/+15
Simplify the test added in 9466e3809d (blame: enable funcname blaming with userdiff driver, 2020-11-01) to use the --author support recently added in 999cfc4f45 (test-lib functions: add --author support to test_commit, 2021-01-12). We also did not need the full fortran-external-function content. Let's cut it down to just the important parts. I'm modifying it to demonstrate that the fortran-specific userdiff function is in effect by adding "DO NOT MATCH ..." and "AS THE ..." lines surrounding the "RIGHT" one. This is to check that we're using the userdiff "fortran" driver, as opposed to the default driver which would match on those lines as part of the general heuristic of matching a line that doesn't begin with whitespace. The test had also been leaving behind a .gitattributes file for later tests to possibly trip over, let's clean it up with "test_when_finished". Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08blame tests: don't rely on t/t4018/ directoryÆvar Arnfjörð Bjarmason1-3/+13
Refactor a test added in 9466e3809d (blame: enable funcname blaming with userdiff driver, 2020-11-01) so that the blame tests don't rely on stealing the contents of "t/t4018/fortran-external-function". I have another patch series that'll possibly (or not) refactor that file, but having this test inter-dependency makes things simple in any case by making this test more readable. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08userdiff: remove support for "broken" testsÆvar Arnfjörð Bjarmason2-10/+1
There have been no "broken" tests since 75c3b6b2e8 (userdiff: improve Fortran xfuncname regex, 2020-08-12). Let's remove the test support for them. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08userdiff tests: list builtin drivers via test-toolÆvar Arnfjörð Bjarmason4-27/+67
Change the userdiff test to list the builtin drivers via the test-tool, using the new for_each_userdiff_driver() API function. This gets rid of the need to modify this part of the test every time a new pattern is added, see 2ff6c34612 (userdiff: support Bash, 2020-10-22) and 09dad9256a (userdiff: support Markdown, 2020-05-02) for two recent examples. I only need the "list-builtin-drivers "argument here, but let's add "list-custom-drivers" and "list-drivers" too, just because it's easy. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-08userdiff tests: explicitly test "default" patternÆvar Arnfjörð Bjarmason1-0/+1
Since 122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05) the internals of the userdiff.c code have understood a "default" name, which is invoked as userdiff_find_by_name("default") and present in the "builtin_drivers" struct. Let's test for this special case. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-07git-apply: allow simultaneous --cached and --3way optionsJerry Zhang1-0/+50
"git apply" does not allow "--cached" and "--3way" to be used together, since "--3way" writes conflict markers into the working tree. Allow "git apply" to accept "--cached" and "--3way" at the same time. When a single file auto-resolves cleanly, the result is placed in the index at stage #0 and the command exits with 0 status. For a file that has a conflict which cannot be cleanly auto-resolved, the original contents from common ancestor (stage conflict at the content level, and the command exists with non-zero status, because there is no place (like the working tree) to leave a half-resolved merge for the user to resolve. The user can use `git diff` to view the contents of the conflict, or `git checkout -m -- .` to regenerate the conflict markers in the working directory. Don't attempt rerere in this case since it depends on conflict markers written to file for its database storage and lookup. There would be two main changes required to get rerere working: 1. Allow the rerere api to accept in memory object rather than files, which would allow us to pass in the conflict markers contained in the result from ll_merge(). 2. Rerere can't write to the working directory, so it would have to apply the result to cache stage #0 directly. A flag would be needed to control this. Signed-off-by: Jerry Zhang <jerry@skydio.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-07Merge branch 'ps/pack-bitmap-optim'Junio C Hamano1-0/+14
Optimize "rev-list --use-bitmap-index --objects" corner case that uses negative tags as the stopping points. * ps/pack-bitmap-optim: pack-bitmap: avoid traversal of objects referenced by uninteresting tag
2021-04-07Merge branch 'zh/commit-trailer'Junio C Hamano1-0/+312
"git commit" learned "--trailer <key>[=<value>]" option; together with the interpret-trailers command, this will make it easier to support custom trailers. * zh/commit-trailer: commit: add --trailer option
2021-04-06git-apply: try threeway first when "--3way" is usedJerry Zhang1-0/+20
The apply_fragments() method of "git apply" can silently apply patches incorrectly if a file has repeating contents. In these cases a three-way merge is capable of applying it correctly in more situations, and will show a conflict rather than applying it incorrectly. However, because the patches apply "successfully" using apply_fragments(), git will never fall back to the merge, even if the "--3way" flag is used, and the user has no way to ensure correctness by forcing the three-way merge method. Change the behavior so that when "--3way" is used, git will always try the three-way merge first and will only fall back to apply_fragments() in cases where blobs are not available or some other error (but not in the case of a merge conflict). Since user-facing results will be different, this has backwards compatibility implications for users depending on the old behavior. In addition, the three-way merge will be slower than direct patch application. Signed-off-by: Jerry Zhang <jerry@skydio.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-06git-send-email: improve --validate error outputÆvar Arnfjörð Bjarmason1-5/+12
Improve the output we emit on --validate error to: * Say "FILE:LINE" instead of "FILE: LINE", to match "grep -n", compiler error messages etc. * Don't say "patch contains a" after just mentioning the filename, just leave it at "FILE:LINE: is longer than[...]. The "contains a" sounded like we were talking about the file in general, when we're actually checking it line-by-line. * Don't just say "rejected by sendemail-validate hook", but combine that with the system_or_msg() output to say what exit code the hook died with. I had an aborted attempt to make the line length checker note all lines that were longer than the limit. I didn't think that was worth the effort, but I've left in the testing change to check that we die as soon as we spot the first long line. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-06git-send-email: test full --validate outputÆvar Arnfjörð Bjarmason1-6/+18
Change the tests that grep substrings out of the output to use a full test_cmp, in preparation for improving the output. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-05test-bloom: fix missing 'bloom' from usage stringChristian Couder1-1/+1
Like 'get_murmur3' and 'generate_filter', 'get_filter_for_commit' is a subcommand of `test-tool bloom` not of `test-tool` itself. Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-02Merge branch 'zh/format-patch-fractional-reroll-count'Junio C Hamano2-0/+58
"git format-patch -v<n>" learned to allow a reroll count that is not an integer. * zh/format-patch-fractional-reroll-count: format-patch: allow a non-integral version numbers