aboutsummaryrefslogtreecommitdiffstats
path: root/t
AgeCommit message (Collapse)AuthorFilesLines
2019-06-19commit-graph: clean up chains after flattened writeDerrick Stolee1-0/+12
If we write a commit-graph file without the split option, then we write to $OBJDIR/info/commit-graph and start to ignore the chains in $OBJDIR/info/commit-graphs/. Unlink the commit-graph-chain file and expire the graph-{hash}.graph files in $OBJDIR/info/commit-graphs/ during every write. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-19commit-graph: verify chains with --shallow modeDerrick Stolee1-0/+62
If we wrote a commit-graph chain, we only modified the tip file in the chain. It is valuable to verify what we wrote, but not waste time checking files we did not write. Add a '--shallow' option to the 'git commit-graph verify' subcommand and check that it does not read the base graph in a two-file chain. Making the verify subcommand read from a chain of commit-graphs takes some rearranging of the builtin code. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-19commit-graph: create options for split filesDerrick Stolee1-0/+47
The split commit-graph feature is now fully implemented, but needs some more run-time configurability. Allow direct callers to 'git commit-graph write --split' to specify the values used in the merge strategy and the expire time. Update the documentation to specify these values. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-19commit-graph: expire commit-graph filesDerrick Stolee1-1/+1
As we merge commit-graph files in a commit-graph chain, we should clean up the files that are no longer used. This change introduces an 'expiry_window' value to the context, which is always zero (for now). We then check the modified time of each graph-{hash}.graph file in the $OBJDIR/info/commit-graphs folder and unlink the files that are older than the expiry_window. Since this is always zero, this immediately clears all unused graph files. We will update the value to match a config setting in a future change. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-19commit-graph: allow cross-alternate chainsDerrick Stolee1-0/+37
In an environment like a fork network, it is helpful to have a commit-graph chain that spans both the base repo and the fork repo. The fork is usually a small set of data on top of the large repo, but sometimes the fork is much larger. For example, git-for-windows/git has almost double the number of commits as git/git because it rebases its commits on every major version update. To allow cross-alternate commit-graph chains, we need a few pieces: 1. When looking for a graph-{hash}.graph file, check all alternates. 2. When merging commit-graph chains, do not merge across alternates. 3. When writing a new commit-graph chain based on a commit-graph file in another object directory, do not allow success if the base file has of the name "commit-graph" instead of "commit-graphs/graph-{hash}.graph". Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-19commit-graph: merge commit-graph chainsDerrick Stolee1-0/+13
When searching for a commit in a commit-graph chain of G graphs with N commits, the search takes O(G log N) time. If we always add a new tip graph with every write, the linear G term will start to dominate and slow the lookup process. To keep lookups fast, but also keep most incremental writes fast, create a strategy for merging levels of the commit-graph chain. The strategy is detailed in the commit-graph design document, but is summarized by these two conditions: 1. If the number of commits we are adding is more than half the number of commits in the graph below, then merge with that graph. 2. If we are writing more than 64,000 commits into a single graph, then merge with all lower graphs. The numeric values in the conditions above are currently constant, but can become config options in a future update. As we merge levels of the commit-graph chain, check that the commits still exist in the repository. A garbage-collection operation may have removed those commits from the object store and we do not want to persist them in the commit-graph chain. This is a non-issue if the 'git gc' process wrote a new, single-level commit-graph file. After we merge levels, the old graph-{hash}.graph files are no longer referenced by the commit-graph-chain file. We will expire these files in a future change. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-19commit-graph: add --split option to builtinDerrick Stolee1-0/+122
Add a new "--split" option to the 'git commit-graph write' subcommand. This option allows the optional behavior of writing a commit-graph chain. The current behavior will add a tip commit-graph containing any commits that are not in the existing commit-graph or commit-graph chain. Later changes will allow merging the chain and expiring out-dated files. Add a new test script (t5324-split-commit-graph.sh) that demonstrates this behavior. Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-19commit-graph: write commit-graph chainsDerrick Stolee1-1/+1
Extend write_commit_graph() to write a commit-graph chain when given the COMMIT_GRAPH_SPLIT flag. This implementation is purposefully simplistic in how it creates a new chain. The commits not already in the chain are added to a new tip commit-graph file. Much of the logic around writing a graph-{hash}.graph file and updating the commit-graph-chain file is the same as the commit-graph file case. However, there are several places where we need to do some extra logic in the split case. Track the list of graph filenames before and after the planned write. This will be more important when we start merging graph files, but it also allows us to upgrade our commit-graph file to the appropriate graph-{hash}.graph file when we upgrade to a chain of commit-graphs. Note that we use the eighth byte of the commit-graph header to store the number of base graph files. This determines the length of the base graphs chunk. A subtle change of behavior with the new logic is that we do not write a commit-graph if we our commit list is empty. This extends to the typical case, which is reflected in t5318-commit-graph.sh. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-12commit-graph: return with errors during writeDerrick Stolee1-0/+8
The write_commit_graph() method uses die() to report failure and exit when confronted with an unexpected condition. This use of die() in a library function is incorrect and is now replaced by error() statements and an int return type. Return zero on success and a negative value on failure. Now that we use 'goto cleanup' to jump to the terminal condition on an error, we have new paths that could lead to uninitialized values. New initializers are added to correct for this. The builtins 'commit-graph', 'gc', and 'commit' call these methods, so update them to check the return value. Test that 'git commit-graph write' returns a proper error code when hitting a failure condition in write_commit_graph(). Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01commit-graph write: don't die if the existing graph is corruptÆvar Arnfjörð Bjarmason1-2/+9
When the commit-graph is written we end up calling parse_commit(). This will in turn invoke code that'll consult the existing commit-graph about the commit, if the graph is corrupted we die. We thus get into a state where a failing "commit-graph verify" can't be followed-up with a "commit-graph write" if core.commitGraph=true is set, the graph either needs to be manually removed to proceed, or core.commitGraph needs to be set to "false". Change the "commit-graph write" codepath to use a new parse_commit_no_graph() helper instead of parse_commit() to avoid this. The latter will call repo_parse_commit_internal() with use_commit_graph=1 as seen in 177722b344 ("commit: integrate commit graph with commit parsing", 2018-04-10). Not using the old graph at all slows down the writing of the new graph by some small amount, but is a sensible way to prevent an error in the existing commit-graph from spreading. Just fixing the current issue would be likely to result in code that's inadvertently broken in the future. New code might use the commit-graph at a distance. To detect such cases introduce a "GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD" setting used when we do our corruption tests, and test that a "write/verify" combo works after every one of our current test cases where we now detect commit-graph corruption. Some of the code changes here might be strictly unnecessary, e.g. I was unable to find cases where the parse_commit() called from write_graph_chunk_data() didn't exit early due to "item->object.parsed" being true in repo_parse_commit_internal() (before the use_commit_graph=1 has any effect). But let's also convert those cases for good measure, we do not have exhaustive tests for all possible types of commit-graph corruption. This might need to be re-visited if we learn to write the commit-graph incrementally, but probably not. Hopefully we'll just start by finding out what commits we have in total, then read the old graph(s) to see what they cover, and finally write a new graph file with everything that's missing. In that case the new graph writing code just needs to continue to use e.g. a parse_commit() that doesn't consult the existing commit-graphs. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01commit-graph verify: detect inability to read the graphÆvar Arnfjörð Bjarmason1-0/+6
Change "commit-graph verify" to error on open() failures other than ENOENT. As noted in the third paragraph of 283e68c72f ("commit-graph: add 'verify' subcommand", 2018-06-27) and the test it added it's intentional that "commit-graph verify" doesn't error out when the file doesn't exist. But let's not be overly promiscuous in what we accept. If we can't read the file for other reasons, e.g. permission errors, bad file descriptor etc. we'd like to report an error to the user. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01commit-graph: don't early exit(1) on e.g. "git status"Ævar Arnfjörð Bjarmason1-1/+1
Make the commit-graph loading code work as a library that returns an error code instead of calling exit(1) when the commit-graph is corrupt. This means that e.g. "status" will now report commit-graph corruption as an "error: [...]" at the top of its output, but then proceed to work normally. This required splitting up the load_commit_graph_one() function so that the code that deals with open()-ing and stat()-ing the graph can now be called independently as open_commit_graph(). This is needed because "commit-graph verify" where the graph doesn't exist isn't an error. See the third paragraph in 283e68c72f ("commit-graph: add 'verify' subcommand", 2018-06-27). There's a bug in that logic where we conflate the intended ENOENT with other errno values (e.g. EACCES), but this change doesn't address that. That'll be addressed in a follow-up change. I'm then splitting most of the logic out of load_commit_graph_one() into load_commit_graph_one_fd_st(), which allows for providing an existing file descriptor and stat information to the loading code. This isn't strictly needed, but it would be redundant and confusing to open() and stat() the file twice for some of the codepaths, this allows for calling open_commit_graph() followed by load_commit_graph_one_fd_st(). The "graph_file" still needs to be passed to that function for the the "graph file %s is too small" error message. This leaves load_commit_graph_one() unused by everything except the internal prepare_commit_graph_one() function, so let's mark it as "static". If someone needs it in the future we can remove the "static" attribute. I could also rewrite its sole remaining user ("prepare_commit_graph_one()") to use load_commit_graph_one_fd_st() instead, but let's leave it at this. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-04-01commit-graph: fix segfault on e.g. "git status"Ævar Arnfjörð Bjarmason1-1/+2
When core.commitGraph=true is set, various common commands now consult the commit graph. Because the commit-graph code is very trusting of its input data, it's possibly to construct a graph that'll cause an immediate segfault on e.g. "status" (and e.g. "log", "blame", ...). In some other cases where git immediately exits with a cryptic error about the graph being broken. The root cause of this is that while the "commit-graph verify" sub-command exhaustively verifies the graph, other users of the graph simply trust the graph, and will e.g. deference data found at certain offsets as pointers, causing segfaults. This change does the bare minimum to ensure that we don't segfault in the common fill_commit_in_graph() codepath called by e.g. setup_revisions(), to do this instrument the "commit-graph verify" tests to always check if "status" would subsequently segfault. This fixes the following tests which would previously segfault: not ok 50 - detect low chunk count not ok 51 - detect missing OID fanout chunk not ok 52 - detect missing OID lookup chunk not ok 53 - detect missing commit data chunk Those happened because with the commit-graph enabled setup_revisions() would eventually call fill_commit_in_graph(), where e.g. g->chunk_commit_data is used early as an offset (and will be 0x0). With this change we get far enough to detect that the graph is broken, and show an error instead. E.g.: $ git status; echo $? error: commit-graph is missing the Commit Data chunk 1 That also sucks, we should *warn* and not hard-fail "status" just because the commit-graph is corrupt, but fixing is left to a follow-up change. A side-effect of changing the reporting from graph_report() to error() is that we now have an "error: " prefix for these even for "commit-graph verify". Pseudo-diff before/after: $ git commit-graph verify -commit-graph is missing the Commit Data chunk +error: commit-graph is missing the Commit Data chunk Changing that is OK. Various errors it emits now early on are prefixed with "error: ", moving these over and changing the output doesn't break anything. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-22commit-graph tests: test a graph that's too smallÆvar Arnfjörð Bjarmason1-0/+6
Use the recently split-up components of the corrupt_graph_and_verify() function to assert that we error on graphs that are too small. The error was added in 2a2e32bdc5 ("commit-graph: implement git commit-graph read", 2018-04-10), but there was no test for it. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-22commit-graph tests: split up corrupt_graph_and_verify()Ævar Arnfjörð Bjarmason1-6/+16
Split up the corrupt_graph_and_verify() function added in d9b9f8a6fd ("commit-graph: verify catches corrupt signature", 2018-06-27) into its logical components of setting up the test itself, doing the corruption in a particular way with "dd", and then finally testing that stderr is what we expect. This allows for re-using everything except the now slimmer corrupt_graph_and_verify() to corrupt the graph in a way that doesn't involve inserting a given byte sequence at a given position, e.g. truncating it entirely to a custom value. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-19Merge branch 'js/test-tool-gen-nuls'Junio C Hamano4-7/+24
* js/test-tool-gen-nuls: tests: teach the test-tool to generate NUL bytes and use it
2019-02-19Merge branch 'mk/t5562-no-input-to-too-large-an-input-test'Junio C Hamano1-2/+2
* mk/t5562-no-input-to-too-large-an-input-test: t5562: do not depend on /dev/zero Revert "t5562: replace /dev/zero with a pipe from generate_zero_bytes"
2019-02-19t5562: do not reuse output filesMax Kirillov1-4/+4
Some expected failures of git-http-backend leaves running its children (receive-pack or upload-pack) which still hold opened descriptors to act.err and with some probability they live long enough to write there their failure messages after next test has already truncated the files. This causes occasional failures of the test script. Avoid the issue by using separated output and error file for each test, apprending the test number to their name. Reported-by: Carlo Arenas <carenas@gmail.com> Helped-by: Carlo Arenas <carenas@gmail.com> Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Max Kirillov <max@max630.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-19tests: teach the test-tool to generate NUL bytes and use itJohannes Schindelin4-7/+24
In cc95bc2025 (t5562: replace /dev/zero with a pipe from generate_zero_bytes, 2019-02-09), we replaced usage of /dev/zero (which is not available on NonStop, apparently) by a Perl script snippet to generate NUL bytes. Sadly, it does not seem to work on NonStop, as t5562 reportedly hangs. Worse, this also hangs in the Ubuntu 16.04 agents of the CI builds on Azure Pipelines: for some reason, the Perl script snippet that is run via `generate_zero_bytes` in t5562's 'CONTENT_LENGTH overflow ssite_t' test case tries to write out an infinite amount of NUL bytes unless a broken pipe is encountered, that snippet never encounters the broken pipe, and keeps going until the build times out. Oddly enough, this does not reproduce on the Windows and macOS agents, nor in a local Ubuntu 18.04. This developer tried for a day to figure out the exact circumstances under which this hang happens, to no avail, the details remain a mystery. In the end, though, what counts is that this here change incidentally fixes that hang (maybe also on NonStop?). Even more positively, it gets rid of yet another unnecessary Perl invocation. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-19t5562: do not depend on /dev/zeroMax Kirillov1-1/+1
It was reported [1] that NonStop platform does not have /dev/zero. The test uses /dev/zero as a dummy input. Passing case (http-backed failed because of too big input size) should not be reading anything from it. If http-backend would erroneously try to read any data returning EOF probably would be even safer than providing some meaningless data. Replace /dev/zero with /dev/null to avoid issues with platforms which do not have /dev/zero. [1] https://public-inbox.org/git/20190209185930.5256-4-randall.s.becker@rogers.com/ Reported-by: Randall S. Becker <rsbecker@nexbridge.com> Signed-off-by: Max Kirillov <max@max630.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-19Revert "t5562: replace /dev/zero with a pipe from generate_zero_bytes"Junio C Hamano1-2/+2
Revert cc95bc20 ("t5562: replace /dev/zero with a pipe from generate_zero_bytes", 2019-02-09), as not feeding anything to the command is a better way to test it.
2019-02-14Merge branch 'jc/no-grepping-for-strerror-in-tests'Junio C Hamano1-1/+1
* jc/no-grepping-for-strerror-in-tests: t1404: do not rely on the exact phrasing of strerror()
2019-02-14Merge branch 'jt/fetch-v2-sideband'Junio C Hamano1-0/+1
"git fetch" and "git upload-pack" learned to send all exchange over the sideband channel while talking the v2 protocol. * jt/fetch-v2-sideband: t/lib-httpd: pass GIT_TEST_SIDEBAND_ALL through Apache
2019-02-14t/lib-httpd: pass GIT_TEST_SIDEBAND_ALL through ApacheTodd Zullinger1-0/+1
07c3c2aa16 ("tests: define GIT_TEST_SIDEBAND_ALL", 2019-01-16) added GIT_TEST_SIDEBAND_ALL to the apache.conf PassEnv list. Avoid warnings from Apache when the variable is unset, as we do for GIT_VALGRIND* and GIT_TRACE, from f628825481 ("t/lib-httpd: handle running under --valgrind", 2012-07-24) and 89c57ab3f0 ("t: pass GIT_TRACE through Apache", 2015-03-13), respectively. Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-14t1404: do not rely on the exact phrasing of strerror()Junio C Hamano1-1/+1
Not even in C locale, it is wrong to expect that the exact phrasing "File exists" is used to show EEXIST. Reported-by: Randall S. Becker <rsbecker@nexbridge.com> Helped-by: Duy Nguyen <pclouds@gmail.com> Helped-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-13Merge branch 'ab/rebase-test-fix'Junio C Hamano1-3/+6
* ab/rebase-test-fix: rebase: fix regression in rebase.useBuiltin=false test mode
2019-02-13Merge branch 'rb/no-dev-zero-in-test'Junio C Hamano3-3/+16
* rb/no-dev-zero-in-test: t5562: replace /dev/zero with a pipe from generate_zero_bytes t5318: replace use of /dev/zero with generate_zero_bytes test-lib-functions.sh: add generate_zero_bytes function
2019-02-13Merge branch 'sg/stress-test'Junio C Hamano3-4/+23
Test improvement. * sg/stress-test: test-lib: fix non-portable pattern bracket expressions test-lib: make '--stress' more bisect-friendly
2019-02-13Merge branch 'kd/t0028-octal-del-is-377-not-777'Junio C Hamano1-4/+4
Test fix. * kd/t0028-octal-del-is-377-not-777: t0028: fix wrong octal values for BOM in setup
2019-02-13rebase: fix regression in rebase.useBuiltin=false test modeÆvar Arnfjörð Bjarmason1-3/+6
Fix a recently introduced regression in c762aada1a ("rebase -x: sanity check command", 2019-01-29) triggered when running the tests with GIT_TEST_REBASE_USE_BUILTIN=false. See 62c23938fa ("tests: add a special setup where rebase.useBuiltin is off", 2018-11-14) for how that test mode works. As discussed on-list[1] it's not worth it to implement the sanity check in the legacy rebase code, we plan to remove it after the 2.21 release. So let's do the bare minimum to make the tests pass under the GIT_TEST_REBASE_USE_BUILTIN=false special setup. 1. https://public-inbox.org/git/xmqqva1nbeno.fsf@gitster-ct.c.googlers.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-13t5562: replace /dev/zero with a pipe from generate_zero_bytesRandall S. Becker1-2/+2
To help platforms that lack /dev/zero (e.g. NonStop), replace use of /dev/zero to feed "git http-backend" with a pipe of output from the generate_zero_bytes helper. Signed-off-by: Randall S. Becker <rsbecker@nexbridge.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-13t5318: replace use of /dev/zero with generate_zero_bytesRandall S. Becker1-1/+1
There are platforms (e.g. NonStop) that lack /dev/zero; use the generate_zero_bytes helper we just introduced to append stream of NULs at the end of the file. The original, even though it uses "dd seek=... count=..." to make it look like it is overwriting the middle part of an existing file, has truncated the file before this step with another use of "dd", which may make it tricky to see why this rewrite is a correct one. Signed-off-by: Randall S. Becker <rsbecker@nexbridge.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-12test-lib-functions.sh: add generate_zero_bytes functionRandall S. Becker1-0/+13
t5318 and t5562 used /dev/zero, which is not portable. This function provides both a fixed block of NUL bytes and an infinite stream of NULs. Signed-off-by: Randall S. Becker <rsbecker@nexbridge.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-11utf8: handle systems that don't write BOM for UTF-16brian m. carlson1-5/+29
When serializing UTF-16 (and UTF-32), there are three possible ways to write the stream. One can write the data with a BOM in either big-endian or little-endian format, or one can write the data without a BOM in big-endian format. Most systems' iconv implementations choose to write it with a BOM in some endianness, since this is the most foolproof, and it is resistant to misinterpretation on Windows, where UTF-16 and the little-endian serialization are very common. For compatibility with Windows and to avoid accidental misuse there, Git always wants to write UTF-16 with a BOM, and will refuse to read UTF-16 without it. However, musl's iconv implementation writes UTF-16 without a BOM, relying on the user to interpret it as big-endian. This causes t0028 and the related functionality to fail, since Git won't read the file without a BOM. Add a Makefile and #define knob, ICONV_OMITS_BOM, that can be set if the iconv implementation has this behavior. When set, Git will write a BOM manually for UTF-16 and UTF-32 and then force the data to be written in UTF-16BE or UTF-32BE. We choose big-endian behavior here because the tests use the raw "UTF-16" encoding, which will be big-endian when the implementation requires this knob to be set. Update the tests to detect this case and write test data with an added BOM if necessary. Always write the BOM in the tests in big-endian format, since all iconv implementations that omit a BOM must use big-endian serialization according to the Unicode standard. Preserve the existing behavior for systems which do not have this knob enabled, since they may use optimized implementations, including defaulting to the native endianness, which may improve performance. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-11t0028: fix wrong octal values for BOM in setupKevin Daudt1-4/+4
The setup code uses octal values with printf to generate a BOM for UTF-16/32 BE/LE. It specifically uses '\777' to emit a 0xff byte. This relies on the fact that most shells truncate the value above 0o377. Ash however interprets '\777' as '\77' + a literal '7', resulting in an invalid BOM. Fix this by using the proper value of 0xff: '\377'. Signed-off-by: Kevin Daudt <me@ikke.info> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-11test-lib: fix non-portable pattern bracket expressionsSZEDER Gábor2-3/+3
Use a '!' character to start a non-matching pattern bracket expression, as specified by POSIX in Shell Command Language section 2.13.1 Patterns Matching a Single Character [1]. I used '^' instead in three places in the previous three commits, to verify that the arguments of the '--stress=' and '--stress-limit=' options and the values of various '*_PORT' environment variables are valid numbers. With certain shells, at least with dash (upstream and in Ubuntu 14.04) and mksh, this led to various undesired behaviors: # error message in case of a valid number $ ~/src/dash/src/dash ./t3903-stash.sh --stress=8 error: --stress=<N> requires the number of jobs to run # not the expected error message $ ~/src/dash/src/dash ./t3903-stash.sh --stress=foo ./t3903-stash.sh: 238: test: Illegal number: foo # no error message at all?! $ mksh ./t3903-stash.sh --stress=foo $ echo $? 0 Some other shells, e.g. Bash (even in posix mode), ksh, dash in Ubuntu 16.04 or later, are apparently happy to accept '^' just as well. [1] http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_13 Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-08Merge branch 'ld/git-p4-remove-flakey-test'Junio C Hamano1-27/+0
A flakey "p4" test has been removed. * ld/git-p4-remove-flakey-test: git-p4: remove ticket expiry test
2019-02-08Merge branch 'js/rebase-i-redo-exec-fix'Junio C Hamano1-3/+0
For "rebase -i --reschedule-failed-exec", we do not want the "-y" shortcut after all. * js/rebase-i-redo-exec-fix: Revert "rebase: introduce a shortcut for --reschedule-failed-exec"
2019-02-08Merge branch 'js/smart-http-detect-remote-error'Junio C Hamano4-0/+13
Some errors from the other side coming over smart HTTP transport were not noticed, which has been corrected. * js/smart-http-detect-remote-error: t5551: test server-side ERR packet remote-curl: tighten "version 2" check for smart-http remote-curl: refactor smart-http discovery
2019-02-08Merge branch 'tz/gpg-test-fix'Junio C Hamano1-2/+1
Test fix. * tz/gpg-test-fix: t/lib-gpg: drop redundant killing of gpg-agent t/lib-gpg: quote path to ${GNUPGHOME}/trustlist.txt
2019-02-08Merge branch 'os/rebase-runs-post-checkout-hook'Junio C Hamano1-1/+1
Test fix. * os/rebase-runs-post-checkout-hook: t5403: correct bash ambiguous redirect error in subtest 8 by quoting $GIT_DIR
2019-02-08test-lib: make '--stress' more bisect-friendlySZEDER Gábor2-2/+21
Let's suppose that a test somehow becomes flaky between 'master' and 'pu', and tends to fail within the first 50 repetitions when run with '--stress'. In such a case we could use 'git bisect' to find the culprit: if the test script fails with '--stress', then the commit is definitely bad, but if it survives, say, 300 repetitions, then we could consider it good with reasonable confidence. Unfortunately, all this could only be done manually, because '--stress' would run the test script repeatedly for all eternity on a good commit, and it would exit with success even when it found a failure on a bad commit. So let's make '--stress' usable with 'git bisect run': - Make it exit with failure if a failure is found. - Add the '--stress-limit=<N>' option to repeat the test script at most N times in each of the parallel jobs, and exit with success when the limit is reached. And then we could simply run something like: $ git bisect start origin/pu master $ git bisect run sh -c 'make && cd t && ./t1234-foo.sh --stress --stress-limit=300' Sure, as a brand new feature it won't be any useful right now, but in a release or three most cooking topics will already contain this, so we could automatically bisect at least newly introduced flakiness. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-08t5403: correct bash ambiguous redirect error in subtest 8 by quoting $GIT_DIRRandall S. Becker1-1/+1
The embedded blanks in the full path of the test git repository cased bash to generate an ambugious redirect error. Signed-off-by: Randall S. Becker <rsbecker@nexbridge.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-08t/lib-gpg: drop redundant killing of gpg-agentTodd Zullinger1-1/+0
In 53fc999306 ("gpg-interface t: extend the existing GPG tests with GPGSM", 2018-07-20), the gpgconf call which kills gpg-agent was copied from the existing gpg setup code. The reason for killing gpg-agent is given in 29ff1f8f74 ("t: lib-gpg: flush gpg agent on startup", 2017-07-20): When running gpg-relevant tests, a gpg-daemon is spawned for each GNUPGHOME used. This daemon may stay running after the test and cache file descriptors for the trash directories, even after the trash directory is removed. This leads to ENOENT errors when attempting to create files if tests are run multiple times. Add a cleanup script to force flushing the gpg-agent for that GNUPGHOME (if any) before setting up the GPG relevant-environment. Killing gpg-agent once per test is sufficient. Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-08t/lib-gpg: quote path to ${GNUPGHOME}/trustlist.txtTodd Zullinger1-1/+1
When gpgsm is installed, lib-gpg.sh attempts to update trustlist.txt to relax the checking of some root certificate requirements. The path to "${GNUPGHOME}" contains spaces which cause an "ambiguous redirect" warning when bash is used to run the tests: $ bash t7030-verify-tag.sh /git/t/lib-gpg.sh: line 66: ${GNUPGHOME}/trustlist.txt: ambiguous redirect ok 1 - create signed tags ok 2 # skip create signed tags x509 (missing GPGSM) ... No warning is issued when using bash called as /bin/sh, dash, or mksh. Quote the path to ensure the redirect works as intended and sets the GPGSM prereq. While we're here, drop the space after ">>". Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-06Merge branch 'ss/describe-dirty-in-the-right-directory'Junio C Hamano1-0/+39
"git --work-tree=$there --git-dir=$here describe --dirty" did not work correctly as it did not pay attention to the location of the worktree specified by the user by mistake, which has been corrected. * ss/describe-dirty-in-the-right-directory: t6120: test for describe with a bare repository describe: setup working tree for --dirty
2019-02-06Merge branch 'jk/loose-object-cache-oid'Junio C Hamano1-1/+1
Code clean-up. * jk/loose-object-cache-oid: prefer "hash mismatch" to "sha1 mismatch" sha1-file: avoid "sha1 file" for generic use in messages sha1-file: prefer "loose object file" to "sha1 file" in messages sha1-file: drop has_sha1_file() convert has_sha1_file() callers to has_object_file() sha1-file: convert pass-through functions to object_id sha1-file: modernize loose header/stream functions sha1-file: modernize loose object file functions http: use struct object_id instead of bare sha1 update comment references to sha1_object_info() sha1-file: fix outdated sha1 comment references
2019-02-06Merge branch 'pw/rebase-x-sanity-check'Junio C Hamano1-0/+19
"git rebase -x $cmd" did not reject multi-line command, even though the command is incapable of handling such a command. It now is rejected upfront. * pw/rebase-x-sanity-check: rebase -x: sanity check command
2019-02-06Merge branch 'js/vsts-ci'Junio C Hamano13-13/+304
Prepare to run test suite on Azure Pipeline. * js/vsts-ci: (22 commits) test-date: drop unused parameter to getnanos() ci: parallelize testing on Windows ci: speed up Windows phase tests: optionally skip bin-wrappers/ t0061: workaround issues with --with-dashes and RUNTIME_PREFIX tests: add t/helper/ to the PATH with --with-dashes mingw: try to work around issues with the test cleanup tests: include detailed trace logs with --write-junit-xml upon failure tests: avoid calling Perl just to determine file sizes README: add a build badge (status of the Azure Pipelines build) mingw: be more generous when wrapping up the setitimer() emulation ci: use git-sdk-64-minimal build artifact ci: add a Windows job to the Azure Pipelines definition Add a build definition for Azure DevOps ci/lib.sh: add support for Azure Pipelines tests: optionally write results as JUnit-style .xml test-date: add a subcommand to measure times in shell scripts ci: use a junction on Windows instead of a symlink ci: inherit --jobs via MAKEFLAGS in run-build-and-tests ci/lib.sh: encapsulate Travis-specific things ...
2019-02-06Merge branch 'br/commit-tree-fully-spelled-gpg-sign-option'Junio C Hamano1-4/+17
The documentation of "git commit-tree" said that the command understands "--gpg-sign" in addition to "-S", but the command line parser did not know about the longhand, which has been corrected. * br/commit-tree-fully-spelled-gpg-sign-option: commit-tree: add missing --gpg-sign flag t7510: invoke git as part of &&-chain
2019-02-06Merge branch 'ds/push-sparse-tree-walk'Junio C Hamano2-0/+140
"git pack-objects" learned another algorithm to compute the set of objects to send, that trades the resulting packfile off to save traversal cost to favor small pushes. * ds/push-sparse-tree-walk: pack-objects: create GIT_TEST_PACK_SPARSE pack-objects: create pack.useSparse setting revision: implement sparse algorithm list-objects: consume sparse tree walk revision: add mark_tree_uninteresting_sparse
2019-02-06Merge branch 'tb/test-lint-sed-options'Junio C Hamano1-1/+1
The test lint learned to catch non-portable "sed" options. * tb/test-lint-sed-options: test-lint: only use only sed [-n] [-e command] [-f command_file]
2019-02-06Merge branch 'lt/date-human'Junio C Hamano2-3/+32
A new date format "--date=human" that morphs its output depending on how far the time is from the current time has been introduced. "--date=auto" can be used to use this new format when the output is going to the pager or to the terminal and otherwise the default format. * lt/date-human: Add `human` date format tests. Add `human` format to test-tool Add 'human' date format documentation Replace the proposed 'auto' mode with 'auto:' Add 'human' date format
2019-02-06Merge branch 'jk/unused-parameter-cleanup'Junio C Hamano1-1/+1
Code cleanup. * jk/unused-parameter-cleanup: convert: drop path parameter from actual conversion functions convert: drop len parameter from conversion checks config: drop unused parameter from maybe_remove_section() show_date_relative(): drop unused "tz" parameter column: drop unused "opts" parameter in item_length() create_bundle(): drop unused "header" parameter apply: drop unused "def" parameter from find_name_gnu() match-trees: drop unused path parameter from score functions
2019-02-06Merge branch 'nd/the-index-final'Junio C Hamano3-2/+4
The assumption to work on the single "in-core index" instance has been reduced from the library-ish part of the codebase. * nd/the-index-final: cache.h: flip NO_THE_INDEX_COMPATIBILITY_MACROS switch read-cache.c: remove the_* from index_has_changes() merge-recursive.c: remove implicit dependency on the_repository merge-recursive.c: remove implicit dependency on the_index sha1-name.c: remove implicit dependency on the_index read-cache.c: replace update_index_if_able with repo_& read-cache.c: kill read_index() checkout: avoid the_index when possible repository.c: replace hold_locked_index() with repo_hold_locked_index() notes-utils.c: remove the_repository references grep: use grep_opt->repo instead of explict repo argument
2019-02-06Merge branch 'tt/bisect-in-c'Junio C Hamano1-1/+1
More code in "git bisect" has been rewritten in C. * tt/bisect-in-c: bisect--helper: `bisect_start` shell function partially in C bisect--helper: `get_terms` & `bisect_terms` shell function in C bisect--helper: `bisect_next_check` shell function in C bisect--helper: `check_and_set_terms` shell function in C wrapper: move is_empty_file() and rename it as is_empty_or_missing_file() bisect--helper: `bisect_write` shell function in C bisect--helper: `bisect_reset` shell function in C
2019-02-06Merge branch 'tb/utf-16-le-with-explicit-bom'Junio C Hamano1-1/+11
A new encoding UTF-16LE-BOM has been invented to force encoding to UTF-16 with BOM in little endian byte order, which cannot be directly generated by using iconv. * tb/utf-16-le-with-explicit-bom: Support working-tree-encoding "UTF-16LE-BOM"
2019-02-06Merge branch 'dt/cat-file-batch-ambiguous'Junio C Hamano1-0/+10
"git cat-file --batch" reported a dangling symbolic link by mistake, when it wanted to report that a given name is ambiguous. * dt/cat-file-batch-ambiguous: t1512: test ambiguous cat-file --batch and --batch-output Do not print 'dangling' for cat-file in case of ambiguity
2019-02-06Merge branch 'en/rebase-merge-on-sequencer'Junio C Hamano6-93/+53
"git rebase --merge" as been reimplemented by reusing the internal machinery used for "git rebase -i". * en/rebase-merge-on-sequencer: rebase: implement --merge via the interactive machinery rebase: define linearization ordering and enforce it git-legacy-rebase: simplify unnecessary triply-nested if git-rebase, sequencer: extend --quiet option for the interactive machinery am, rebase--merge: do not overlook --skip'ed commits with post-rewrite t5407: add a test demonstrating how interactive handles --skip differently rebase: fix incompatible options error message rebase: make builtin and legacy script error messages the same
2019-02-06git-p4: remove ticket expiry testLuke Diamand1-27/+0
The git-p4 login ticket expiry test causes unreliable test runs. Since the handling of ticket expiry in git-p4 is far from polished anyway, let's remove it for now. A better way to actually run the test is to create a python "fake" version of "p4" which returns whatever expiry results the test requires. Ideally git-p4 would look at the expiry time before starting any long operations, and cleanup gracefully if there is not enough time left. But that's quite hard to do. Signed-off-by: Luke Diamand <luke@diamand.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-06t5551: test server-side ERR packetJosh Steadmon4-0/+13
When a smart HTTP server sends an error message via pkt-line, we detect the error due to using PACKET_READ_DIE_ON_ERR_PACKET. This case was added by 2d103c31c2 (pack-protocol.txt: accept error packets in any context, 2018-12-29), but not covered by tests. Signed-off-by: Josh Steadmon <steadmon@google.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-06test-date: drop unused parameter to getnanos()Jeff King1-2/+2
The getnanos() helper always gets the current time from our getnanotime() facility. The caller cannot override it via TEST_DATE_NOW, and hence we simply ignore the "now" parameter to the function. Let's remove it, as it may mislead callers into thinking it does something. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-06Revert "rebase: introduce a shortcut for --reschedule-failed-exec"Johannes Schindelin1-3/+0
This patch was contributed only as a tentative "we could introduce a convenient short option if we do not want to change the default behavior in the long run" patch, opening the discussion whether other people agree with deprecating the current behavior in favor of the rescheduling behavior. But the consensus on the Git mailing list was that it would make sense to show a warning in the near future, and flip the default rebase.rescheduleFailedExec to reschedule failed `exec` commands by default. See e.g. <CAGZ79kZL5CRqCDRb6B-EedUm8Z_i4JuSF2=UtwwdRXMitrrOBw@mail.gmail.com> So let's back out that patch that added the `-y` short option that we agreed was not necessary or desirable. This reverts commit 81ef8ee75d5f348d3c71ff633d13d302124e1a5e. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-05Merge branch 'pw/no-editor-in-rebase-i-implicit'Junio C Hamano3-6/+11
When GIT_SEQUENCE_EDITOR is set, the command was incorrectly started when modes of "git rebase" that implicitly uses the machinery for the interactive rebase are run, which has been corrected. * pw/no-editor-in-rebase-i-implicit: implicit interactive rebase: don't run sequence editor
2019-02-05Merge branch 'jk/diff-cc-stat-fixes'Junio C Hamano6-23/+133
"git diff --color-moved --cc --stat -p" did not work well due to funny interaction between a bug in color-moved and the rest, which has been fixed. * jk/diff-cc-stat-fixes: combine-diff: treat --dirstat like --stat combine-diff: treat --summary like --stat combine-diff: treat --shortstat like --stat combine-diff: factor out stat-format mask diff: clear emitted_symbols flag after use t4006: resurrect commented-out tests
2019-02-05Merge branch 'bp/checkout-new-branch-optim'Junio C Hamano1-0/+9
"git checkout -b <new> [HEAD]" to create a new branch from the current commit and check it out ought to be a no-op in the index and the working tree in normal cases, but there are corner cases that do require updates to the index and the working tree. Running it immediately after "git clone --no-checkout" is one of these cases that an earlier optimization kicked in incorrectly, which has been fixed. * bp/checkout-new-branch-optim: checkout: fix regression in checkout -b on intitial checkout checkout: add test demonstrating regression with checkout -b on initial commit
2019-02-05Merge branch 'jk/attr-macro-fix'Junio C Hamano1-0/+20
Asking "git check-attr" about a macro (e.g. "binary") on a specific path did not work correctly, even though "git check-attr -a" listed such a macro correctly. This has been corrected. * jk/attr-macro-fix: attr: do not mark queried macros as unset
2019-02-05Merge branch 'js/test-git-installed'Junio C Hamano1-1/+1
Test fix for Windows. * js/test-git-installed: tests: explicitly use `test-tool.exe` on Windows
2019-02-05Merge branch 'js/abspath-part-inside-repo'Junio C Hamano1-0/+7
On a case-insensitive filesystem, we failed to compare the part of the path that is above the worktree directory in an absolute pathname, which has been corrected. * js/abspath-part-inside-repo: abspath_part_inside_repo: respect core.ignoreCase
2019-02-05Merge branch 'jt/namespaced-ls-refs-fix'Junio C Hamano1-0/+21
Fix namespace support in protocol v2. * jt/namespaced-ls-refs-fix: ls-refs: filter refs using namespace-stripped name
2019-02-05Merge branch 'ab/commit-graph-write-progress'Junio C Hamano1-7/+7
The codepath to show progress meter while writing out commit-graph file has been improved. * ab/commit-graph-write-progress: commit-graph write: emit a percentage for all progress commit-graph write: add itermediate progress commit-graph write: remove empty line for readability commit-graph write: add more descriptive progress output commit-graph write: show progress for object search commit-graph write: more descriptive "writing out" output commit-graph write: add "Writing out" progress output commit-graph: don't call write_graph_chunk_extra_edges() unnecessarily commit-graph: rename "large edges" to "extra edges"
2019-02-05Merge branch 'js/t6042-timing-fix'Junio C Hamano1-2/+2
Test update. * js/t6042-timing-fix: t6042: work around speed optimization on Windows
2019-02-05Merge branch 'jk/add-ignore-errors-bit-assignment-fix'Junio C Hamano1-0/+9
"git add --ignore-errors" did not work as advertised and instead worked as an unintended synonym for "git add --renormalize", which has been fixed. * jk/add-ignore-errors-bit-assignment-fix: add: use separate ADD_CACHE_RENORMALIZE flag
2019-02-05Merge branch 'js/mingw-unc-path-w-backslashes'Junio C Hamano2-0/+15
In Git for Windows, "git clone \\server\share\path" etc. that uses UNC paths from command line had bad interaction with its shell emulation. * js/mingw-unc-path-w-backslashes: mingw: special-case arguments to `sh` mingw (t5580): document bug when cloning from backslashed UNC paths
2019-02-05Merge branch 'cc/test-ref-store-typofix'Junio C Hamano1-1/+1
An obvious typo in an assertion error message has been fixed. * cc/test-ref-store-typofix: helper/test-ref-store: fix "new-sha1" vs "old-sha1" typo
2019-02-05Merge branch 'jt/fetch-v2-sideband'Junio C Hamano6-6/+13
"git fetch" and "git upload-pack" learned to send all exchange over the sideband channel while talking the v2 protocol. * jt/fetch-v2-sideband: tests: define GIT_TEST_SIDEBAND_ALL {fetch,upload}-pack: sideband v2 fetch response sideband: reverse its dependency on pkt-line pkt-line: introduce struct packet_writer pack-protocol.txt: accept error packets in any context Use packet_reader instead of packet_read_line
2019-02-05Merge branch 'js/commit-graph-chunk-table-fix'Junio C Hamano1-3/+13
The codepath to read from the commit-graph file attempted to read past the end of it when the file's table-of-contents was corrupt. * js/commit-graph-chunk-table-fix: Makefile: correct example fuzz build commit-graph: fix buffer read-overflow commit-graph, fuzz: add fuzzer for commit-graph
2019-02-05Merge branch 'ld/git-p4-shelve-update-fix'Junio C Hamano1-3/+54
"git p4" failed to update a shelved change when there were moved files, which has been corrected. * ld/git-p4-shelve-update-fix: git-p4: handle update of moved/copied files when updating a shelve git-p4: add failing test for shelved CL update involving move/copy
2019-02-05Merge branch 'js/filter-options-should-use-plain-int'Junio C Hamano1-1/+138
Update the protocol message specification to allow only the limited use of scaled quantities. This is ensure potential compatibility issues will not go out of hand. * js/filter-options-should-use-plain-int: filter-options: expand scaled numbers tree:<depth>: skip some trees even when collecting omits list-objects-filter: teach tree:# how to handle >0
2019-02-05Merge branch 'sb/more-repo-in-api'Junio C Hamano1-0/+10
The in-core repository instances are passed through more codepaths. * sb/more-repo-in-api: (23 commits) t/helper/test-repository: celebrate independence from the_repository path.h: make REPO_GIT_PATH_FUNC repository agnostic commit: prepare free_commit_buffer and release_commit_memory for any repo commit-graph: convert remaining functions to handle any repo submodule: don't add submodule as odb for push submodule: use submodule repos for object lookup pretty: prepare format_commit_message to handle arbitrary repositories commit: prepare logmsg_reencode to handle arbitrary repositories commit: prepare repo_unuse_commit_buffer to handle any repo commit: prepare get_commit_buffer to handle any repo commit-reach: prepare in_merge_bases[_many] to handle any repo commit-reach: prepare get_merge_bases to handle any repo commit-reach.c: allow get_merge_bases_many_0 to handle any repo commit-reach.c: allow remove_redundant to handle any repo commit-reach.c: allow merge_bases_many to handle any repo commit-reach.c: allow paint_down_to_common to handle any repo commit: allow parse_commit* to handle any repo object: parse_object to honor its repository argument object-store: prepare has_{sha1, object}_file to handle any repo object-store: prepare read_object_file to deal with any repo ...
2019-02-04t6120: test for describe with a bare repositorySebastian Staudt1-0/+6
This ensures that nothing breaks the basic functionality of describe for bare repositories. Please note that --broken and --dirty need a working tree. Signed-off-by: Sebastian Staudt <koraktor@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-02-04describe: setup working tree for --dirtySebastian Staudt1-0/+33
We don't use NEED_WORK_TREE when running the git-describe builtin, since you should be able to describe a commit even in a bare repository. However, the --dirty flag does need a working tree. Since we don't call setup_work_tree(), it uses whatever directory we happen to be in. That's unlikely to match our index, meaning we'd say "dirty" even when the real working tree is clean. We can fix that by calling setup_work_tree() once we know that the user has asked for --dirty. The --broken option also needs a working tree. But because its implementation calls git-diff-index we don‘t have to setup the working tree in the git-describe process. Signed-off-by: Sebastian Staudt <koraktor@gmail.com> Helped-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-31t1512: test ambiguous cat-file --batch and --batch-outputEric Wong1-0/+10
Test the new "ambiguous" result from cat-file --batch and --batch-check. This is in t1512 instead of t1006 since we need a repo with ambiguous object_id names. Signed-off-by: Eric Wong <e@80x24.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-31Support working-tree-encoding "UTF-16LE-BOM"Torsten Bögershausen1-1/+11
Users who want UTF-16 files in the working tree set the .gitattributes like this: test.txt working-tree-encoding=UTF-16 The unicode standard itself defines 3 allowed ways how to encode UTF-16. The following 3 versions convert all back to 'g' 'i' 't' in UTF-8: a) UTF-16, without BOM, big endian: $ printf "\000g\000i\000t" | iconv -f UTF-16 -t UTF-8 | od -c 0000000 g i t b) UTF-16, with BOM, little endian: $ printf "\377\376g\000i\000t\000" | iconv -f UTF-16 -t UTF-8 | od -c 0000000 g i t c) UTF-16, with BOM, big endian: $ printf "\376\377\000g\000i\000t" | iconv -f UTF-16 -t UTF-8 | od -c 0000000 g i t Git uses libiconv to convert from UTF-8 in the index into ITF-16 in the working tree. After a checkout, the resulting file has a BOM and is encoded in "UTF-16", in the version (c) above. This is what iconv generates, more details follow below. iconv (and libiconv) can generate UTF-16, UTF-16LE or UTF-16BE: d) UTF-16 $ printf 'git' | iconv -f UTF-8 -t UTF-16 | od -c 0000000 376 377 \0 g \0 i \0 t e) UTF-16LE $ printf 'git' | iconv -f UTF-8 -t UTF-16LE | od -c 0000000 g \0 i \0 t \0 f) UTF-16BE $ printf 'git' | iconv -f UTF-8 -t UTF-16BE | od -c 0000000 \0 g \0 i \0 t There is no way to generate version (b) from above in a Git working tree, but that is what some applications need. (All fully unicode aware applications should be able to read all 3 variants, but in practise we are not there yet). When producing UTF-16 as an output, iconv generates the big endian version with a BOM. (big endian is probably chosen for historical reasons). iconv can produce UTF-16 files with little endianess by using "UTF-16LE" as encoding, and that file does not have a BOM. Not all users (especially under Windows) are happy with this. Some tools are not fully unicode aware and can only handle version (b). Today there is no way to produce version (b) with iconv (or libiconv). Looking into the history of iconv, it seems as if version (c) will be used in all future iconv versions (for compatibility reasons). Solve this dilemma and introduce a Git-specific "UTF-16LE-BOM". libiconv can not handle the encoding, so Git pick it up, handles the BOM and uses libiconv to convert the rest of the stream. (UTF-16BE-BOM is added for consistency) Rported-by: Adrián Gimeno Balaguer <adrigibal@gmail.com> Signed-off-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29rebase -x: sanity check commandPhillip Wood1-0/+19
If the user gives an empty argument to --exec then git creates a todo list that it cannot parse. The rebase starts to run before erroring out with error: missing arguments for exec error: invalid line 2: exec You can fix this with 'git rebase --edit-todo' and then run 'git rebase --continue'. Or you can abort the rebase with 'git rebase --abort'. Instead check for empty commands before starting the rebase. Also check that the command does not contain any newlines as the todo-list format is unable to cope with multiline commands. Note that this changes the behavior, before this change one could do git rebase --exec='echo one exec echo two' and it would insert two exec lines in the todo list, now it will error out. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29Merge branch 'it/log-format-source'Junio C Hamano2-0/+54
Custom userformat "log --format" learned %S atom that stands for the tip the traversal reached the commit from, i.e. --source. * it/log-format-source: log: add %S option (like --source) to log --format
2019-01-29Merge branch 'jt/upload-pack-deepen-relative-proto-v2'Junio C Hamano1-0/+47
"git fetch --deepen=<more>" has been corrected to work over v2 protocol. * jt/upload-pack-deepen-relative-proto-v2: upload-pack: teach deepen-relative in protocol v2 fetch-pack: do not take shallow lock unnecessarily
2019-01-29Merge branch 'ms/http-no-more-failonerror'Junio C Hamano2-0/+29
Debugging help for http transport. * ms/http-no-more-failonerror: test: test GIT_CURL_VERBOSE=1 shows an error remote-curl: unset CURLOPT_FAILONERROR remote-curl: define struct for CURLOPT_WRITEFUNCTION http: enable keep_error for HTTP requests http: support file handles for HTTP_KEEP_ERROR
2019-01-29Merge branch 'os/rebase-runs-post-checkout-hook'Junio C Hamano1-54/+42
"git rebase" internally runs "checkout" to switch between branches, and the command used to call the post-checkout hook, but the reimplementation stopped doing so, which is getting fixed. * os/rebase-runs-post-checkout-hook: rebase: run post-checkout hook on checkout t5403: simplify by using a single repository
2019-01-29Merge branch 'bc/sha-256'Junio C Hamano7-51/+188
Add sha-256 hash and plug it through the code to allow building Git with the "NewHash". * bc/sha-256: hash: add an SHA-256 implementation using OpenSSL sha256: add an SHA-256 implementation using libgcrypt Add a base implementation of SHA-256 support commit-graph: convert to using the_hash_algo t/helper: add a test helper to compute hash speed sha1-file: add a constant for hash block size t: make the sha1 test-tool helper generic t: add basic tests for our SHA-1 implementation cache: make hashcmp and hasheq work with larger hashes hex: introduce functions to print arbitrary hashes sha1-file: provide functions to look up hash algorithms sha1-file: rename algorithm to "sha1"
2019-01-29Merge branch 'sb/submodule-recursive-fetch-gets-the-tip'Junio C Hamano2-3/+122
"git fetch --recurse-submodules" may not fetch the necessary commit that is bound to the superproject, which is getting corrected. * sb/submodule-recursive-fetch-gets-the-tip: fetch: ensure submodule objects fetched submodule.c: fetch in submodules git directory instead of in worktree submodule: migrate get_next_submodule to use repository structs repository: repo_submodule_init to take a submodule struct submodule: store OIDs in changed_submodule_names submodule.c: tighten scope of changed_submodule_names struct submodule.c: sort changed_submodule_names before searching it submodule.c: fix indentation sha1-array: provide oid_array_filter
2019-01-29Merge branch 'jt/fetch-pack-v2'Junio C Hamano1-7/+15
"git fetch-pack" now can talk the version 2 protocol. * jt/fetch-pack-v2: fetch-pack: support protocol version 2
2019-01-29Merge branch 'jk/proto-v2-hidden-refs-fix'Junio C Hamano1-0/+6
The v2 upload-pack protocol implementation failed to honor hidden-ref configuration, which has been corrected. An earlier attempt reverted out of 'next'. * jk/proto-v2-hidden-refs-fix: upload-pack: support hidden refs with protocol v2
2019-01-29Merge branch 'pw/diff-color-moved-ws-fix'Junio C Hamano1-3/+96
"git diff --color-moved-ws" updates. * pw/diff-color-moved-ws-fix: diff --color-moved-ws: handle blank lines diff --color-moved-ws: modify allow-indentation-change diff --color-moved-ws: optimize allow-indentation-change diff --color-moved=zebra: be stricter with color alternation diff --color-moved-ws: fix false positives diff --color-moved-ws: demonstrate false positives diff: allow --no-color-moved-ws Use "whitespace" consistently diff: document --no-color-moved
2019-01-29Merge branch 'js/rebase-i-redo-exec'Junio C Hamano1-0/+14
"git rebase -i" learned to re-execute a command given with 'exec' to run after it failed the last time. * js/rebase-i-redo-exec: rebase: introduce a shortcut for --reschedule-failed-exec rebase: add a config option to default to --reschedule-failed-exec rebase: introduce --reschedule-failed-exec
2019-01-29Add `human` date format tests.Stephen P. Smith1-0/+18
When using `human` several fields are suppressed depending on the time difference between the reference date and the local computer date. In cases where the difference is less than a year, the year field is supppressed. If the time is less than a day; the month and year is suppressed. Use TEST_DATE_NOW environment variable when using the test-tool to hold the expected output strings constant. Signed-off-by: Stephen P. Smith <ischis2@cox.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29Add `human` format to test-toolStephen P. Smith2-3/+14
Add the human format support to the test tool so that GIT_TEST_DATE_NOW can be used to specify the current time. The get_time() helper function was created and and checks the GIT_TEST_DATE_NOW environment variable. If GIT_TEST_DATE_NOW is set, then that date is used instead of the date returned by by gettimeofday(). All calls to gettimeofday() were replaced by calls to get_time(). Renamed occurances of TEST_DATE_NOW to GIT_TEST_DATE_NOW since the variable is now used in the get binary and not just in the test-tool. Signed-off-by: Stephen P. Smith <ischis2@cox.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29ci: parallelize testing on WindowsJohannes Schindelin1-0/+31
The fact that Git's test suite is implemented in Unix shell script that is as portable as we can muster, combined with the fact that Unix shell scripting is foreign to Windows (and therefore has to be emulated), results in pretty abysmal speed of the test suite on that platform, for pretty much no other reason than that language choice. For comparison: while the Linux build & test is typically done within about 8 minutes, the Windows build & test typically lasts about 80 minutes in Azure Pipelines. To help with that, let's use the Azure Pipeline feature where you can parallelize jobs, make jobs depend on each other, and pass artifacts between them. The tests are distributed using the following heuristic: listing all test scripts ordered by size in descending order (as a cheap way to estimate the overall run time), every Nth script is run (where N is the total number of parallel jobs), starting at the index corresponding to the parallel job. This slicing is performed by a new function that is added to the `test-tool`. To optimize the overall runtime of the entire Pipeline, we need to move the Windows jobs to the beginning (otherwise there would be a very decent chance for the Pipeline to be run only the Windows build, while all the parallel Windows test jobs wait for this single one). We use Azure Pipelines Artifacts for both the minimal Git for Windows SDK as well as the built executables, as deduplication and caching close to the agents makes that really fast. For comparison: while downloading and unpacking the minimal Git for Windows SDK via PowerShell takes only one minute (down from anywhere between 2.5 to 7 when using a shallow clone), uploading it as Pipeline Artifact takes less than 30s and downloading and unpacking less than 20s (sometimes even as little as only twelve seconds). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: optionally skip bin-wrappers/Johannes Schindelin2-6/+22
This speeds up the tests by a bit on Windows, where running Unix shell scripts (and spawning processes) is not exactly a cheap operation. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29t0061: workaround issues with --with-dashes and RUNTIME_PREFIXJohannes Schindelin1-1/+2
When building Git with RUNTIME_PREFIX and starting a test helper from t/helper/, it fails to detect a system prefix. The reason is that the RUNTIME_PREFIX feature wants to use the location of the Git executable to determine where the support files can be found, e.g. system-wide Git config or the translations. This does not make any sense for the test helpers, though, as they are distinctly not in a directory structure resembling the final installation location of Git. That is the reason why the test helpers rely on environment variables to indicate the location of the needed support files, e.g. GIT_TEXTDOMAINDIR. If this information is missing, the output will contain warnings like this one: RUNTIME_PREFIX requested, but prefix computation failed. [...] In t0061, we did not expect that to happen, and it actually does not happen in the regular case, because bin-wrappers/test-tool specifically sets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool needs to know anything about any runtime prefix). However, with --with-dashes, bin-wrappers/test-tool is no longer called, but t/helper/test-tool is called directly instead. So let's just ignore the RUNTIME_PREFIX warning. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: add t/helper/ to the PATH with --with-dashesJohannes Schindelin1-1/+1
We really need to be able to find the test helpers... Really. This change was forgotten when we moved the test helpers into t/helper/ Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29mingw: try to work around issues with the test cleanupJohannes Schindelin1-1/+5
It seems that every once in a while in the Git for Windows SDK, there are some transient file locking issues preventing the test clean up to delete the trash directory. Let's be gentle and try again five seconds later, and only error out if it still fails the second time. This change helps Windows, and does not hurt any other platform (normally, it is highly unlikely that said deletion fails, and if it does, normally it will fail again even 5 seconds later). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: include detailed trace logs with --write-junit-xml upon failureJohannes Schindelin2-1/+42
The JUnit XML format lends itself to be presented in a powerful UI, where you can drill down to the information you are interested in very quickly. For test failures, this usually means that you want to see the detailed trace of the failing tests. With Travis CI, we passed the `--verbose-log` option to get those traces. However, that seems excessive, as we do not need/use the logs in almost all of those cases: only when a test fails do we have a way to include the trace. So let's do something different when using Azure DevOps: let's run all the tests with `--quiet` first, and only if a failure is encountered, try to trace the commands as they are executed. Of course, we cannot turn on `--verbose-log` after the fact. So let's just re-run the test with all the same options, adding `--verbose-log`. And then munging the output file into the JUnit XML on the fly. Note: there is an off chance that re-running the test in verbose mode "fixes" the failures (and this does happen from time to time!). That is a possibility we should be able to live with. Ideally, we would label this as "Passed upon rerun", and Azure Pipelines even know about that outcome, but it is not available when using the JUnit XML format for now: https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: avoid calling Perl just to determine file sizesJohannes Schindelin5-4/+16
It is a bit ridiculous to spin up a full-blown Perl instance (especially on Windows, where that means spinning up a full POSIX emulation layer, AKA the MSYS2 runtime) just to tell how large a given file is. So let's just use the test-tool to do that job instead. This command will also be used over the next commits, to allow for cutting out individual test cases' verbose log from the file generated via --verbose-log. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29tests: optionally write results as JUnit-style .xmlJohannes Schindelin5-0/+174
This will come in handy when publishing the results of Git's test suite during an automated Azure DevOps run. Note: we need to make extra sure that invalid UTF-8 encoding is turned into valid UTF-8 (using the Replacement Character, \uFFFD) because t9902's trace contains such invalid byte sequences, and the task in the Azure Pipeline that uploads the test results would refuse to do anything if it was asked to parse an .xml file with invalid UTF-8 in it. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-28test-lint: only use only sed [-n] [-e command] [-f command_file]Torsten Bögershausen1-1/+1
From `man sed` (on a Mac OS X box): The -E, -a and -i options are non-standard FreeBSD extensions and may not be available on other operating systems. From `man sed` on a Linux box: REGULAR EXPRESSIONS POSIX.2 BREs should be supported, but they aren't completely because of performance problems. The \n sequence in a regular expression matches the newline character, and similarly for \a, \t, and other sequences. The -E option switches to using extended regular expressions instead; the -E option has been supported for years by GNU sed, and is now included in POSIX. Well, there are still a lot of systems out there, which don't support it. Beside that, IEEE Std 1003.1TM-2017, see http://pubs.opengroup.org/onlinepubs/9699919799/ does not mention -E either. To be on the safe side, don't allow -E (or -r, which is GNU). Change check-non-portable-shell.pl to only accept the portable options: sed [-n] [-e command] [-f command_file] Reported-by: SZEDER Gábor <szeder.dev@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-28test-date: add a subcommand to measure times in shell scriptsJohannes Schindelin1-0/+12
In the next commit, we want to teach Git's test suite to optionally output test results in JUnit-style .xml files. These files contain information about the time spent. So we need a way to measure time. While we could use `date +%s` for that, this will give us only seconds, i.e. very coarse-grained timings. GNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output), but there is no equivalent in BSD `date` (read: on macOS, we would not be able to obtain precise timings). So let's introduce `test-tool date getnanos`, with an optional start time, that outputs preciser values. Note that this might not actually give us nanosecond precision on some platforms, but it will give us as precise information as possible, without the portability issues of shell commands. Granted, it is a bit pointless to try measuring times accurately in shell scripts, certainly to nanosecond precision. But it is better than second-granularity. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-28implicit interactive rebase: don't run sequence editorPhillip Wood3-6/+11
If GIT_SEQUENCE_EDITOR is set then rebase runs it when executing implicit interactive rebases which are supposed to appear non-interactive to the user. Fix this by setting GIT_SEQUENCE_EDITOR=: rather than GIT_EDITOR=:. A couple of tests relied on the old behavior so they are updated to work with the new regime. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-24show_date_relative(): drop unused "tz" parameterJeff King1-1/+1
The timestamp we receive is in epoch time, so there's no need for a timezone parameter to interpret it. The matching show_date() uses "tz" to show dates in author local time, but relative dates show only the absolute time difference. The author's location is irrelevant, barring relativistic effects from using Git close to the speed of light. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-24combine-diff: treat --dirstat like --statJeff King2-0/+4
Currently "--cc --dirstat" will show nothing for a merge. Like --shortstat and --summary in the previous two patches, it probably makes sense to treat it like we do --stat, and show a stat against the first-parent. This case is less obviously correct than for --shortstat and --summary, as those are basically variants of --stat themselves. It's possible we could develop a multi-parent combined dirstat format, in which case we might regret defining this first-parent behavior. But the same could be said for --stat, and in the 12+ years of it showing first-parent stats, nobody has complained. So showing the first-parent dirstat is at least _useful_, and if we later develop a clever multi-parent stat format, we'd probably have to deal with --stat anyway. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-24combine-diff: treat --summary like --statJeff King2-0/+13
Currently "--cc --summary" on a merge shows nothing. Since we show "--cc --stat" as a stat against the first parent, and because --summary is typically used in combination with --stat, it makes sense to treat them both the same way. Note that we have to tweak t4013's setup a bit to test this case, as the existing merges do not have any --summary results against their first parent. But since the merge at the tip of 'master' does add and remove files with respect to the second parent, we can just make a reversed doppelganger merge where the parents are swapped. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-24combine-diff: treat --shortstat like --statJeff King2-0/+5
The --stat of a combined diff is defined as the first-parent stat, going all the way back to 965f803c32 (combine-diff: show diffstat with the first parent., 2006-04-17). Naturally, we gave --numstat the same treatment in 74e2abe5b7 (diff --numstat, 2006-10-12). But --shortstat, which is really just the final line of --stat, does nothing, which produces confusing results: $ git show --oneline --stat eab7584e37 eab7584e37 Merge branch 'en/show-ref-doc-fix' Documentation/git-show-ref.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) $ git show --oneline --shortstat eab7584e37 eab7584e37 Merge branch 'en/show-ref-doc-fix' [nothing! We'd expect to see the "1 file changed..." line] This patch teaches combine-diff to treats the two formats identically. Reported-by: David Turner <novalis@novalis.org> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-24diff: clear emitted_symbols flag after useJeff King1-0/+79
There's an odd bug when "log --color-moved" is used with the combination of "--cc --stat -p": the stat for merge commits is erroneously shown with the diff of the _next_ commit. The included test demonstrates the issue. Our history looks something like this: A-B-M--D \ / C When we run "git log --cc --stat -p --color-moved" starting at D, we get this sequence of events: 1. The diff for D is using -p, so diff_flush() calls into diff_flush_patch_all_file_pairs(). There we see that o->color_moved is in effect, so we point o->emitted_symbols to a static local struct, causing diff_flush_patch() to queue the symbols instead of actually writing them out. We then do our move detection, emit the symbols, and clear the struct. But we leave o->emitted_symbols pointing to our struct. 2. Next we compute the diff for M. This is a merge, so we use the combined diff code. In find_paths_generic(), we compute the pairwise diff between each commit and its parent. Normally this is done with DIFF_FORMAT_NO_OUTPUT, since we're just looking for intersecting paths. But since "--stat --cc" shows the first-parent stat, and since we're computing that diff anyway, we enable DIFF_FORMAT_DIFFSTAT for the first parent. This outputs the stat information immediately, saving us from running a separate first-parent diff later. But where does that output go? Normally it goes directly to stdout, but because o->emitted_symbols is set, we queue it. As a result, we don't actually print the diffstat for the merge commit (yet), which is wrong. 3. Next we compute the diff for C. We're actually showing a patch again, so we end up in diff_flush_patch_all_file_pairs(), but this time we have the queued stat from step 2 waiting in our struct. We add new elements to it for C's diff, and then flush the whole thing. And we see the diffstat from M as part of C's diff, which is wrong. So triggering the bug really does require the combination of all of those options. To fix it, we can simply restore o->emitted_symbols to NULL after flushing it, so that it does not affect anything outside of diff_flush_patch_all_file_pairs(). This intuitively makes sense, since nobody outside of that function is going to bother flushing it, so we would not want them to write to it either. In fact, we could take this a step further and turn the local "esm" struct into a non-static variable that goes away after the function ends. However, since it contains a dynamically sized array, we benefit from amortizing the cost of allocations over many calls. So we'll leave it as static to retain that benefit. But let's push the zero-ing of esm.nr into the conditional for "if (o->emitted_symbols)" to make it clear that we do not expect esm to hold any values if we did not just try to use it. With the code as it is written now, if we did encounter such a case (which I think would be a bug), we'd silently leak those values without even bothering to display them. With this change, we'd at least eventually show them, and somebody would notice. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-24t4006: resurrect commented-out testsJeff King1-23/+32
This set of tests was added by 4434e6ba6c (tests: check --[short]stat output after chmod, 2012-05-01), and is primarily about the handling of binary versus text files. Later, 74faaa16f0 (Fix "git diff --stat" for interesting - but empty - file changes, 2012-10-17) changed the stat output so that the empty text file is mentioned rather than omitted. That commit just comments out these tests. There's no discussion in the commit message, but the original email[1] says: NOTE! This does break two of our tests, so we clearly did this on purpose, or at least tested for it. I just uncommented the subtests that this makes irrelevant, and changed the output of another one. I don't think they're irrelevant, though. We should be testing this "mode change only" case and making sure that it has the post-74faaa16f0 behavior. So this commit brings back those tests, with the current expected output. [1] https://public-inbox.org/git/CA+55aFz88GPJcfMSqiyY+u0Cdm48bEyrsTGxHVJbGsYsDg=Q5w@mail.gmail.com/ Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-24cache.h: flip NO_THE_INDEX_COMPATIBILITY_MACROS switchNguyễn Thái Ngọc Duy3-2/+4
By default, index compat macros are off from now on, because they could hide the_index dependency. Only those in builtin can use it. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-23checkout: fix regression in checkout -b on intitial checkoutBen Peart1-1/+1
When doing a 'checkout -b' do a full checkout including updating the working tree when doing the initial checkout. As the new test involves an filesystem access, do it later in the sequence to give chance to other cheaper tests to leave early. This fixes the regression in behavior caused by fa655d8411 (checkout: optimize "git checkout -b <new_branch>", 2018-08-16). Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-23checkout: add test demonstrating regression with checkout -b on initial commitBen Peart1-0/+9
Commit fa655d8411 (checkout: optimize "git checkout -b <new_branch>", 2018-08-16) introduced an unintentional change in behavior for 'checkout -b' after doing 'clone --no-checkout'. Add a test to demonstrate the changed behavior to be used in a later patch to verify the fix. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-22attr: do not mark queried macros as unsetJeff King1-0/+20
Since 60a12722ac (attr: remove maybe-real, maybe-macro from git_attr, 2017-01-27), we will always mark an attribute macro (e.g., "binary") that is specifically queried for as "unspecified", even though listing _all_ attributes would display it at set. E.g.: $ echo "* binary" >.gitattributes $ git check-attr -a file file: binary: set file: diff: unset file: merge: unset file: text: unset $ git check-attr binary file file: binary: unspecified The problem stems from an incorrect conversion of the optimization from 06a604e670 (attr: avoid heavy work when we know the specified attr is not defined, 2014-12-28). There we tried in collect_some_attrs() to avoid even looking at the attr_stack when the user has asked for "foo" and we know that "foo" did not ever appear in any .gitattributes file. It used a flag "maybe_real" in each attribute struct, where "real" meant that the attribute appeared in an actual file (we have to make this distinction because we also create an attribute struct for any names that are being queried). But as explained in that commit message, the meaning of "real" was tangled with some special cases around macros. When 60a12722ac later refactored the macro code, it dropped maybe_real entirely. This missed the fact that "maybe_real" could be unset for two reasons: because of a macro, or because it was never found during parsing. This had two results: - the optimization in collect_some_attrs() ceased doing anything meaningful, since it no longer kept track of "was it found during parsing" - worse, it actually kicked in when the caller _did_ ask about a macro by name, causing us to mark it as unspecified It should be possible to salvage this optimization, but let's start with just removing the remnants. It hasn't been doing anything (except creating bugs) since 60a12722ac, and nobody seems to have noticed the performance regression. It's more important to fix the correctness problem clearly first. I've added two tests here. The second one actually shows off the bug. The test of "check-attr -a" is not strictly necessary, but we currently do not test attribute macros much, and the builtin "binary" not at all. So this increases our general test coverage, as well as making sure we didn't mess up this related case. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-22tests: explicitly use `test-tool.exe` on WindowsJohannes Schindelin1-1/+1
In 8abfdf44c882 (tests: explicitly use `git.exe` on Windows, 2018-11-14), we made sure to use the `.exe` file extension when using an absolute path to `git.exe`, to avoid getting confused with a file or directory in the same place that lacks said file extension. For the same reason, we need to handle test-tool.exe the same way. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-22commit-graph: rename "large edges" to "extra edges"SZEDER Gábor1-7/+7
The optional 'Large Edge List' chunk of the commit graph file stores parent information for commits with more than two parents, and the names of most of the macros, variables, struct fields, and functions related to this chunk contain the term "large edges", e.g. write_graph_chunk_large_edges(). However, it's not a really great term, as the edges to the second and subsequent parents stored in this chunk are not any larger than the edges to the first and second parents stored in the "main" 'Commit Data' chunk. It's the number of edges, IOW number of parents, that is larger compared to non-merge and "regular" two-parent merge commits. And indeed, two functions in 'commit-graph.c' have a local variable called 'num_extra_edges' that refer to the same thing, and this "extra edges" term is much better at describing these edges. So let's rename all these references to "large edges" in macro, variable, function, etc. names to "extra edges". There is a GRAPH_OCTOPUS_EDGES_NEEDED macro as well; for the sake of consistency rename it to GRAPH_EXTRA_EDGES_NEEDED. We can do so safely without causing any incompatibility issues, because the term "large edges" doesn't come up in the file format itself in any form (the chunk's magic is {'E', 'D', 'G', 'E'}, there is no 'L' in there), but only in the specification text. The string "large edges", however, does come up in the output of 'git commit-graph read' and in tests looking at its input, but that command is explicitly documented as debugging aid, so we can change its output and the affected tests safely. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-22commit-tree: add missing --gpg-sign flagBrandon Richardson1-3/+12
Add --gpg-sign option in commit-tree, which was documented, but not implemented, in 55ca3f99ae. Add tests for the --gpg-sign option. Signed-off-by: Brandon Richardson <brandon1024.br@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-22t7510: invoke git as part of &&-chainMartin Ågren1-2/+6
If `git commit-tree HEAD^{tree}` fails on us and produces no output on stdout, we will substitute that empty string and execute `git tag ninth-unsigned`, i.e., we will tag HEAD rather than a newly created object. But we are lucky: we have a signature on HEAD, so we should eventually fail the next test, where we verify that "ninth-unsigned" is indeed unsigned. We have a similar problem a few lines later. If `git commit-tree -S` fails with no output, we will happily tag HEAD as "tenth-signed". Here, we are not so lucky. The tag ends up on the same commit as "eighth-signed-alt", and that's a signed commit, so t7510-signed-commit will pass, despite `git commit-tree -S` failing. Make these `git commit-tree` invocations a direct part of the &&-chain, so that we can rely less on luck and set a better example for future tests modeled after this one. Fix a 9/10 copy/paste error while at it. Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Brandon Richardson <brandon1024.br@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-18Merge branch 'ot/ref-filter-object-info'Junio C Hamano1-0/+6
The "--format=<placeholder>" option of for-each-ref, branch and tag learned to show a few more traits of objects that can be learned by the object_info API. * ot/ref-filter-object-info: ref-filter: give uintmax_t to format with %PRIuMAX ref-filter: add docs for new options ref-filter: add tests for deltabase ref-filter: add deltabase option ref-filter: add tests for objectsize:disk ref-filter: add check for negative file size ref-filter: add objectsize:disk option
2019-01-18Merge branch 'sg/stress-test'Junio C Hamano9-133/+308
Flaky tests can now be repeatedly run under load with the "--stress" option. * sg/stress-test: test-lib: add the '--stress' option to run a test repeatedly under load test-lib-functions: introduce the 'test_set_port' helper function test-lib: set $TRASH_DIRECTORY earlier test-lib: consolidate naming of test-results paths test-lib: parse command line options earlier test-lib: parse options in a for loop to keep $@ intact test-lib: extract Bash version check for '-x' tracing test-lib: translate SIGTERM and SIGHUP to an exit
2019-01-18Merge branch 'tg/t5570-drop-racy-test'Junio C Hamano2-24/+3
An inherently racy test that caused intermittent failures has been removed. * tg/t5570-drop-racy-test: Revert "t/lib-git-daemon: record daemon log" t5570: drop racy test
2019-01-18Merge branch 'so/cherry-pick-always-allow-m1'Junio C Hamano3-11/+15
"git cherry-pick -m1" was forbidden when picking a non-merge commit, even though there _is_ parent number 1 for such a commit. This was done to avoid mistakes back when "cherry-pick" was about picking a single commit, but is no longer useful with "cherry-pick" that can pick a range of commits. Now the "-m$num" option is allowed when picking any commit, as long as $num names an existing parent of the commit. Technically this is a backward incompatible change; hopefully nobody is relying on the error-checking behaviour. * so/cherry-pick-always-allow-m1: t3506: validate '-m 1 -ff' is now accepted for non-merge commits t3502: validate '-m 1' argument is now accepted for non-merge commits cherry-pick: do not error on non-merge commits when '-m 1' is specified t3510: stop using '-m 1' to force failure mid-sequence of cherry-picks
2019-01-18Merge branch 'nd/worktree-remove-with-uninitialized-submodules'Junio C Hamano1-0/+37
"git worktree remove" and "git worktree move" refused to work when there is a submodule involved. This has been loosened to ignore uninitialized submodules. * nd/worktree-remove-with-uninitialized-submodules: worktree: allow to (re)move worktrees with uninitialized submodules
2019-01-18Merge branch 'sg/test-bash-version-fix'Junio C Hamano1-2/+2
The test suite tried to see if it is run under bash, but the check itself failed under some other implementations of shell (notably under NetBSD). This has been corrected. * sg/test-bash-version-fix: test-lib: check Bash version for '-x' without using shell arrays
2019-01-18Merge branch 'cy/zsh-completion-SP-in-path'Junio C Hamano1-5/+5
With zsh, "git cmd path<TAB>" was completed to "git cmd path name" when the completed path has a special character like SP in it, without any attempt to keep "path name" a single filename. This has been fixed to complete it to "git cmd path\ name" just like Bash completion does. * cy/zsh-completion-SP-in-path: completion: treat results of git ls-tree as file paths zsh: complete unquoted paths with spaces correctly
2019-01-18Merge branch 'sb/submodule-unset-core-worktree-when-worktree-is-lost'Junio C Hamano3-3/+14
The core.worktree setting in a submodule repository should not be pointing at a directory when the submodule loses its working tree (e.g. getting deinit'ed), but the code did not properly maintain this invariant. * sb/submodule-unset-core-worktree-when-worktree-is-lost: submodule deinit: unset core.worktree submodule--helper: fix BUG message in ensure_core_worktree submodule: unset core.worktree if no working tree is present submodule update: add regression test with old style setups
2019-01-18Merge branch 'jn/stripspace-wo-repository'Junio C Hamano1-3/+9
"git stripspace" should be usable outside a git repository, but under the "-s" or "-c" mode, it didn't. * jn/stripspace-wo-repository: stripspace: allow -s/-c outside git repository
2019-01-18Merge branch 'sb/submodule-fetchjobs-default-to-one'Junio C Hamano1-0/+2
"git submodule update" ought to use a single job unless asked, but by mistake used multiple jobs, which has been fixed. * sb/submodule-fetchjobs-default-to-one: submodule update: run at most one fetch job unless otherwise set
2019-01-18Merge branch 'nd/style-opening-brace'Junio C Hamano1-1/+2
Code clean-up. * nd/style-opening-brace: style: the opening '{' of a function is in a separate line
2019-01-18mingw: special-case arguments to `sh`Johannes Schindelin2-1/+11
The MSYS2 runtime does its best to emulate the command-line wildcard expansion and de-quoting which would be performed by the calling Unix shell on Unix systems. Those Unix shell quoting rules differ from the quoting rules applying to Windows' cmd and Powershell, making it a little awkward to quote command-line parameters properly when spawning other processes. In particular, git.exe passes arguments to subprocesses that are *not* intended to be interpreted as wildcards, and if they contain backslashes, those are not to be interpreted as escape characters, e.g. when passing Windows paths. Note: this is only a problem when calling MSYS2 executables, not when calling MINGW executables such as git.exe. However, we do call MSYS2 executables frequently, most notably when setting the use_shell flag in the child_process structure. There is no elegant way to determine whether the .exe file to be executed is an MSYS2 program or a MINGW one. But since the use case of passing a command line through the shell is so prevalent, we need to work around this issue at least when executing sh.exe. Let's introduce an ugly, hard-coded test whether argv[0] is "sh", and whether it refers to the MSYS2 Bash, to determine whether we need to quote the arguments differently than usual. That still does not fix the issue completely, but at least it is something. Incidentally, this also fixes the problem where `git clone \\server\repo` failed due to incorrect handling of the backslashes when handing the path to the git-upload-pack process. Further, we need to take care to quote not only whitespace and backslashes, but also curly brackets. As aliases frequently go through the MSYS2 Bash, and as aliases frequently get parameters such as HEAD@{yesterday}, this is really important. As an early version of this patch broke this, let's make sure that this does not regress by adding a test case for that. Helped-by: Kim Gybels <kgybels@infogroep.be> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-18mingw (t5580): document bug when cloning from backslashed UNC pathsJohannes Schindelin1-0/+5
Due to a quirk in Git's method to spawn git-upload-pack, there is a problem when passing paths with backslashes in them: Git will force the command-line through the shell, which has different quoting semantics in Git for Windows (being an MSYS2 program) than regular Win32 executables such as git.exe itself. The symptom is that the first of the two backslashes in UNC paths of the form \\myserver\folder\repository.git is *stripped off*. Document this bug by introducing a test case. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-18ls-refs: filter refs using namespace-stripped nameJonathan Tan1-0/+21
If a user fetches refs/heads/master from a repo with namespace "ns", the remote is expected to (1) not send the real refs/heads/master, and (2) send refs/namespaces/ns/refs/heads/master with the name refs/heads/master. (1) indeed happens now, but not (2) - Git only sends refs that have the user-given prefix, but it checks them against the full name of the ref (the one starting with refs/namespaces), and not the namespace-stripped one. This is demonstrated by the patch in the test. Currently, it results in "fatal: couldn't find remote ref refs/heads/master" despite both unnamespaced and namespaced master being present. With the code change, it produces the expected result. Check the ref prefixes against the namespace-stripped name. This bug was discovered through applying patches [1] that override protocol.version to 2 in repositories when running tests, allowing us to notice differences in behavior across different protocol versions. [1] https://public-inbox.org/git/cover.1547677183.git.jonathantanmy@google.com/ Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-18abspath_part_inside_repo: respect core.ignoreCaseJohannes Schindelin1-0/+7
If the file system is case-insensitive, we really must be careful to ignore differences in case only. This fixes https://github.com/git-for-windows/git/issues/735 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-18git-p4: handle update of moved/copied files when updating a shelveLuke Diamand1-1/+1
Perforce requires a complete list of files being operated on. If git is updating an existing shelved changelist, then any files which are moved or copied were not being added to this list. Signed-off-by: Luke Diamand <luke@diamand.org> Acked-by: Andrey Mazo <amazo@checkvideo.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-18git-p4: add failing test for shelved CL update involving move/copyLuke Diamand1-3/+54
Updating a shelved P4 changelist where one or more files have been moved or copied does not work. Add a test for this. The problem is that P4 requires a complete list of the files being changed, and move/copy only includes the _source_ in the case of updating a shelved changelist. This results in errors from Perforce such as: //depot/src - needs tofile //depot/dst Submit aborted -- fix problems then use 'p4 submit -c 1234' Signed-off-by: Luke Diamand <luke@diamand.org> Acked-by: Andrey Mazo <amazo@checkvideo.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17pack-objects: create GIT_TEST_PACK_SPARSEDerrick Stolee2-2/+6
Create a test variable GIT_TEST_PACK_SPARSE to enable the sparse object walk algorithm by default during the test suite. Enabling this variable ensures coverage in many interesting cases, such as shallow clones, partial clones, and missing objects. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17pack-objects: create pack.useSparse settingDerrick Stolee1-0/+15
The '--sparse' flag in 'git pack-objects' changes the algorithm used to enumerate objects to one that is faster for individual users pushing new objects that change only a small cone of the working directory. The sparse algorithm is not recommended for a server, which likely sends new objects that appear across the entire working directory. Create a 'pack.useSparse' setting that enables this new algorithm. This allows 'git push' to use this algorithm without passing a '--sparse' flag all the way through four levels of run_command() calls. If the '--no-sparse' flag is set, then this config setting is overridden. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17revision: implement sparse algorithmDerrick Stolee1-3/+11
When enumerating objects to place in a pack-file during 'git pack-objects --revs', we discover the "frontier" of commits that we care about and the boundary with commit we find uninteresting. From that point, we walk trees to discover which trees and blobs are uninteresting. Finally, we walk trees from the interesting commits to find the interesting objects that are placed in the pack. This commit introduces a new, "sparse" way to discover the uninteresting trees. We use the perspective of a single user trying to push their topic to a large repository. That user likely changed a very small fraction of the paths in their working directory, but we spend a lot of time walking all reachable trees. The way to switch the logic to work in this sparse way is to start caring about which paths introduce new trees. While it is not possible to generate a diff between the frontier boundary and all of the interesting commits, we can simulate that behavior by inspecting all of the root trees as a whole, then recursing down to the set of trees at each path. We already had taken the first step by passing an oidset to mark_trees_uninteresting_sparse(). We now create a dictionary whose keys are paths and values are oidsets. We consider the set of trees that appear at each path. While we inspect a tree, we add its subtrees to the oidsets corresponding to the tree entry's path. We also mark trees as UNINTERESTING if the tree we are parsing is UNINTERESTING. To actually improve the performance, we need to terminate our recursion. If the oidset contains only UNINTERESTING trees, then we do not continue the recursion. This avoids walking trees that are likely to not be reachable from interesting trees. If the oidset contains only interesting trees, then we will walk these trees in the final stage that collects the intersting objects to place in the pack. Thus, we only recurse if the oidset contains both interesting and UNINITERESTING trees. There are a few ways that this is not a universally better option. First, we can pack extra objects. If someone copies a subtree from one tree to another, the first tree will appear UNINTERESTING and we will not recurse to see that the subtree should also be UNINTERESTING. We will walk the new tree and see the subtree as a "new" object and add it to the pack. A test is modified to demonstrate this behavior and to verify that the new logic is being exercised. Second, we can have extra memory pressure. If instead of being a single user pushing a small topic we are a server sending new objects from across the entire working directory, then we will gain very little (the recursion will rarely terminate early) but will spend extra time maintaining the path-oidset dictionaries. Despite these potential drawbacks, the benefits of the algorithm are clear. By adding a counter to 'add_children_by_path' and 'mark_tree_contents_uninteresting', I measured the number of parsed trees for the two algorithms in a variety of repos. For git.git, I used the following input: v2.19.0 ^v2.19.0~10 Objects to pack: 550 Walked (old alg): 282 Walked (new alg): 130 For the Linux repo, I used the following input: v4.18 ^v4.18~10 Objects to pack: 518 Walked (old alg): 4,836 Walked (new alg): 188 The two repos above are rather "wide and flat" compared to other repos that I have used in the past. As a comparison, I tested an old topic branch in the Azure DevOps repo, which has a much deeper folder structure than the Linux repo. Objects to pack: 220 Walked (old alg): 22,804 Walked (new alg): 129 I used the number of walked trees the main metric above because it is consistent across multiple runs. When I ran my tests, the performance of the pack-objects command with the same options could change the end-to-end time by 10x depending on the file system being warm. However, by repeating the same test on repeat I could get more consistent timing results. The git.git and Linux tests were too fast overall (less than 0.5s) to measure an end-to-end difference. The Azure DevOps case was slow enough to see the time improve from 15s to 1s in the warm case. The cold case was 90s to 9s in my testing. These improvements will have even larger benefits in the super- large Windows repository. In our experiments, we see the "Enumerate objects" phase of pack-objects taking 60-80% of the end-to-end time of non-trivial pushes, taking longer than the network time to send the pack and the server time to verify the pack. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17list-objects: consume sparse tree walkDerrick Stolee1-0/+113
When creating a pack-file using 'git pack-objects --revs' we provide a list of interesting and uninteresting commits. For example, a push operation would make the local topic branch be interesting and the known remote refs as uninteresting. We want to discover the set of new objects to send to the server as a thin pack. We walk these commits until we discover a frontier of commits such that every commit walk starting at interesting commits ends in a root commit or unintersting commit. We then need to discover which non-commit objects are reachable from uninteresting commits. This commit walk is not changing during this series. The mark_edges_uninteresting() method in list-objects.c iterates on the commit list and does the following: * If the commit is UNINTERSTING, then mark its root tree and every object it can reach as UNINTERESTING. * If the commit is interesting, then mark the root tree of every UNINTERSTING parent (and all objects that tree can reach) as UNINTERSTING. At the very end, we repeat the process on every commit directly given to the revision walk from stdin. This helps ensure we properly cover shallow commits that otherwise were not included in the frontier. The logic to recursively follow trees is in the mark_tree_uninteresting() method in revision.c. The algorithm avoids duplicate work by not recursing into trees that are already marked UNINTERSTING. Add a new 'sparse' option to the mark_edges_uninteresting() method that performs this logic in a slightly different way. As we iterate over the commits, we add all of the root trees to an oidset. Then, call mark_trees_uninteresting_sparse() on that oidset. Note that we include interesting trees in this process. The current implementation of mark_trees_unintersting_sparse() will walk the same trees as the old logic, but this will be replaced in a later change. Add a '--sparse' flag in 'git pack-objects' to call this new logic. Add a new test script t/t5322-pack-objects-sparse.sh that tests this option. The tests currently demonstrate that the resulting object list is the same as the old algorithm. This includes a case where both algorithms pack an object that is not needed by a remote due to limits on the explored set of trees. When the sparse algorithm is changed in a later commit, we will add a test that demonstrates a change of behavior in some cases. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17add: use separate ADD_CACHE_RENORMALIZE flagJeff King1-0/+9
Commit 9472935d81 (add: introduce "--renormalize", 2017-11-16) taught git-add to pass HASH_RENORMALIZE to add_to_index(), which then passes the flag along to index_path(). However, the flags taken by add_to_index() and the ones taken by index_path() are distinct namespaces. We cannot take HASH_* flags in add_to_index(), because they overlap with the ADD_CACHE_* flags we already take (in this case, HASH_RENORMALIZE conflicts with ADD_CACHE_IGNORE_ERRORS). We can solve this by adding a new ADD_CACHE_RENORMALIZE flag, and using it to set HASH_RENORMALIZE within add_to_index(). In order to make it clear that these two flags come from distinct sets, let's also change the name "newflags" in the function to "hash_flags". Reported-by: Dmitriy Smirnov <dmitriy.smirnov@jetbrains.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17t6042: work around speed optimization on WindowsJohannes Schindelin1-2/+2
When Git determines whether a file has changed, it looks at the mtime, at the file size, and to detect changes even if the mtime is the same (on Windows, the mtime granularity is 100ns, read: if two files are written within the same 100ns time slot, they have the same mtime) and even if the file size is the same, Git also looks at the inode/device numbers. This design obviously comes from a Linux background, where `lstat()` calls were designed to be cheap. On Windows, there is no `lstat()`. It has to be emulated. And while obtaining the mtime and the file size is not all that expensive (you can get both with a single `GetFileAttributesW()` call), obtaining the equivalent of the inode and device numbers is very expensive (it requires a call to `GetFileInformationByHandle()`, which in turn requires a file handle, which is *a lot* more expensive than one might imagine). As it is very uncommon for developers to modify files within 100ns time slots, Git for Windows chooses not to fill inode/device numbers properly, but simply sets them to 0. However, in t6042 the files file_v1 and file_v2 are typically written within the same 100ns time slot, and they do not differ in file size. So the minor modification is not picked up. Let's work around this issue by avoiding the `git mv` calls in the 'mod6-setup: chains of rename/rename(1to2) and rename/rename(2to1)' test case. The target files are overwritten anyway, so it is not like we really rename those files. This fixes the issue because `git add` will now add the files as new files (as opposed to existing, just renamed files). Functionally, we do not change anything because we replace two `git mv <old> <new>` calls (where `<new>` is completely overwritten and `git add`ed later anyway) by `git rm <old>` calls (removing other files, too, that are also completely overwritten and `git add`ed later). Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17tests: define GIT_TEST_SIDEBAND_ALLJonathan Tan5-4/+11
Define a GIT_TEST_SIDEBAND_ALL environment variable meant to be used from tests. When set to true, this overrides uploadpack.allowsidebandall to true, allowing the entire test suite to be run as if this configuration is in place for all repositories. As of this patch, all tests pass whether GIT_TEST_SIDEBAND_ALL is unset or set to 1. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-17helper/test-ref-store: fix "new-sha1" vs "old-sha1" typoChristian Couder1-1/+1
It looks like it is a copy-paste error made in 80f2a6097c (t/helper: add test-ref-store to test ref-store functions, 2017-03-26) to pass "old-sha1" instead of "new-sha1" to notnull() when we get the new sha1 argument from const char **argv. Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Acked-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-15commit-graph: fix buffer read-overflowJosh Steadmon1-3/+13
fuzz-commit-graph identified a case where Git will read past the end of a buffer containing a commit graph if the graph's header has an incorrect chunk count. A simple bounds check in parse_commit_graph() prevents this. Signed-off-by: Josh Steadmon <steadmon@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-15filter-options: expand scaled numbersJosh Steadmon1-0/+17
When communicating with a remote server or a subprocess, use expanded numbers rather than numbers with scaling suffix in the object filter spec (e.g. "limit:blob=1k" becomes "limit:blob=1024"). Update the protocol docs to note that clients should always perform this expansion, to allow for more compatibility between server implementations. Signed-off-by: Josh Steadmon <steadmon@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-15tree:<depth>: skip some trees even when collecting omitsMatthew DeVore1-1/+10
If a tree has already been recorded as omitted, we don't need to traverse it again just to collect its omits. Stop traversing trees a second time when collecting omits. Signed-off-by: Matthew DeVore <matvore@google.com> Reviewed-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-15list-objects-filter: teach tree:# how to handle >0Matthew DeVore1-0/+111
Implement positive values for <depth> in the tree:<depth> filter. The exact semantics are described in Documentation/rev-list-options.txt. The long-term goal at the end of this is to allow a partial clone to eagerly fetch an entire directory of files by fetching a tree and specifying <depth>=1. This, for instance, would make a build operation fast and convenient. It is fast because the partial clone does not need to fetch each file individually, and convenient because the user does not need to supply a sparse-checkout specification. Another way of considering this feature is as a way to reduce round-trips, since the client can get any number of levels of directories in a single request, rather than wait for each level of tree objects to come back, whose entries are used to construct a new request. Signed-off-by: Matthew DeVore <matvore@google.com> Reviewed-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-14Merge branch 'tb/use-common-win32-pathfuncs-on-cygwin'Junio C Hamano1-1/+1
Cygwin update. * tb/use-common-win32-pathfuncs-on-cygwin: git clone <url> C:\cygwin\home\USER\repo' is working (again)
2019-01-14Merge branch 'tb/log-G-binary'Junio C Hamano1-0/+35
"git log -G<regex>" looked for a hunk in the "git log -p" patch output that contained a string that matches the given pattern. Optimize this code to ignore binary files, which by default will not show any hunk that would match any pattern (unless textconv or the --text option is in effect, that is). * tb/log-G-binary: log -G: ignore binary files
2019-01-14Merge branch 'sb/diff-color-moved-config-option-fixup'Junio C Hamano1-0/+18
Minor inconsistency fix. * sb/diff-color-moved-config-option-fixup: diff: align move detection error handling with other options
2019-01-14Merge branch 'hn/highlight-sideband-keywords'Junio C Hamano1-0/+2
Lines that begin with a certain keyword that come over the wire, as well as lines that consist only of one of these keywords, ought to be painted in color for easier eyeballing, but the latter was broken ever since the feature was introduced in 2.19, which has been corrected. * hn/highlight-sideband-keywords: sideband: color lines with keyword only
2019-01-14Merge branch 'cb/test-lint-cp-a'Junio C Hamano1-0/+1
BSD port update. * cb/test-lint-cp-a: tests: add lint for non portable cp -a
2019-01-14Merge branch 'cb/t5004-empty-tar-archive-fix'Junio C Hamano1-9/+8
BSD port update. * cb/t5004-empty-tar-archive-fix: t5004: avoid using tar for empty packages
2019-01-14Merge branch 'hb/t0061-dot-in-path-fix'Junio C Hamano1-1/+9
Test update. * hb/t0061-dot-in-path-fix: t0061: do not fail test if '.' is part of $PATH
2019-01-14Merge branch 'nd/checkout-noisy'Junio C Hamano1-2/+2
"git checkout [<tree-ish>] path..." learned to report the number of paths that have been checked out of the index or the tree-ish, which gives it the same degree of noisy-ness as the case in which the command checks out a branch. * nd/checkout-noisy: t0027: squelch checkout path run outside test_expect_* block checkout: print something when checking out paths
2019-01-14Merge branch 'nd/attr-pathspec-in-tree-walk'Junio C Hamano1-1/+57
The traversal over tree objects has learned to honor ":(attr:label)" pathspec match, which has been implemented only for enumerating paths on the filesystem. * nd/attr-pathspec-in-tree-walk: tree-walk: support :(attr) matching dir.c: move, rename and export match_attrs() pathspec.h: clean up "extern" in function declarations tree-walk.c: make tree_entry_interesting() take an index tree.c: make read_tree*() take 'struct repository *'
2019-01-14Merge branch 'md/list-lazy-objects-fix'Junio C Hamano1-2/+14
"git rev-list --exclude-promisor-objects" had to take an object that does not exist locally (and is lazily available) from the command line without barfing, but the code dereferenced NULL. * md/list-lazy-objects-fix: list-objects.c: don't segfault for missing cmdline objects
2019-01-14Merge branch 'ms/packet-err-check' into jt/fetch-v2-sidebandJunio C Hamano1-2/+2
* ms/packet-err-check: pack-protocol.txt: accept error packets in any context Use packet_reader instead of packet_read_line
2019-01-11log: add %S option (like --source) to log --formatIssac Trotts2-0/+54
Make it possible to write for example git log --format="%H,%S" where the %S at the end is a new placeholder that prints out the ref (tag/branch) for each commit. Using %d might seem like an alternative but it only shows the ref for the last commit in the branch. Signed-off-by: Issac Trotts <issactrotts@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-10test: test GIT_CURL_VERBOSE=1 shows an errorMasaya Suzuki2-0/+29
This tests GIT_CURL_VERBOSE shows an error when an URL returns 500. This exercises the code in remote_curl. Signed-off-by: Masaya Suzuki <masayasuzuki@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-10upload-pack: teach deepen-relative in protocol v2Jonathan Tan1-0/+29
Commit 685fbd3291 ("fetch-pack: perform a fetch using v2", 2018-03-15) attempted to teach Git deepen-relative in protocol v2 (among other things), but it didn't work: (1) fetch-pack.c needs to emit "deepen-relative". (2) upload-pack.c needs to ensure that the correct deepen_relative variable is passed to deepen() (there are two - the static variable and the one in struct upload_pack_data). (3) Before deepen() computes the list of reachable shallows, it first needs to mark all "our" refs as OUR_REF. v2 currently does not do this, because unlike v0, it is not needed otherwise. Fix all this and include a test demonstrating that it works now. For (2), the static variable deepen_relative is also eliminated, removing a source of confusion. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Reviewed-by: Josh Steadmon <steadmon@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-10fetch-pack: do not take shallow lock unnecessarilyJonathan Tan1-0/+18
When fetching using protocol v2, the remote may send a "shallow-info" section if the client is shallow. If so, Git as the client currently takes the shallow file lock, even if the "shallow-info" section is empty. This is not a problem except that Git does not support taking the shallow file lock after modifying the shallow file, because is_repository_shallow() stores information that is never cleared. And this take-after-modify occurs when Git does a tag-following fetch from a shallow repository on a transport that does not support tag following (since in this case, 2 fetches are performed). To solve this issue, take the shallow file lock (and perform all other shallow processing) only if the "shallow-info" section is non-empty; otherwise, behave as if it were empty. A full solution (probably, ensuring that any action of committing shallow file locks also includes clearing the information stored by is_repository_shallow()) would solve the issue without need for this patch, but this patch is independently useful (as an optimization to prevent writing a file in an unnecessary case), hence why I wrote it. I have included a NEEDSWORK outlining the full solution. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-10fetch-pack: support protocol version 2Jonathan Tan1-7/+15
When the scaffolding for protocol version 2 was initially added in 8f6982b4e1 ("protocol: introduce enum protocol_version value protocol_v2", 2018-03-14). As seen in: git log -p -G'support for protocol v2 not implemented yet' --full-diff --reverse v2.17.0..v2.20.0 Many of those scaffolding "die" placeholders were removed, but we hadn't gotten around to fetch-pack yet. The test here for "fetch refs from cmdline" is very minimal. There's much better coverage when running the entire test suite under the WIP GIT_TEST_PROTOCOL_VERSION=2 mode[1], we should ideally have better coverage without needing to invoke a special test mode. 1. https://public-inbox.org/git/20181213155817.27666-1-avarab@gmail.com/ Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-10upload-pack: support hidden refs with protocol v2Jeff King1-0/+6
In the v2 protocol, upload-pack's advertisement has been moved to the "ls-refs" command. That command does not respect hidden-ref config (like transfer.hiderefs) at all, and advertises everything. While there are some features that are not supported in v2 (e.g., v2 always allows fetching any sha1 without respect to advertisements), the lack of this feature is not documented and is likely just a bug. Let's make it work, as otherwise upgrading a server to a v2-capable git will start exposing these refs that the repository admin has asked to remain hidden. Note that we assume we're operating on behalf of a fetch here, since that's the only thing implemented in v2 at this point. See the in-code comment. We'll have to figure out how this works when the v2 push protocol is designed (both here in ls-refs, but also rejecting updates to hidden refs). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-10diff --color-moved-ws: handle blank linesPhillip Wood1-4/+37
When using --color-moved-ws=allow-indentation-change allow lines with the same indentation change to be grouped across blank lines. For now this only works if the blank lines have been moved as well, not for blocks that have just had their indentation changed. This completes the changes to the implementation of --color-moved=allow-indentation-change. Running git diff --color-moved=allow-indentation-change v2.18.0 v2.19.0 now takes 5.0s. This is a saving of 41% from 8.5s for the optimized version of the previous implementation and 66% from the original which took 14.6s. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Reviewed-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-10diff --color-moved-ws: modify allow-indentation-changePhillip Wood1-0/+56
Currently diff --color-moved-ws=allow-indentation-change does not support indentation that contains a mix of tabs and spaces. For example in commit 546f70f377 ("convert.h: drop 'extern' from function declaration", 2018-06-30) the function parameters in the following lines are not colored as moved [1]. -extern int stream_filter(struct stream_filter *, - const char *input, size_t *isize_p, - char *output, size_t *osize_p); +int stream_filter(struct stream_filter *, + const char *input, size_t *isize_p, + char *output, size_t *osize_p); This commit changes the way the indentation is handled to track the visual size of the indentation rather than the characters in the indentation. This has the benefit that any whitespace errors do not interfer with the move detection (the whitespace errors will still be highlighted according to --ws-error-highlight). During the discussion of this feature there were concerns about the correct detection of indentation for python. However those concerns apply whether or not we're detecting moved lines so no attempt is made to determine if the indentation is 'pythonic'. [1] Note that before the commit to fix the erroneous coloring of moved lines each line was colored as a different block, since that commit they are uncolored. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Reviewed-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-10diff --color-moved=zebra: be stricter with color alternationPhillip Wood1-3/+3
Currently when using --color-moved=zebra the color of moved blocks depends on the number of lines separating them. This means that adding an odd number of unmoved lines between blocks that are already separated by one or more unmoved lines will change the color of subsequent moved blocks. This does not make much sense as the blocks were already separated by unmoved lines and causes problems when adding lines to test cases. Fix this by only using the alternate colors for adjacent moved blocks. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Reviewed-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-10diff --color-moved-ws: demonstrate false positivesPhillip Wood1-2/+6
'diff --color-moved-ws=allow-indentation-change' can highlight lines that have internal whitespace changes rather than indentation changes. For example in commit 1a07e59c3e ("Update messages in preparation for i18n", 2018-07-21) the lines - die (_("must end with a color")); + die(_("must end with a color")); are highlighted as moved when they should not be. Modify an existing test to show the problem that will be fixed in the next commit. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Reviewed-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-08prefer "hash mismatch" to "sha1 mismatch"Jeff King1-1/+1
To future-proof ourselves against a change in the hash, let's use the more generic "hash mismatch" to refer to integrity problems. Note that we do advertise this exact string in git-fsck(1). However, the message itself is marked for translation, meaning we do not expect it to be machine-readable. While we're touching that documentation, let's also update it for grammar and clarity. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07rebase: implement --merge via the interactive machineryElijah Newren4-84/+13
As part of an ongoing effort to make rebase have more uniform behavior, modify the merge backend to behave like the interactive one, by re-implementing it on top of the latter. Interactive rebases are implemented in terms of cherry-pick rather than the merge-recursive builtin, but cherry-pick also calls into the recursive merge machinery by default and can accept special merge strategies and/or special strategy options. As such, there really is not any need for having both git-rebase--merge and git-rebase--interactive anymore. Delete git-rebase--merge.sh and instead implement it in builtin/rebase.c. This results in a few deliberate but small user-visible changes: * The progress output is modified (see t3406 and t3420 for examples) * A few known test failures are now fixed (see t3421) * bash-prompt during a rebase --merge is now REBASE-i instead of REBASE-m. Reason: The prompt is a reflection of the backend in use; this allows users to report an issue to the git mailing list with the appropriate backend information, and allows advanced users to know where to search for relevant control files. (see t9903) testcase modification notes: t3406: --interactive and --merge had slightly different progress output while running; adjust a test to match the new expectation t3420: these test precise output while running, but rebase--am, rebase--merge, and rebase--interactive all were built on very different commands (am, merge-recursive, cherry-pick), so the tests expected different output for each type. Now we expect --merge and --interactive to have the same output. t3421: --interactive fixes some bugs in --merge! Wahoo! t9903: --merge uses the interactive backend so the prompt expected is now REBASE-i. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07worktree: allow to (re)move worktrees with uninitialized submodulesNguyễn Thái Ngọc Duy1-0/+37
Uninitialized submodules have nothing valueable for us to be worried about. They are just SHA-1. Let "worktree remove" and "worktree move" continue in this case so that people can still use multiple worktrees on repos with optional submodules that are never populated, like sha1collisiondetection in git.git when checked out by doc-diff script. Note that for "worktree remove", it is possible that a user initializes a submodule (*), makes some commits (but not push), then deinitializes it. At that point, the submodule is unpopulated, but the precious new commits are still in $GIT_COMMON_DIR/worktrees/<worktree>/modules/<submodule> directory and we should not allow removing the worktree or we lose those commits forever. The new directory check is added to prevent this. (*) yes they are screwed anyway by doing this since "git submodule" would add submodule.* in $GIT_COMMON_DIR/config, which is shared across multiple worktrees. But it does not mean we let them be screwed even more. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: add the '--stress' option to run a test repeatedly under loadSZEDER Gábor3-5/+130
Unfortunately, we have a few flaky tests, whose failures tend to be hard to reproduce. We've found that the best we can do to reproduce such a failure is to run the test script repeatedly while the machine is under load, and wait in the hope that the load creates enough variance in the timing of the test's commands that a failure is evenually triggered. I have a command to do that, and I noticed that two other contributors have rolled their own scripts to do the same, all choosing slightly different approaches. To help reproduce failures in flaky tests, introduce the '--stress' option to run a test script repeatedly in multiple parallel jobs until one of them fails, thereby using the test script itself to increase the load on the machine. The number of parallel jobs is determined by, in order of precedence: the number specified as '--stress=<N>', or the value of the GIT_TEST_STRESS_LOAD environment variable, or twice the number of available processors (as reported by the 'getconf' utility), or 8. Make '--stress' imply '--verbose -x --immediate' to get the most information about rare failures; there is really no point in spending all the extra effort to reproduce such a failure, and then not know which command failed and why. To prevent the several parallel invocations of the same test from interfering with each other: - Include the parallel job's number in the name of the trash directory and the various output files under 't/test-results/' as a '.stress-<Nr>' suffix. - Add the parallel job's number to the port number specified by the user or to the test number, so even tests involving daemons listening on a TCP socket can be stressed. - Redirect each parallel test run's verbose output to 't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the output of several parallel running tests to the terminal would create a big ugly mess. For convenience, print the output of the failed test job at the end, and rename its trash directory to end with the '.stress-failed' suffix, so it's easy to find in a predictable path (OTOH, all absolute paths recorded in the trash directory become invalid; we'll see whether this causes any issues in practice). If, in an unlikely case, more than one jobs were to fail nearly at the same time, then print the output of all failed jobs, and rename the trash directory of only the last one (i.e. with the highest job number), as it is the trash directory of the test whose output will be at the bottom of the user's terminal. Based on Jeff King's 'stress' script. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib-functions: introduce the 'test_set_port' helper functionSZEDER Gábor7-13/+41
Several test scripts run daemons like 'git-daemon' or Apache, and communicate with them through TCP sockets. To have unique ports where these daemons are accessible, the ports are usually the number of the corresponding test scripts, unless the user overrides them via environment variables, and thus all those tests and test libs contain more or less the same bit of one-liner boilerplate code to find out the port. The last patch in this series will make this a bit more complicated. Factor out finding the port for a daemon into the common helper function 'test_set_port' to avoid repeating ourselves. Take special care of test scripts with "low" numbers: - Test numbers below 1024 would result in a port that's only usable as root, so set their port to '10000 + test-nr' to make sure it doesn't interfere with other tests in the test suite. This makes the hardcoded port number in 't0410-partial-clone.sh' unnecessary, remove it. - The shell's arithmetic evaluation interprets numbers with leading zeros as octal values, which means that test number below 1000 and containing the digits 8 or 9 will trigger an error. Remove all leading zeros from the test numbers to prevent this. Note that the 'git p4' tests are unlike the other tests involving daemons in that: - 'lib-git-p4.sh' doesn't use the test's number for unique port as is, but does a bit of additional arithmetic on top [1]. - The port is not overridable via an environment variable. With this patch even 'git p4' tests will use the test's number as default port, and it will be overridable via the P4DPORT environment variable. [1] Commit fc00233071 (git-p4 tests: refactor and cleanup, 2011-08-22) introduced that "unusual" unique port computation without explaining why it was necessary (as opposed to simply using the test number as is). It seems to be just unnecessary complication, and in any case that commit came way before the "test nr as unique port" got "standardized" for other daemons in commits c44132fcf3 (tests: auto-set git-daemon port, 2014-02-10), 3bb486e439 (tests: auto-set LIB_HTTPD_PORT from test name, 2014-02-10), and bf9d7df950 (t/lib-git-svn.sh: improve svnserve tests with parallel make test, 2017-12-01). Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: set $TRASH_DIRECTORY earlierSZEDER Gábor1-6/+6
A later patch in this series will need to know the path to the trash directory early in 'test-lib.sh', but $TRASH_DIRECTORY is set much later. Set $TRASH_DIRECTORY earlier, where the other test-specific path variables are set. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: consolidate naming of test-results pathsSZEDER Gábor1-11/+11
There are two places where we strip off any leading path components and the '.sh' suffix from the test script's pathname, and there are four places where we construct the name of the 't/test-results' directory or the name of various test-specific files in there. The last patch in this series will add even more. Factor these out into helper variables to avoid repeating ourselves. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: parse command line options earlierSZEDER Gábor1-109/+124
'test-lib.sh' looks for the presence of certain options like '--tee' and '--verbose-log', so it can execute the test script again to save its standard output and error. It looks for '--valgrind' as well, to set up some Valgrind-specific stuff. These all happen before the actual option parsing loop, and the conditions looking for these options look a bit odd, too. They are not completely correct, either, because in a bogus invocation like './t1234-foo.sh -r --tee' they recognize '--tee', although it should be handled as the required argument of the '-r' option. This patch series will add two more options to look out for early, and, in addition, will have to extract these options' stuck arguments (i.e. '--opt=arg') as well. So let's move the option parsing loop and the couple of related conditions following it earlier in 'test-lib.sh', before the place where the test script is executed again for '--tee' and its friends. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: parse options in a for loop to keep $@ intactSZEDER Gábor1-36/+42
'test-lib.sh' looks for the presence of certain options like '--tee' and '--verbose-log', so it can execute the test script again to save its standard output and error, and to do so it needs the original command line options the test was invoked with. The next patch is about to move the option parsing loop earlier in 'test-lib.sh', but it is implemented using 'shift' in a while loop, effecively destroying "$@" by the end of the option parsing. Not good. As a preparatory step, turn that option parsing loop into a 'for opt in "$@"' loop to preserve "$@" intact while iterating over the options, and taking extra care to handle the '-r' option's required argument (or the lack thereof). Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07test-lib: extract Bash version check for '-x' tracingSZEDER Gábor1-18/+19
One of our test scripts, 't1510-repo-setup.sh' [1], still can't be reliably run with '-x' tracing enabled, unless it's executed with a Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy condition to check the version of the shell running the test script, and disable tracing if it's not executed with a suitable Bash version [2]. Move this check out from the option parsing loop, so other options can imply '-x' by setting 'trace=t', without missing this Bash version check. [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x', 2018-02-24) [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual test scripts, 2018-02-24) Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07t3506: validate '-m 1 -ff' is now accepted for non-merge commitsSergey Organov1-3/+3
Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07t3502: validate '-m 1' argument is now accepted for non-merge commitsSergey Organov1-6/+6
Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-07Revert "t/lib-git-daemon: record daemon log"Thomas Gummerer1-11/+3
This reverts commit 314a73d658 (t/lib-git-daemon: record daemon log, 2018-01-25), which let tests use the output of git-daemon. The previous commit removed the last user of deamon.log in the tests, there's no good way to make checking for output in the log race-proof. Revert this commit as well, to make sure others are not tempted to use daemon.log in tests in the future, which would lead to racy tests. The original commit had one change that still makes sense, namely switching read/echo for "read -r" and "printf", which relays the data more faithfully. Don't revert that piece here, as it is still a useful change. Suggested-by: Jeff King <peff@peff.net> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-04Merge branch 'sd/stash-wo-user-name'Junio C Hamano1-0/+28
A properly configured username/email is required under user.useConfigOnly in order to create commits; now "git stash" (even though it creates commit objects to represent stash entries) command is exempt from the requirement. * sd/stash-wo-user-name: stash: tolerate missing user identity
2019-01-04Merge branch 'sg/clone-initial-fetch-configuration'Junio C Hamano1-0/+47
Refspecs configured with "git -c var=val clone" did not propagate to the resulting repository, which has been corrected. * sg/clone-initial-fetch-configuration: Documentation/clone: document ignored configuration variables clone: respect additional configured fetch refspecs during initial fetch clone: use a more appropriate variable name for the default refspec
2019-01-04Merge branch 'nd/checkout-dwim-fix'Junio C Hamano2-1/+33
"git checkout frotz" (without any double-dash) avoids ambiguity by making sure 'frotz' cannot be interpreted as a revision and as a path at the same time. This safety has been updated to check also a unique remote-tracking branch 'frotz' in a remote, when dwimming to create a local branch 'frotz' out of a remote-tracking branch 'frotz' from a remote. * nd/checkout-dwim-fix: checkout: disambiguate dwim tracking branches and local files
2019-01-04Merge branch 'ab/push-dwim-dst'Junio C Hamano1-0/+55
"git push $there $src:$dst" rejects when $dst is not a fully qualified refname and not clear what the end user meant. The codepath has been taught to give a clearer error message, and also guess where the push should go by taking the type of the pushed object into account (e.g. a tag object would want to go under refs/tags/). * ab/push-dwim-dst: push doc: document the DWYM behavior pushing to unqualified <dst> push: test that <src> doesn't DWYM if <dst> is unqualified push: add an advice on unqualified <dst> push push: move unqualified refname error into a function push: improve the error shown on unqualified <dst> push i18n: remote.c: mark error(...) messages for translation remote.c: add braces in anticipation of a follow-up change
2019-01-04Merge branch 'en/fast-export-import'Junio C Hamano1-3/+77
Small fixes and features for fast-export and fast-import, mostly on the fast-export side. * en/fast-export-import: fast-export: add a --show-original-ids option to show original names fast-import: remove unmaintained duplicate documentation fast-export: add --reference-excluded-parents option fast-export: ensure we export requested refs fast-export: when using paths, avoid corrupt stream with non-existent mark fast-export: move commit rewriting logic into a function for reuse fast-export: avoid dying when filtering by paths and old tags exist fast-export: use value from correct enum git-fast-export.txt: clarify misleading documentation about rev-list args git-fast-import.txt: fix documentation for --quiet option fast-export: convert sha1 to oid
2019-01-04Merge branch 'en/merge-path-collision'Junio C Hamano3-78/+829
Updates for corner cases in merge-recursive. * en/merge-path-collision: t6036: avoid non-portable "cp -a" merge-recursive: combine error handling t6036, t6043: increase code coverage for file collision handling merge-recursive: improve rename/rename(1to2)/add[/add] handling merge-recursive: use handle_file_collision for add/add conflicts merge-recursive: improve handling for rename/rename(2to1) conflicts merge-recursive: fix rename/add conflict handling merge-recursive: new function for better colliding conflict resolutions merge-recursive: increase marker length with depth of recursion t6036, t6042: testcases for rename collision of already conflicting files t6042: add tests for consistency in file collision conflict handling
2019-01-04Merge branch 'nd/i18n'Junio C Hamano8-44/+44
More _("i18n") markings. * nd/i18n: fsck: mark strings for translation fsck: reduce word legos to help i18n parse-options.c: mark more strings for translation parse-options.c: turn some die() to BUG() parse-options: replace opterror() with optname() repack: mark more strings for translation remote.c: mark messages for translation remote.c: turn some error() or die() to BUG() reflog: mark strings for translation read-cache.c: add missing colon separators read-cache.c: mark more strings for translation read-cache.c: turn die("internal error") to BUG() attr.c: mark more string for translation archive.c: mark more strings for translation alias.c: mark split_cmdline_strerror() strings for translation git.c: mark more strings for translation
2019-01-03test-lib: translate SIGTERM and SIGHUP to an exitSZEDER Gábor1-1/+1
Right now if a test script receives SIGTERM or SIGHUP (e.g., because a test was hanging and the user 'kill'-ed it or simply closed the terminal window the test was running in), the shell exits immediately. This can be annoying if the test script did any global setup, like starting apache or git-daemon, as it will not have an opportunity to clean up after itself. A subsequent run of the test won't be able to start its own daemon, and will either fail or skip the tests. Instead, let's trap SIGTERM and SIGHUP as well to make sure we do a clean shutdown, and just chain it to a normal exit (which will trigger any cleanup). This patch follows suit of da706545f7 (t: translate SIGINT to an exit, 2015-03-13), and even stole its commit message as well. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-03Merge branch 'sg/test-bash-version-fix'Junio C Hamano1-2/+2
* sg/test-bash-version-fix: test-lib: check Bash version for '-x' without using shell arrays
2019-01-03test-lib: check Bash version for '-x' without using shell arraysSZEDER Gábor1-2/+2
One of our test scripts, 't1510-repo-setup.sh' [1], still can't be reliably run with '-x' tracing enabled, unless it's executed with a Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy condition to check the version of the shell running the test script, and disable tracing if it's not executed with a suitable Bash version [2]. This condition uses non-portable shell array accesses to easily get Bash's major and minor version number. This didn't seem to be problematic, because the simple commands expanding those array accesses are only executed when the test script is actually run with Bash. When run with Dash, the only shell I have at hand that doesn't support shell arrays, there are no issues, as it apparently skips right over the non-executed simple commands without noticing the non-supported constructs. Alas, it has been reported that NetBSD's /bin/sh does complain about them: ./test-lib.sh: 327: Syntax error: Bad substitution where line 327 contains the first ${BASH_VERSINFO[0]} array access. To my understanding both shells are right and conform to POSIX, because the standard allows both behaviors by stating the following under '2.8.1 Consequences of Shell Errors' [3]: "An expansion error is one that occurs when the shell expansions define in wordexp are carried out (for example, "${x!y}", because '!' is not a valid operator); an implementation may treat these as syntax errors if it is able to detect them during tokenization, rather than during expansion." Avoid this issue with NetBSD's /bin/sh (and potentially with other, less common shells) by hiding the shell array syntax behind 'eval' that is only executed with Bash. [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x', 2018-02-24) [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual test scripts, 2018-02-24) [3] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01 Reported-by: Max Kirillov <max@max630.net> Helped-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-03completion: treat results of git ls-tree as file pathsChayoung You1-5/+5
Let's say there are files named 'foo bar.txt', and 'abc def/test.txt' in repository. When following commands trigger a completion: git show HEAD:fo<Tab> git show HEAD:ab<Tab> The completion results in bash/zsh: git show HEAD:foo bar.txt git show HEAD:abc def/ Where the both of them have an unescaped space in paths, so they'll be misread by git. All entries of git ls-tree either a filename or a directory, so __gitcomp_file() is proper rather than __gitcomp_nl(). Note the commit f12785a3, which handles quoted paths properly. Like this case, we should dequote $cur_ for ?*:* case. For example, let's say there is untracked directory 'abc deg', then trigger a completion: git show HEAD:abc\ de<Tab> git show HEAD:'abc de<Tab> git show HEAD:"abc de<Tab> should uniquely complete 'abc def', but bash completes 'abc def' and 'abc deg' instead. In zsh, triggering a completion: git show HEAD:abc\ def/<Tab> should complete 'test.txt', but nothing comes. The both problems will be resolved by dequoting paths. __git_complete_revlist_file() passes arguments to __gitcomp_nl() where the first one is a list something like: abc def/Z foo bar.txt Z where Z is the mark of the EOL. - The trailing space of blob in __git ls-tree | sed. It makes the completion results become: git show HEAD:foo\ bar.txt\ <CURSOR> So git will try to find a file named 'foo bar.txt ' instead. - The trailing slash of tree in __git ls-tree | sed. It makes the completion results on zsh become: git show HEAD:abc\ def/ <CURSOR> So that the last space on command like should be removed on zsh to complete filenames under 'abc def/'. Signed-off-by: Chayoung You <yousbe@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-02rebase: run post-checkout hook on checkoutOrgad Shaneh1-0/+20
The scripted version of rebase used to run this hook on the initial checkout. The transition to built-in introduced a regression. Signed-off-by: Orgad Shaneh <orgads@gmail.com> Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-02t5403: simplify by using a single repositoryOrgad Shaneh1-55/+23
There is no strong reason to use separate clones to run these tests; just use a single test repository prepared with more modern test_commit shell helper function. While at it, replace three "awk '{print $N}'" on the same file with shell built-in "read" into three variables. Revert d42ec126aa717d00549e387d5a95fd55683c2e2c which is a workaround for Cygwin that is no longer needed. Signed-off-by: Orgad Shaneh <orgads@gmail.com> Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-02pack-protocol.txt: accept error packets in any contextMasaya Suzuki1-2/+2
In the Git pack protocol definition, an error packet may appear only in a certain context. However, servers can face a runtime error (e.g. I/O error) at an arbitrary timing. This patch changes the protocol to allow an error packet to be sent instead of any packet. Without this protocol spec change, when a server cannot process a request, there's no way to tell that to a client. Since the server cannot produce a valid response, it would be forced to cut a connection without telling why. With this protocol spec change, the server can be more gentle in this situation. An old client may see these error packets as an unexpected packet, but this is not worse than having an unexpected EOF. Following this protocol spec change, the error packet handling code is moved to pkt-line.c. Implementation wise, this implementation uses pkt-line to communicate with a subprocess. Since this is not a part of Git protocol, it's possible that a packet that is not supposed to be an error packet is mistakenly parsed as an error packet. This error packet handling is enabled only for the Git pack protocol parsing code considering this. Signed-off-by: Masaya Suzuki <masayasuzuki@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-02bisect--helper: `get_terms` & `bisect_terms` shell function in CPranit Bauva1-1/+1
Reimplement the `get_terms` and `bisect_terms` shell function in C and add `bisect-terms` subcommand to `git bisect--helper` to call it from git-bisect.sh . Using `--bisect-terms` subcommand is a temporary measure to port shell function in C so as to use the existing test suite. As more functions are ported, this subcommand will be retired but its implementation will be called by some other methods. Also use error() to report "no terms defined" and accordingly change the test in t6030. We need to use PARSE_OPT_KEEP_UNKNOWN here to allow for parameters that look like options (e.g --term-good) but should not be parsed by cmd_bisect__helper(). This change is safe because all other cmdmodes have strict argc checks already. Mentored-by: Lars Schneider <larsxschneider@gmail.com> Mentored-by: Christian Couder <chriscool@tuxfamily.org> Mentored-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Pranit Bauva <pranit.bauva@gmail.com> Signed-off-by: Tanushree Tumane <tanushreetumane@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>