@() (array-subexpression operator) should not be optimized away when a cast to an array type is involved

[mklement0]

^{This is a follow-up to the discussion involving @daxian-dbw and @PetSerAl in #4257}

Context:

• PSv5.1 introduced a generally helpful optimization that optimizes @() away in cases where an array is directly constructed inside of it.

  • I presume that the motivation for this optimization was the widespread, but redundant and inefficient practice of enclosing ,-constructed arrays ("array literals") in @(), such as in @(1, 2), where 1, 2 would not only suffice, but, in PSv5-, @() would actually (effectively) clone the array.

• Unfortunately, this optimization is also applied when a cast to an array type (e.g, [int[]]) is involved, which can have unpleasant side effects - see below.
  Not optimizing in this scenario makes for conceptually clearer, predictable behavior.

• While a change would technically be a breaking one, it is hard to imagine anyone having relied on the optimized behavior with casts - a typical Bucket 3: Unlikely Grey Area change.

  • Also note that when the optimization was introduced in PSv5.1, it was technically a breaking change too - though I suspect it wasn't announced as such or even at all.

Steps to reproduce

@([int[]] (1, 2)).GetType().Name

$arr = 1, 2; @([int[]] $arr).GetType().Name

Expected behavior

Object[]
Object[]

Actual behavior

Int32[]
Int32[]

Note that versions up to and including v5 - before the optimization was introduced - behave as expected.

Why is the current behavior problematic?

^{The examples in this section are courtesy of @PetSerAl.}

All examples are undoubtedly edge cases, but the larger point is that this behavior should never have been introduced, and, given its "stealth status", this is an opportunity to get rid of it.

• It subverts the fundamental assumption that @() always creates regular PowerShell arrays (System.Object[]):

> @([int[]] (1, 2))[-1] = 'foo'   # BREAKS, because the array is [int[]], not [object[]]
Cannot convert value "foo" to type "System.Int32".  ...
...

@PetSerAl points out that this fundamental assumption is even still reflected in other places in the source code: ArrayExpressionAst.StaticType (a @(...) expression is of type [System.Management.Automation.Language.ArrayExpressionAst]).

• It subverts the expectation that @() always clones an array:

> $a = [int[]] (1, 2); $b = @([int[]] $a); [object]::ReferenceEquals($a, $b)
True    # $a and $b unexpectedly reference the same array.

Note that the accidental reference equality only happens if the source type and the cast type are identical if the cast type is either the same as the source type or a type that the source type is (directly or indirectly) derived from (covariance).

Thus, if the cast type is [object[]] and the source type is a reference type too, the problem always surfaces.

Environment data

PowerShell Core v6.0.0-beta.4 on macOS 10.12.5
PowerShell Core v6.0.0-beta.4 on Ubuntu 16.04.2 LTS
PowerShell Core v6.0.0-beta.4 on Microsoft Windows 10 Pro (64-bit; v10.0.15063)
Windows PowerShell v5.1.15063.413 on Microsoft Windows 10 Pro (64-bit; v10.0.15063)

[PetSerAl]

Note that the accidental reference equality only happens if the source type and the cast type are identical.

It not necessary to be identical due to array covariance in .NET:

$a = [string[]]@()
$b = @([object[]]$a)
[object]::ReferenceEquals($a, $b)

[mklement0]

@PetSerAl: Thanks for the correction; I've fixed the original post.

[daxian-dbw]

Moving the discussions about this issue from #4257. Starting from this comment

From @mklement0:

Given the potentially problematic behavior of something like @([int[]] (1, 2)) (the only case where @() doesn't output an [object[]] array), is it worth creating a new issue for that?

It's hard to imagine that anyone would have relied on that behavior, and eliminating it makes for a more predictable environment.

[daxian-dbw]

From @daxian-dbw:

Here is the code that handles this special case. In my understanding, this happens only in one of the following two conditions:

• the expression is a ConvertExpressionAst or a ParenExpression that wraps a ConvertExpressionAst, and the convert-to type is an array.
• the expression is an ArrayLiteralAst or a ParenExpression that wraps an ArrayLiteralAst.

In the first case, since the conversion is explicitly specified, it's very unlikely the resulting array will again be used as object[]. In the second case, the resulting array from ArrayLiteralAst is already object[] so it doesn't matter. Plus, altering this behavior would be a breaking change. Given the above, I prefer to not make a behavior change. But feel free to notify the PowerShell-Committee if you want a further discussion.

[daxian-dbw]

From @mklement0:

I can't imagine much existing code breaking, but, conversely, I can't imagine many people running into a problem with the current behavior in real-world scenarios, so, personally, I'm happy to leave it at that; perhaps @PetSerAl feels differently.

[daxian-dbw]

From @PetSerAl:

@daxian-dbw, how about this:

$a = 1..3
$b = @([object[]]$a)
[object]::ReferenceEquals($a, $b)
$b[1] = 123
$a[1]

Now $b is not copy of $a but is $a. Also if current behavior will be kept, is not ArrayExpressionAst.StaticType should be adjusted to match current behavior?

[daxian-dbw]

From @daxian-dbw:

Now $b is not copy of $a but is $a

Yes, $b and $a point to the same array in this case. But I don't think the array expression in powershell language is defined to always return a new array object.

Is not ArrayExpressionAst.StaticType should be adjusted to match current behavior?

Good catch. It is inconsistent and should be fixed. But IMHO, the fix should be to change ArrayExpressionAst.StaticType to always return System.Array (also a breaking change :)). StaticType is by definition not accurate, because sometimes the actual type can be known only at runtime, for example, BinaryExpressionAst.StaticType returns System.Object when it's -bor, and the actual type may vary at runtime.

[daxian-dbw]

From @PetSerAl:

@daxian-dbw But where I can see that definition of array expression? Before now I always expect it to return new array object each time. I myself use $b = @($a) (without cast although) as array copy operator, and I really do not like if it stop copying array at some point in the future.

[daxian-dbw]

From @daxian-dbw:

You can find the Windows PowerShell Langauge Specification Version 3.0 here. It hasn't been updated for a while, so new language features like DSC or PowerShell Class are not in it, but I believe the content in the specification should still apply to PowerShell 6.0.