Add some additional support for Intersection and Union in the Calculation Engine. This allows me to reinstate 2 tests which were formerly skipped, without breaking any existing tests. There are almost certainly edge cases which I haven't thought of yet; I will leave this PR as a draft for several weeks before moving it formward.
This change gives some opportunities for users to go wrong. If you place the following formula in a cell:
```
=B1:B8 A7:D7
```
Excel will treat the space as an intersection operator, and will return the value in B7, which is where the ranges intersect. PhpSpreadsheet will do so as well. But, if you use the following formula:
```
=B1:B8,A7:D7
```
Excel will return `#VALUE!`. This seems like something they forgot to take care of when adding dynamic arrays. The comma should be interpreted as a union operator, and PhpSpreadsheet will now return the union of the ranges. It seems very difficult for PhpSpreadsheet to return an error when the formula seems easily evaluated. Furthermore, if you use the following formula:
```
=SUM(B1:B8,A7:D7)
```
Excel evaluates it as you would expect, summing the union of the ranges. So does PhpSpreadsheet. I guess there's no rule requiring Excel to be consistent, but ...
Another way that users might go wrong is by actually entering the union or intersection symbols in a formula rather than comma or space. Excel will not allow this; PhpSpreadsheet needs to change comma or space to the appropriate symbol in order for the rest of this PR to work properly. By the time the parser gets to it, it can't tell whether the symbol was part of the cell's original formula or if PhpSpreadsheet substituted it, so it just has to permit it. I do not expect many users to fall afoul of this problem, at least not more than once.
Just by way of explanation, the reason why PhpSpreadsheet has to make the substitution is because intersection has a higher priority than union, just as multiplication has a higher priority than addition. So, if you don't change the symbols beforehand, you may wind up figuring out that you need to perform an intersection too late - a lower-priority union may have already taken place.
With PhpUnit 10 came the ability to use Php attributes rather than doc-block annotations for things like "data provider". PhpUnit 11 deprecates the use of annotations, and PhpUnit 12 will not not permit their use. Since PhpUnit 11 requires Php8.2+, we cannot adopt it as long as we support Php8.1, which will continue to be the case for some time. However, there is no penalty for early adoption.
Php-cs-fixer can use:
```
'php_unit_attributes' => ['keep_annotations' => false],
```
This allows us to run `composer fix` to automate all the needed changes. No manual changes were needed for any of the test members.
With this change, PhpUnit 9 can no longer be used with the test suite. File composer.json is updated to reflect that reality, and phpunit9.xml.dist, which has been supplied in case anyone needed to use PhpUnit 9, is no longer required, and is thus deleted. For now, PhpUnit 11 is not being added as a possibility.
No source code is changed in this PR.
Fix#1310, which was closed as stale in 2020, but which I will now reopen. Supersedes PR #1311 (@jaiminmoslake7020), from which I will remove the stale label but leave closed. The issue and the PR were too limited - they detected that the use of two equal signs at the start of a string made for an invalid formula, but there are variations, trivial and otherwise, which might also be detected. Using `setValue` with a string which starts with an equal sign will now attempt to parse (not evaluate) the formula; for certain situations in which the parser throws an exception, the string will be treated as a string rather than a formula. An example where it will still be treated as a formula is a 3D range reference, where the problem is not that it can't be parsed, but rather that the formula isn't supported (see unit test Calculation/Engine/RangeTest::test3dRangeEvaluation). Allowing such a formula might cause problems later on, but that is already what happens.
A string beginning with an equal sign but which isn't treated as a formula will automatically set the `quotePrefix` attribute to `true`; all other `setValue` attempts will set it to `false`. This avoids the problem of a lingering value causing problems later on.
It has long been a matter of discontent that setting a style can change the selected cells. A new method is added to `Worksheet`:
```php
applyStylesFromArray(string $coordinate, array $styleArray)
```
This will attempt to guarantee that the active sheet in the current spreadsheet, and the selected cells in the current worksheet, remain undisturbed after the call. The setting of `quotePrefix` above is the first use of the new method.
This was suggested by the investigation of issue #3811. No fix is necessary for the issue. However, two possible code solutions (Php setlocale, which comes with certain design flaws, and StringHelper set(Decimal/Thousands)Separator were suggested, and neither is adequately tested. This PR adds such tests.
Unusually, getting StringHelper Decimal Separator, Thousands Separator, and Currency Code can result in a change to those properties. So, the existing design in several tests where those properties are captured in Setup and restored in Teardown do not work quite as designed. Instead, the ability to set those properties to their default value (null) is added, and the tests re-done to restore the default in Teardown.
The two methods yield the same results when parsing input. However, they diverge when examining output fields through `getFormattedValue`. Such output is currently correct (usually) when using setlocale, but not when using StringHelper. The former works through the 'trick' of using `sprintf(%f)`, which generates a locale-aware string. However, using non-locale-aware `sprintf(%F)` followed by `str_replace` will produce the correct result for both setlocale and StringHelper. One place in the code uses a cast to string, which is incorrect for both methods. Following that up with the same str_replace makes it correct for both. These changes permit, but do not require, the user to avoid setlocale altogether.
It remains an open question whether Settings/Calculation::setLocale should set DecimalSeparator, CurrencySeparator, and CurrencyCode. That makes logical sense, but it would be a breaking change, and having to explicitly set those values when using setLocale does not seem especially burdensome. For now, such a change will not be made.
While we might never be able to have 100% of our code strict, we can at
the very least do it for all of our tests. This ensures that our tests
are using our API with the types as intended by the test author, and not
silently be cast to what our API requires.
And quite a bit more manual changes. The idea is that typing of our
tests can be a bit more loose, so we assume PHPDoc is mostly correct. If
that happens to be wrong, it should be caught by the tests themselves.
PR #3513, developed by @SaidkhojaIftikhor, has been stuck for some time awaiting tests. This is the first of three PRs to replace that one. This PR also allows the use of slash as a thousands separator or decimal separator or currency symbol. These are, of course, very unusual situations; the main reason to support them is so that PhpSpreadsheet code will not crash when users set those options. New Calculation/Engine and AdvancedValueBinder tests confirm these. While making these changes, a few errors were found in AdvancedValueBinder and Calculation/Engine/FormattedNumber, e.g. currencies weren't parsed correctly when period was used as the group separator and comma as the decimal separator. Tests have been added for those situations.
PR #3513, developed by @SaidkhojaIftikhor, has been stuck for some time awaiting tests. This is the second of three PRs to replace that one. This accomodates the use of slash as a delimiter in column names in Tables and Structured References. The source changes are very simple. Additional tests exercise all the source changes.
There is also a preg_quote call when a table is renamed. I have also changed it to accomodate slash, because it's the right thing to do. But ... I can't think how to test it. PhpSpreadsheet will not allow you to set a table name to a string containing a slash (a test is added to confirm), and, if I manually update the Xml in an Xlsx spreadsheet so that the name does contain a slash, Excel will, understandably, complain that the file is corrupt.
Successor to PR #3523. There are 494 single-line changes (`public function provider` to `public static function provider`) in this PR. None of these were made manually; they were all created with the following script (adapted from
https://stackoverflow.com/questions/25909820/how-to-recursively-iterate-through-files-in-php):
```php
$dir = 'C:/git/unit10prep2/tests/PhpSpreadsheetTests';
$it = new RecursiveDirectoryIterator($dir);
// Loop through files
foreach(new RecursiveIteratorIterator($it) as $file) {
if ($file->getExtension() === 'php') {
$contents = file_get_contents($file);
$new = preg_replace('/public function (\\w*)([Pp])rovider/', 'public static function $1$2rovider', $contents);
if ($new !== $contents) {
echo "changing $file\n";
file_put_contents($file, $new);
}
}
}
```
After this PR, there will be one more, with a small number of test changes, and enabling PhpUnit 10 for Php 8.1+.
* Fix Unintential Deprecated Calls in Tests - INFORMATION
I think it's best to install these before PR #3166. There are no changes to source code, only to doc-blocks and to test members which continue to inadvertently use calls to deprecated functions.
* Missed Some Deprecated Calls
Fix them now.
Added a function to the new `FormattedNumber` helper that will use the currency code pulled from `localeconv` by `StringHelper::getCurrencyCode()`. The currency code is `preg_quoted` and then dropped into a regexp that is modelled on the expression developed for `convertToNumberIfPercent`. This will allow locale independent operation.
Unfortunately `localeconv` only provides information about standard currency formats and not accounting formats. This means that we can't say for sure whether or not a locale has an accounting format where the currency symbol shows up in a non-standard position (eg. left justified for some countries instead of appearing directly in front of the value). The regexp should handle these cases.
The primary issue with this approach is that the regexp will incorrectly match invalid currency formats for some locals as it checks for the symbol before and after the value.
A possible improvement would be to pull `p_cs_precedes` and `n_cs_precedes` from `localeconv` and using them to determine if the currency symbol should appear before or after the value for the current locale. Since white-space is ignored by the regexp, accounting formats should still work, as long as the accounting format doesn't involve moving the symbol to the opposite side of the value.
It gets awkward when the defined name is for an actual range rather than for an individual named cell; because we need to manipulate the stack when that happens.
The code is ugly, and this is a rather simplistic approach, but it works as long as the named range is a cell, a cell range, or even a "chained" range - it won't work if we have union or intersection operators in the defined range - but it does provide formula support that never existed before.
These changes have already been implemented twice, and been regressed twice. I'll try once more (with a different approach), then give up ...
As configured, Phpstan running under Php7 reports no errors. However, running under Php8, it reports 100 (!) errors. The vast majority of these are due to two reasons:
- renaming parameters in Php builtin functions in preparation for named parameters.
- using the new class GdImage rather than type resource as the argument type for many image-based functions.
Regardless of the cause, this will be a problem sooner or later. This PR is an attempt to get ahead of that problem. For source members, it mostly adds annotations or updates doc-blocks. Only 2 members have changes to executable code, and these are very minor - BitWise and Writer/Xlsx. For test members, all baseline errors are deleted and the code is fixed. Php7 and Php8 both report no errors with this configuration.
* Initial work modifying the way named ranges are stored, and handled by the calculation engine
This should provide better support for:
- both union and intersection operators in composite named range values
- MS Excel implementation of the union operator duplicating values
- named formulae
- named ranges and formulae that reference other named ranges and formulae
- ranges and formulae that reference multiple ranges across multiple worksheets
* Initial work on handling defined names (named ranges and named formulae) correctly
- UTF-8 names (already extracted as a separate PR and merged)
- distinction between named ranges and named formulae
- correct handling of union and intersection operators in named ranges
- correct evaluation of named range operators in calculations
- calculation support for named formulae
- support for nested ranges and formulae (named ranges and formulae that reference other named ranges/formulae) in calculations
* Minor tweaks before resolving merge conflicts
* Fix extractSheetTitle() method to work on the last ! in a cell reference rather than the first
* Throw exception if a the reference to a defined name in a formula doesn't exist as a defined name
* Properly assess scope for defined names in calculation engine
* Elimination of some redundant code
* Minor tweaks to simplify entries o the stack where we need to check type
* Ensure correct scoping rules are applied when evaluating named ranges and formulae
* Adjustments to Gnumeric Reader for new defined names structure
* Initial work modifying the Ods Reader to handle named ranges, they weren't actually supported previously... this is still ongoing work
* Handle Ranges formatted as 3-d ranges, as long as the references are both to the same worksheet
* Additional testing for Named Ranges formatted as 3-d ranges, as long as the references are both to the same worksheet
* Skip composite named range tests for the moment
* Clean handling for `undefined name` exception when thrown in the calculation engine. Catch and replace with `#NAME?`
* Adjust method we use to determine whether a defined name is a range or a formula
* PHPCS Recommendations
* PHP doesn't support `mixed` yet, at least not at the minium version that we're working with
* More phpcs fixes
* More phpcs appeasements
* Final phpcs fixes for the moment
Still have a lot of echo and var_dump() statements in the code that scrutinizer will hate, but they stay for the moment while this is still WIP
* Please let this be the last of the phpcs fixes
* Unit tests to determine whether a defined name value is a range value or a formula
* phpcs appeasement
* Named tests from provider
* Initial steps for named ranges and formulae in the Ods Reader
* Reading pseudo-3d range addresses in Ods; treat second sheet reference as being identical to the first, which is the majority of cases where this will occur
* Initial work on Gnumeric reader for named ranges and formulae
* Suppress debug logging again
* Remove more debugging displays
* Last minor tweaks before phase two
* Minor refinements
* And all for the want of a space
* A little tidying up
* More tidying up
* phpcs fix
* Modify defined names in rebindParent()
* Renaming variables
* Resolve an issue with locally scoped defined names that don't contain any worksheet reference
* Keep phpcs happy
* Fix quote handling in regexp
* Fix a couple of scrutinizer issues
* Fix a couple of scrutinizer issues
* Update Xlsx Writer to work with the new defined name internal definition
Additional validation checks
* When adding new defined names through the readers, worksheet may not exist if we're only loading selected sheets rather than the full spreadsheet
* If the only thing that phpcs can pickup on is strings in double quotes instead of single quotes, then I know I'm getting close to ready
* Refactor Defined Names logic for Xlsx Writer into its own class
* phpcs keeping me on my toes
* Restore a couple of files that I managed to change without intending to
* Initial work on Ods Write to provide support for saving named ranges and formulae
* Resolve commas to semi-colons s argument separator when writing named formulae for Ods
* Extract Named Expression Writer for Ods into its own class
* Keep phpcs happy
* Refactoring of formula conversion when reading SpreadsheetML; preparation for reading named ranges because they will also need to use the same conversion method
* First pass at reading Named Ranges/Formulae from SpreadsheetML format xml files
* Remove unused namespace reference
* Defined names being written correctly for Xls; but not yet writing cell formulae that reference those defined names... that's the next big step
And I anticipate that defined names that reference other defined names will also be a problem
* Just to keep phpcs happy
... and yes, I know that there are still diagnostic echo statements in the code
* I had to miss some of the phpcs issues didn't I
* Work on the Xls Writer's Parser Tree to identify named range tokens in a formula, and to distinguish them from function tokens
* Still working on packing that d*** defined name reference in the writer
* Throw an exception in the Parser for saving Xls output if we encounter a defined name in a formula... writer will simply write the calculated cell value, and not the formula as at present
Strip out diagnostic output
* Some phpcs appeasement
* Fix a couple of Scrutinizer issues
* Additional verifications to differentiate a formula from a range value
Add explicit getters/setters for named ranges, named formulae and defined names
Additional unit tests
* Styling for closures
* Remove redundant docblocks
* Spaces
* Gah! Namespace use complaints
* Consistency of making calls to DefinedName rather than NamedRange; NamedRange should now be used only for Named Ranges, and should exclude Named Formulae
* Styling
* spurious newline
* No need to test for variable === null when we're typing it in the function argument definition
* Additional unit tests for local/global scoped named ranges and formulae; and a fix to getNamedFormula()
* Fix silly typo that led to breaking test
* Void return signature for unit tests
* Why weren't these picked up in the last pass?
* Refactoring of getNamedRange()/getNamedFormula()
* Eliminate unused constants, and defaults for private method parameters when always called with a value
* Use strict comparisons when comparing object hash codes
* Initial update to documentation for working with named formulae
* Fix for calculation of relative cell references in named ranges/formulae
* Fix current named range tests, because we should be using absolute references; tests for relative named ranges to be added later
* Fix for calculation of relative cell references in named ranges/formulae
* Updates to changelog and documentation for handling of absolute/relative references in named ranges
* Fix last remaining unit test with a named range reference
* Refactor formula conversion for Ods into a separate class; I hadn't realised that it previously wrote formulae as the MS Excel syntax without any conversion to Ods format
* Fix Ods Writer test xml to reflect Ods-native format for formula
* Docblocks
* Drop dollar prefix from Ods formulae and ranges unless it's necessary
* Set the formula convertor in the content writer constructor
* Documentation update
* Minor updates
* Remove var_dumps from file
* Fix the spurious single quote that was breaking named expressions in the Ods Writer... big sigh of relief that I finally spotted it
* Starting work on documentation for Defined Names, and some examples of using Named Ranges and Formulae
* Starting work on documentation for Defined Names, and some examples of using Named Ranges and Formulae
* Example of a relative named range for the documentation
* Mustn't have phpcs problems in sample code either
* More updates to the documentation
* That should conclude the documentation for Named Ranges, now time to move on to documenting Named Formulae
* That should conclude the documentation for Named Ranges, now time to move on to documenting Named Formulae
* PHPCS appeasement in sample code
* Initial documentation on Named Formulae
* PHPCS appeasements
* Additional comments in the documentation, and modify the named range name validation to support a \ as the first character in a name
* Fix breaking build
* Make defined names case-insensitive
* Fix case-insensitivity
* Improved documentation, and additional unit tests
* Additional unit tests, and a fix for removing a globally scoped defined name even if a worksheet is specified in the method call
* Fix unit test for removing named formulae
* Use assertCount instead of assertSame
* Forgotten voids
* Fix arguments for assertCount
* Unit tests for removing defined names, and a fix for removing locally scoped names
* Unit tests for absolute and relative named ranges in calculation engine, and fix an issue with worksheet name in the offset adjustments for relative references
* PHPCS Appeasement
* Additional unit tests, more documentation, and a fix to the calculation engine when no worksheet reference is provided with a named formula
* PHPCS appeasements
* Additional documentation and examples of using Named Formulae
* Additional examples to go with documentation
* A few minor phpcs appeasements
* Minor refactor of updateFormulaReferencesAnyWorksheet() method
* Discard an unused method argument
* Additional unit tests
* Additional unit tests
* Remove unused argument
* Stricter typing
* Fix return typehinting from remove named range/formula; should return the Spreadsheet object
* Use return typehint of self rather than explicit object type
* Redundant code just to keep scrutinizer happy
* Minor change to handle merge conflict
* phpcs fixes after merge
* Namespace usage ordering
* Please let this be the last phpcs fix needed
Co-authored-by: Adrien Crivelli <adrien.crivelli@gmail.com>
* Improved handling of named ranges, although there are still some issues (names ranges using a union type with an overlap don't handle the overlap twice, which as the MS Excel approach to set overlaps as opposed to the mathematical approach which only applies overlap values once)
* Fix tests that misused space and comma as simple separators in cell ranges
oleibman
Adrien Crivelli
MarkBaker
fjohnston@avatarasoftware.com
MarkBaker