As was pointed out, the entire `pow` piece, as well as extracting exponent and exponent sign, was not necessary as casting to `float` will correctly handle scientific notation in a string.
Also added case-insensitive flag to the regexp to handle upper and lower case `e`.
Originally thought I was being clever by using zero in `pow` to get 1, however after thinking about it all day I think it's probably better to avoid the call to `pow` if the exponent is zero; one less function call that has to be made.
I didn't take the time to look into benchmarks of `pow` to the zero vs the ternary to multiply by 1, but my gut tells me this is likely slightly faster.
* Eliminate Some Scrutinizer 'Major' Problems Part 4
Mostly docblock changes and annotations. Some code changes. A few more of these to go still.
* Missed a Couple
Fix them now.
Implemented the suggested named capture groups (nice trick) in the regexp to pull out only the values we need to perform the calculation. Making use of the null coalescing operator to keep things nice and clean. Note that I did have to add the `PREG_UNMATCHED_AS_NULL` to `preg_match` to ensure PHP returned null (since null coalescing only works on null and not '') when a match wasn't found for a capture group.
Also extended the regular expression to handle scientific notation (both positive and negative, and positive and negative exponents) as well as a negative symbol before or after a leading percentage sign.
* Problems Formatting Very Small and Very Large Numbers
Fix#3128 (author found a workaround but the problem remains). For some complex masks, when a cast of the cell value from float to string results in the use of scientific notation, the result of the formatting is unusable. I believe this PR solves the problem for numbers close to zero (Php cast from float to string uses scientific notation starting with 1E-5), and for a range of large numbers which are not handled correctly now.
However, I have not found a way to ensure that the results match Excel for very large numbers (1E18 or larger); this change at least ensures that the resulting string is an accurate rendition of the number (which is not the case now) even if doesn't match Excel. As an example, if you use the mask reported in the original issue `0 000.0` and enter a value of 1E90 into the cell, Excel will show it as 1 followed by 87 zeros, a space, 3 more zeros, decimal point and zero. I have not figured out how to get PhpSpreadsheet to do that; for now, it will just return the formatted value as 1 followed by 90 zeroes instead (I might have chosen to go with scientific notation instead). I will continue to think about those, but do not feel it is worth delaying the improvements in this ticket while I do so.
The affected section of code also truncated to the appropriate precision. It now rounds, as Excel does.
This seemed to be an area of code where problems might arise on 32-bit systems, and, indeed, I found something in the formatting code which had to be changed for 32-bit to work correctly. As long as I was doing that anyhow, I ran the full test suite, and found that Php8.1 had introduced some new stringencies which caused problems in a handful of places. All were found in Xls Reader, and all are corrected now.
* Remove Dead Assignment
Scrutinizer will be happy now.
As suggested, the initial check has been tightened up using a regular expression.
Expressing allows for leading and trailing spaces, leading and trailing percentage symbols, with spaces mixed throughout.
Added 57 permutations of leading and trailing spaces, leading and trailing percent symbols, decimals, and negatives. All seem to be working. I will also attach an Excel worksheet that verifies the behaviour of all 57 test cases aligns with Excel.
* MATCH Problems with Int/Float Compare and Wildcards
Fix#3141. Function matchSmallestValue did not recognize that an integer could match a float. Adding test cases, it seems that matchFirstValue had the same problem. However, matchLargestValue seemed to handle things correctly - but see below.
In addition, the wildcard logic in matchFirstValue is faulty. It ignored tilde as a wildcard character. Although it would have been easy to just add that, I think it was wrong to determine on its own if a wildcard was in use. Just using the already available wildcard functions whenever comparing two strings is sufficient.
I note that Excel doesn't seem to follow its own rules for MATCH (https://support.microsoft.com/en-us/office/match-function-e8dffd45-c762-47d6-bf89-533f4a37673a?ns=excel&version=90&syslcid=1033&uilcid=1033&appver=zxl900&helpid=xlmain11.chm60112&ui=en-us&rs=en-us&ad=us). PhpSpreadsheet's results match Excel's, so no problem. However, when match_type is not zero, the match array is supposed to be sorted, so I would expect `#N/A` when it isn't; but that's not how Excel operates. I have no idea what Excel is doing. If `MATCH(2,{2,0,4,3},1)` isn't `#N/A` because of the unsorted array, then surely it should be `1` (item 1 of the array is the largest number less than or equal to the lookup value); but Excel and PhpSpreadsheet (before and after changes) return `2`. I have moved this example to be the first of the test cases.
One would think strings would behave similarly. But, no - see the second test case. This time Excel does look for an exact match. But the existing logic doesn't get the matching result in PhpSpreadsheet. It requires a whole new block of code, one which doesn't work correctly for numeric lookup value. Ugh.
LibreOffice doesn't always agree with Excel. It seems that it will use wildcard matching even when the match type is not zero (Excel documentation says wildcards are only for type zero, which is just as well because I don't really know what greater/less mean when wildcards are involved). I have not attempted to duplicate this behavior. For the record, Gnumeric agrees with Excel here.
* More Changes - LibreOffice
Add support for LibreOffice matching wildcard strings when type is not zero. Add support for type to be specified as integer other than 0/1/-1, or as float, or as numeric string; non-numeric string should case `#VALUE!` error.
I have found an example of undefined behavior (unsorted array where type is non-zero) where PhpSpreadsheet does not produce the same result as Excel. It is present as a new `incomplete` test case. I can fix it, but not without breaking other tests where the proper behavior is undefined. IMO, this is not a problem we should be concerned about.
Many test cases are added. Chances are I will add some more before merging this change.
Adding two test cases:
1) Test the function in `StringHelper.php`. Modeled this test on the test for the `convertToNumberIfFraction` function.
2) Test a spreadsheet with a string percentage in a formula to see if it calculates the formula correctly.
Further to https://github.com/PHPOffice/PhpSpreadsheet/issues/3155, I noticed that percentages stored as strings in Excel were still being used in formulas; Excel is apparently converting them behind the scenes. The same spreadsheet imported into PHPSpreadsheet was outputting a #VALUE! error.
Added a function, modeled on the `convertToNumberIfFraction()` function, to detect if a string contains a percentage, and if so, convert it to a numeric. If the last character of the string is a % sign, the function will string the % sign and divide the value by 100, overwriting the original param with the new value.
New helper function was added to `StringHelper.php` along the original faction helper.
* Generation3 Copy With Image in Footer
Fix#3126. A worksheet contained an image in its footer. It could be loaded and saved as another spreadsheet. However, if you tried to load and save that spreadsheet, PhpSpreadsheet would be unable to find the footer image and would therefore throw an exception.
This error was introduced a long time ago, in PhpSpreadsheet 1.3.0. The apparent cause of the problem was PR #435, sometime around June 2018. That change was very useful, but it had problems which exposed themselves only with a third generation copy. An additional contributor to the issue at hand was PR #1690 (December 2020), which again exposed itself with a third generation copy.
The issue from 1690 is easier to explain and deal with. It added a 'ps' suffix to printer settings resources in Xlsx Reader (to avoid name conflicts), but did not limit itself to a single addition (so subseqent generations would have multiple ps's). It also neglected to add the suffix in Reader/Xlsx/PageSetup.
As for 435, it loops through all the worksheet relationships, and uses the last that it finds as the base for header/footer drawings. It has been changed to use only the relationship whose `rId` matches the worksheet's `legacyDrawingHF` `rId`. It also needs a bit extra validation to make sure a drawing exists before adding it to its array of header/footer images. It also meant that Xlsx/Writer/Rels might write an entry with the same rId twice. I have also changed the header/footer image processing to be namespace aware (see PR #3137).
* Minor Change
I didn't like the way I performed one operation.
* Fix Test
Array index should not have been constant.
* Xlsx Reader Namespace Aware Handling of Drawings, RowAndColumnAttributes, MergeCells
Fix#1482 (actually fix a problem recently attached to that ticket long after it closed). There were problems processing a spreadsheet generated by third party software. That spreadsheet used unexpected namespacing, and absolute paths within the zip file where relative paths were expected.
Xlsx Reader handles most, but not all, of its processing in a namespace-aware manner. Two versions of a worksheet's xml are available - `$xmlSheet` is not namespace aware and `$xmlSheetNS` is aware. This was necessary in order to add namespace support in an incremental manner. The primary reason to continue to use the unaware version is the absence of test cases. In particular, drawings, row and column attributes, and merge cells continue to use the unaware version; this PR changes those to use the aware version.
As noted in the summary above, a couple of new places in the handling of the those items were expecting file locations to be specified as relative paths in the zip file, but the file used absolute paths instead. Those unexpected usages are now addressed.
The user reporting the new problem tried a change which effectively made all uses of `$xmlSheet` namespace aware, and that seemed helpful. It may be time eliminate its usage altogether, whether or not we have appropriate examples of unexpected namespaces to test with. I will not do that with this change, but I may add a new PR to do so after this one is merged. Remaining areas which still use the unaware version include conditional formatting (internal or external), sheet view options, sheet protection, auto filters, unparsed loaded data, data validation (internal or external), alternate content, and header/footer images.
There is an interesting anomaly with the new test file. When I load it and save it, the appearance of the output file does not quite match the input. Oddly, the output file seems much better than the input - the picture no longer covers any data, for example. This is because, in particular, the output file row heights and column widths seem to match the xml, but the input file does not. For example, the xml in both files seems to indicate that row 5 should have a height of 234, which it does in the output file, but the height of that row when the input file is opened is 156. It appears that all row heights and column widths when the input file is opened are very close to 2/3 of what is expected. I will continue to research that anomaly for a few days, but I will not let it prevent me from moving forward with this PR if I don't find the explanation. Whatever that problem is, it seems distinct from the namespacing/pathing problems which the PR addresses.
* Scrutinizer New False Positives
Eliminate them with annotations.
* Using htmlspecialchars to fix issue #3145: this prevents the SimpleXMLElement error "unterminated entity reference" when HTML entities are present in a worksheet
* replace htmlspecialchars with proposal of oleibman
* eliminate string cast, because PhpStan has a problem with that
* ignore string cast for phpstan + enhance unit test
* Eliminate Some Scrutinizer 'Major' Problems Part 3
Continuing the work of PR #3109 and PR #3122. Nothing unusual about the changes in this ticket. I expect 4 or 5 more after it will be needed to finish the job.
* Scrutinzer Strikes Again
A couple of new false positives.
* Tidy Up Earlier Change
Conform to a better approach we figured out later.
* Eliminate Some Scrutinizer 'Major' Problems Part 2
Mostly Scrutinizer bizarre false positives. From Calculation/Engineering/Bessell (and several others):
```php
private static function besselI2(float $x, int $ord): float
{
if ($x === 0.0) {
return 0.0;
}
```
Scrutinizer complains that `$x` can never equal 0.0 here. Huh???
Another example repeated several times, from Calculation/Engineering/BesselJ:
```php
$bool = false;
foreach (whatever) {
...
$bool = !$bool;
...
}
```
Scrutinizer complains about the `!$bool` assignment because it says `$bool` is always false. Again, huh???
* Change Logical Not Handling
From a suggestion from @MarkBaker, sidestep bug in Scrutinizer by recoding
```php
$jsum = !$jsum
```
as:
```php
$jsum = $jsum === false;
```
* Permit Date/Time Entered on Spreadsheet to be Calculated as Float
Fix#1416. I do not entirely understand the use case for this old issue, but resolving it seems straightforward. Issue complains that user-entered date/time fields may be interpreted as either float or int when PhpSpreadsheet reads them. Issue suggests getCalculatedValue treat all date/time fields as float; that seems like a breaking change. However, adding an option to permit it seems okay. That option might be implemented as either a property of Calculation, or a static property of Cell. Since the changed logic is found in Cell (and Shared/Date), I opted for the latter.
In Cell, the property `$parent` is incorrectly described in doc block as `Cells`, and should be `?Cells`. This change eliminates some Phpstan and Scrutinizer problems, and should allow the elimination of some try/catch blocks - I have not done an exhaustive search for those.
Calls to `isDateTime` could have affected activeSheet and selectedCells; they no longer can. Optional parameters are added to it and the functions it calls to accommodate the new functionality; the defaults for the new parameters will, of course, return the same result as the earlier versions of the functions would have returned.
* Scrutinizer - Self-inflicted
Tests used constant which I deprecated.
When adding support for non-standard namespacing to Reader Xlsx, I changed most (hopefully all) the uses of string literals for the namespaces to class constants instead. Writer Xlsx naturally uses all the same namespaces, but has continued to use string literals. This PR replaces those with the same constants used by Reader Xlsx.
Almost all of these are handled through annotations. This shouldn't be our "go-to" solution, but it becomes necessary because Scrutinizer's analysis is often incorrect. Here is a typical example, from Cells.php.
```php
if ($this->currentCellIsDirty && isset($this->currentCoordinate, $this->currentCell)) {
$this->currentCell->detach();
```
Scrutinizer complains that `$this->currentCell` can be null here, but the `isset` condition guarantees that it must be non-null. Perhaps Scrutinizer is worried that `isset` might be overridden, and will accept only an explicit equality test for null for each of the isset arguments. Changing the code to do this seems riskier than just adding the annotation.
A far more common, and more frustrating, example is:
```php
foreach ($simpleXmlElement as $element) {
var_dump($element->method());
}
```
Scrutinizer complains that element might be null. I don't think it can. I have previously added code in places to eliminate the objection, and that may be a practical solution when `$element` is used many times in the loop. But, when it's used only once, annotating the objection away seems like a better solution (less overhead, clearer code). Many of the changes in this PR fall into this category.
* Fix Remainder of Calculation vs. Phpstan Issues
I had tried to include these changes as part of an earlier effort, but something about them broke Phpstan. I removed them until I could determine the actual cause, which is ...
The array `$phpSpreadsheetFunctions` is a very large and very complicated array, typehinted as `array`. It is declared as private static; however, its content never changes (at least not now - there are some outstanding proposals that might change that). I have had some success with changing unmodifiable private static to private const. However, such a change here causes Phpstan to perform a lot more processing and eventually time out. So, leave it as static.
Having identified the cause of the problem, none of the other changes were problematic, so this PR applies the rest of them.
* Scrutinizer
One problem.
* Scrutinizer False Positives Without Suggested Annotation
See if minor code changes can make these go away.
* Scrutinizer Still Experimenting
Trying again.
MarkBaker
fjohnston@avatarasoftware.com
oleibman
Jokūbas
Robin
Rod Elias