This change is extracted from PR #2787 by @MarkBaker. That change mostly deals with array functions, and that part will be superseded by PR #3962. However, this part of 2787 is not included in 3962.
Fix#1036 (closed as stale in 2019 and just reopened). Excel spreadsheets can have either of 2 base dates, 1900 or 1904, and the numeric value of any date cells will vary depending on which base date is in use. PhpSpreadsheet has, till now, handled that as a static property of Shared/Date. This does not work well if two spreadsheets with different base dates are open simultaneously. The code is changed to store the base date as a property of the spreadsheet when an Xls/Xlsx spreadsheet is loaded, and use that property when saving an Xls/Xlsx spreadsheet. Any call to `getCalculatedValue` or `getFormattedValue` will temporarily set the Shared/Date value to that of the spreadsheet, and restore it at completion. In order to avoid a BC break, the Xls and Xlsx readers will continue to populate the Shared/Date value as before.
Following up from PR #4006. There is an additional RTL property available. It controls the placement of bidirectional neutral characters (mainly punctuation), as opposed to strong (alphabetic characters) or weak (numeric characters), especially at the beginning or end of a line. The new Comment property textboxDirection will be used for that purpose.
In a discussion in issue #4004 following the implementation of the PR, the comment was mixed RTL and LTR, and this led to some formatting problems. The user was able to overcome these with the timely insertion of Unicode directional control characters, but it would be preferable to have it happen automatically, which this change will permit. However, the use of these control characters cannot be entirely done away with. In the new test case, if one of the all-English lines ended with, say, a colon, it would not display correctly; LRM (left-to-right mark) after the colon would be needed. Likewise, one or two of the comment lines with mixed RTL and LTR (discussed in the issue) is not formatted correctly, and might require LRO/PDF or equivalent.
Fix#4063. Fix#1560. Fix#1293. PhpSpreadsheet is not accounting for an empty string in Xlsx sharedStrings.xml.The code which parses it in Reader/Xlsx looks for a `t` or `r` tag descending from `si`, but, in this case, the tag is coded as `<si/>`, with neither t nor r tag descending. An else clause is added to set the string to empty string in this case.
I was surprised that this had not turned up before, and a search through the archives found at least 2 earlier reports from 4 years ago. Those had been marked stale; the stale indicator is removed, and the issues are re-opened, to be closed when this PR is merged.
Continuing work started with PR #4016 and PR #4026. Improve documentation within program by making explicit what types of values are allowed for variables described as "mixed". In order to avoid broken functionality, this is done mainly through doc-blocks. This will get us closer to Phpstan Level 9, but many changes will be needed before we can consider that.
This change has more executable code changes than its predecessor. I will wait longer than normal before merging it to allow for additional testing.
Dependabot suggested some changes this month which required an unusual effort to implement successfully. With the elimination of Php 8.0 as a supported environment, it became possible to use Phpunit 10 rather than 9. Among other considerations, the configuration file for Phpunit is changed. I preserved the Phpunit 9 version under a different name. Aside from the configuration change, several other changes needed to be made to accommodate the change:
- Fix#3993. A peculiar problem indeed. One of the reporters said it had something to do with mocking, but I couldn't duplicate it. But Phpunit 10 revealed the problem in one test (Reader/Xlsx/AutoFilterTest), and that was sufficient for me to apply the trivial source code change to Worksheet/Worksheet.
- Some extra stringency on data providers required extra work in Calculation/CalculationFunctionListTest.
- There was a misplaced label in Calculation/ParseFormulaTest. Likewise in the data member CellGetRangeBoundaries.
- More stringency required changes to data members Shared/Trend/ExponentialBestFit and Shared/Trend/LinearBestFit.
- Issue3982Test testLoadAllRows seemed to go into a memory-acquiring loop in 10 that was not evident in 9. This particular test uses a lot of memory by design, but was included only to establish a base level for the other tests in that member. I feel it is acceptable to skip it for 10.
Phpstan errors with a new release are not unusual. One of the problems this time around was, however, unusual - it is fixed when Phpstan runs under Php8.3, but not for earlier Php releases. Our tools currently use Php 8.1. It is on my to-do list to get Phpstan and other test tools running under 8.3 before 8.1 goes EOL. There are also some configuration file changes needed for Phpstan.
Php-cs-fixer has been taking an increasingly long time to run. They've added an experimental option to permit it to run its checks in parallel. I've changed its configuration to use that option.
Fix#4049. Some possible options were not included for read or write. In addition, although it isn't well documented, it appears that 2-color scale always has 2 cvfo entries in Xml in order minimum/maximum, and 3-color scale always has 3 entries in order minimum/midpoint/maximum.
Provide a means to convert a range, possibly with unions and possibly with intersections, into something that both Excel and PhpSpreadsheet can handle. Intersections are changed into unions of the individual cells which they comprise. With this change, Xls Writer now handles intersections (previously it would have thrown an Exception or created a corrupt worksheet if this was attempted), and Xlsx Writer works correctly (it seemed to before, but Excel didn't understand what it wrote). Worksheet::getConditionalRange and ::getConditionalStyles would previously have thrown an Exception when presented with an intersection, and will no longer do so.
**NOTE:** Intersection support is limited to Conditional ranges. Use of intersections in other contexts will usually not achieve the desired result.
Fix#3557. Borders around merged cells are not handled correctly for Mpdf. Although a perfectly acceptable workaround is suggested in the issue, it would be better if things just worked without the workaround. Html and Dompdf work with the existing code. As it turns out, Tcpdf does not work, but for a different reason than Mpdf.
Mpdf was not working because Mpdf does not honor the `!important` attribute in Css. We can get it working almost perfectly by suppressing `border*:none`; the exception is fairly Byzantine, and I'll be glad to discuss the matter should anyone report a problem with it. At any rate, it's not working now in the exception case, so we won't be any worse off.
Tcpdf was not working because the merging of attributes happened only when `useInlineCss` was not being used, but Tcpdf does use it. Merging of border attributes is now added for useInlineCss.
Xls Reader processing Conditionals interferes with the previously established SelectedCells. Make sure that value is restored.
StopIfTrue should always be set for Xls spreadsheet.
Set NoFormatSet to true unless any of Font, Fill, or Borders is specified in Conditional Style.
In my notes for PR #3372, I mentioned that I could not include some Xls tests because of errors in the software at that time. This PR fixes those errors, so I am adding the missing test, and making the equivalent Xlsx test more comprehensive.
Fix#4039. Excel sometimes stores the location for a Conditional as, say, `B1:B10 C1:C10` rather than `B1:C10`. PhpSpreadsheet does not have a problem with this, but internally stores it as separate Conditionals, one for each range. (There may be more than 2.) User would like it stored as a single Conditional. Since the range is used as the index of an array which holds the conditionals, there is no technical reason why this can't be done. And it does seem like being able to change multiple ranges all at once has some advantages, whether you're doing it in PhpSpreadsheet or in Excel itself.
Making such a change in Xlsx Reader showed no problem in Xlsx Reader and Writer tests. A number of problems did show up in CellMatcherTest, and one in WizardFactoryTest. This was good news, because it meant some of our test spreadsheets used the same type of construction as the test spreadsheet supplied with the issue. So, if additional fixes make the test problems go away, I think no new tests are required. And a smattering of source changes did indeed result in a clean test suite once more.
This is a change, but I don't really think it should break anyone. So I don't think it's necessary to add an option to opt in to the old or new behavior. I could be wrong. I'll leave this ticket open for at least a couple of weeks to see if anyone thinks otherwise.
Fix#4036. The issue was originally reported as #564 (and #811) and fixed for the most part, but this is a variation that was not covered by the original. Cells with html fragments can cause `mime_content_type` to identify the file as `text\html`. Original fix was to ignore mime_content_type when file extension is 'csv' or 'tsv'. However, if the file does not have one of those extensions, it will be rejected by Csv Reader as invalid mimetype. This PR adds text\html to the list of valid mimetypes.
I imagine that this type of problem might occur for other mimetypes. If any of those are reported in future, it might be better to just add a "suppress mimetype" check option, rather than extending the list forever. Html is unusual in that its rules are so lax, which is why it seems appropriate to add it here.
Note that IOFactory may still identify a file as Html even when intended as Csv. The sample associated with this issue does not fall into this category, but one of the unit tests on this ticket does. The file will still be read correctly by Csv Reader, but IOFactory load may cause it to use Html Reader instead.
Fix#3982. A number of issues submitted about Xlsx read performance have a common theme, namely that row 1,048,576 and a few rows before it are defined in the worksheet Xml with no cells attached to them. These might be the work of a third party product. While these extraneous rows do not cause any problems for the cells that are actually used on the worksheet, they can lead to excessive memory use. This PR provides an option for the application to ignore rows with no cells when loading.
Recent changes to the load logic had already made a significant difference to memory consumption and load time. For the spreadsheet attached to issue 3982, which had caused out-of-memory errors on the user's system, peak memory usage was already reduced to 40-odd MB. With the new option, this is drastically reduced again, to just over 9MB. Specifying the new option is very easy:
```php
$reader->setIgnoreRowsWithNoCells(true);
```
Note that there are cases where you might not want this (non-default) behavior. For example, if you set a row height on a row with no cells, the height would be lost with this option. Unfortunately, the extraneous row definitions in the problematic spreadsheets claim to have a custom height, so I can't just use "no custom row styles" as an additional filter.
Xls Writer Conditional Border had been creating corrupt spreadsheets. This was mainly because a pack statement that should have specified `V` instead specified `v`. Even changing that, the logic was still slightly wrong on write, and missing altogether on read. This PR corrects the write problems and adds the missing read code. It also adds italic and strikethrough support for Xls Writer Conditional Font italic and strikethrough (read code was already in place). With this, Xls Conditional Writer is completely supported except for NumberFormat. Xls does support that, but I cannot figure out how from the available documentation.
Investigating issue #1622, I found that its extremely complicated formula, which had been leading to an incorrect result, now led to an exception. I am able to fix the exception; unfortunately, I am no closer to resolving the original issue. So I'll apply the baby step while continuing to investigate. Function POWER had changed from untyped args to a defined set of types. The set of types was determined according to the doc block, but that was incomplete - it had neglected to include `null` and `bool`. This PR corrects the function prototype and the doc block, and adds the missing tests for those conditions.
While researching another problem, I noticed that font color was not working as expected for Xls Conditional Formats, at least not when a "non-standard" color is used. In such cases, the color might wind up being rendered as black. The reason is as follows. Xls Writer includes a color palette which is dynamically generated from the (non-Conditional) styles used in the workbook. Any colors used in the workbook are indexes to this dynamic palette. However, Conditional colors use a static palette found in class ColorMap to determine the index, so the determination of index will often not find a match, and, if a match is found, it is not necessarily correct. (Also, the ColorMap method was case-sensitive and needs to be insensitive.)
In order to correct this, the `addColor` method in Xls Writer Workbook needs to be accessible to the Conditional logic which is found in Xls Writer Worksheet. This is accomplished by passing the Workbook in the Worksheet's constructor, and changing the method to public, and changing Conditional Font to use this method rather than ColorMap.
The logic for Conditional Fill colors is similarly changed. Although Xls Conditional Fill has appeared to just not work, I was finally able to figure out the problem. Excel Xls Conditional Fill with fill type Solid requires that the fill color be specified as startColor, and that endColor be omitted. Our conditional samples used endColor, and are now changed to use startColor instead; the same is true for our online documentation, and for some tests. Xlsx continues to work as expected, and now Xls does at least some of the time. If the condition is one that Excel Xls does not recognize (e.g. cell contains), it will, of course, not work. A surprising situation that also doesn't work is the use of ISODD or ISEVEN in formulas. Those are "add-in functions" which are handled differently than other functions, and I'm not sure how to support them. I will document this in issue #3403.
Samples 08_Conditional_Formatting(_2) had produced corrupt Xls versions. This turned out to be because the code was using hash codes to avoid having to write out duplicate conditionals; this is often a good idea, but not in this case. Allowing the duplicates fixes the corruption problem.
Conditional Border colors also ought to figure in this change, but the current code does not support Border colors, and I have not yet been able to figure out how to implement it (BIFF format can be very messy to figure out).
With this change, I could delete ColorMap altogether. However, it is a public class with a static public method, so maybe someone is using it for a purpose I'm not familiar with. I will just deprecate it.
Fix#4025. With Conditional Formatting, Excel can change Font, Fill, Borders, and NumberFormat. It does not support Alignment nor Protection, at least not natively. This PR removes all code supporting Conditional Alignment or Protection from Xlsx and Xls Writer. Attached to the issue report is a spreadsheet where a conditionally formatted merged cell shows its value twice when Alignment is included with the Conditional Style; it is fixed by this PR. It is not entirely certain which particular combination of CF, Alignment, Merge, AutoFilter, and right-to-left sheet alignment trigger this problem. I am sure that it shows up for Vertical Alignment Center, and I am sure that Xls Writer (which is somewhat buggy anyhow regarding CF support - see issue #3403) has the same problem. The now-unused Xls code is commented out rather than deleted, for documentation purposes.
Fix#3995. Fix#866. Fix#1681. Php DOM loadhtml defaults to character set ISO-8859-1, but our data is UTF-8. So Html Reader alters its html so that loadhtml will not misinterpret characters outside the ASCII range. This works for UTF-8, but breaks other charsets. However, loadhtml uses the correct non-default charset when charset is specified in a meta tag, or when the html starts with a BOM. So, it is sufficient for us to alter the non-ASCII characters only when (a) the data does not start with a BOM, and (b) there is no charset tag.
This will allow us to use:
- UTF-8 files or snippets without BOM, with or without charset
- UTF-8 files with BOM (charset should not be specified and will be ignored if it is)
- UTF-16 files with BOM (charset should not be specified and will be ignored if it is)
- all charsets which are ASCII-compatible for 0x00-0x7f when the charset is declared. This applies to ASCII itself, many Windows and Mac charsets, all of ISO-8859, and most CJK and other-language-specific charsets.
We cannot use:
- UTF-16BE or UTF-16LE declared in a meta tag
- UTF-32, with or without a BOM (browser recommendation is to not support UTF-32, and most browsers do not support it)
- unknown (to loadhtml) or non-ASCII-compatible charsets (EBCDIC?)
I will note that the way I detect the `charset` attribute is imperfect (e.g. might find it in text rather than a meta tag). I think we'd need to write a browser to get it perfect. Anyhow, it is about the same as XmlScanner's attempt to find the `encoding` attribute, and, if it's good enough there, it ought to be good enough here.
Scrutinizer balked at quotes around key 'Unit Name'; no reason not to use 'UnitName' instead.
Having done that, several ConvertUOM tests seemed a little unconvincing. Replaced them with improved versions.
Improve documentation within program by making explicit what types of values are allowed for variables described as "mixed". In order to avoid broken functionality, this is done mainly through doc-blocks. This will get us closer to Phpstan Level 9, but many changes will be needed before we can consider that.
Executable code change is to add 2 new methods to Cell - `getValueString` and `getCalculatedValueString`, which are the same as the methods without the String suffix, except that they always return a string result. These are invoked at several places in the code where it makes sense, in particular by Writer code.
Fix#4001. Thanks to @SlowFox71 who reported the problem and developed most of the solution. This PR adds Rich Text support to the XML reader. The Xml Spreadsheet stores Rich Text as Html tags, children of the ss:Data tag using a specific namespace. These can be parsed into a RichText object using existing method Helper/Html::toRichTextObject. There are 2 items which need special attention.
First, for attributes like bold or italic, Excel uses the appropriate Html tag (e.g. `<B>`). However, for an attribute like color, Excel uses `<Font html:Color="#FF0000">`, with a prefix on the Color tag. PhpSpreadsheet's Html parser cannot cope with the prefix. The parser is changed to strip `html:` from attribute names for the Font tag.
The example cited by the user used a `<BR />` to indicate a line break in the data. However, it appears that, at least some of the time, Excel will instead use ` ` to indicate a line break. The existing parser reduces one or more whitespace characters in the text to a single space, and so ` ` will wind up disappearing. I am not sure why the existing code does this, but I do know that I am not willing to break it. Instead, I've added an optional boolean parameter `$preserveWhiteSpace` to `toRichTextObject`. If false (default), the existing logic will be used; but if true, substitution for whitespace characters in the text will not happen.
Fix#4004. RTL text can be included in a comment, and will display correctly, but the comment as a whole will be left-aligned. There already exists an `alignment` property for Comment, but it is unused. This PR will allow that property to be set, and to be read from and written to an Xlsx spreadsheet when possible.
The Xml tags that govern this property are found, unusually, in a drawing Vml file. The important property is `x:TextHAlign`, which can be set to Left, Right, Center, Justified, or Distributed. There are other tags which seemed like they were relevant to this problem, but I don't think they actually matter:
```xml
<v:textbox style='mso-direction-alt:auto'>
<div style='text-align:right;direction:rtl'></div>
</v:textbox>
```
Fix#3999. Fix#4000. Fix#4002. Several bug reports and feature requests for Xml Reader arrived practically simultaneously. They are all small and hit the same code modules, so I have bundled them together in one PR.
- `loadSpreadsheetFromString` might try to open a file with a falsy name (like '0'), which results in an exception with a misleading message (or a completely unexpected result if a file with that name exists). Code will still throw an exception, but the message will no longer be misleading, and no file I/O will be attempted.
- function `trySimpleXmlLoadString` is deprecated. It should never have been implemented with public visibility, and the fact that it was made the fix above a little more difficult than it would otherwise have been. It is replaced with a private equivalent.
- Style reader function `parseStyles` will now use a better namespace-aware method of reading its Xml data. Peculiarly, the Xml for the Style elements can either include or not a namespace prefix. This is probably because the global namespace and the styles namespace are the same. The existing prefix-based code does not recognize their equivalence, but the new namespace-based code does. Xml Reader continues to use prefix-based code in several other places.
- Border line styles with Weight omitted or equal to 0 have been treated as no border, but they should be treated as 'hair' thickness.
- Support for Zoom is added to Xml Reader.
- In support of the above, new properties (and getters and setters) zoomScalePageLayoutView and zoomScaleSheetLayoutView are added to Worksheet/SheetView. (As far as I can tell, Excel does not support Sheet Layout View for Xml spreadsheets).
- Support is added for those new properties in Xlsx Reader and Writer.
- Xls Reader and Writer seem to work okay without changes. There is one test where Xls shows a different value for one of the properties than Xml or Xlsx, but the spreadsheet looks okay and I don't see any practical consequences of the difference.
- PageBreak support is added to Xml Reader.
- Code for writing out Column Page Breaks in Xlsx Writer was wrong (and, unsurprisingly, untested). A one-line change fixes it, and tests are added.
Fix#3988. Excel allows a table to have some colums with filter buttons exposed and some with filter buttons hidden. However you cannot do this in "native" Excel - VBA is required for this feature. PhpSpreadsheet is supposed to be able to handle this, but Xlsx/Writer/Table had a bug. The `filterColumn` tags in the xml should be children of the `autoFilter` tag, but were generated as siblings. Moving one statement fixes that problem. Fixing that exposed another problem - autoFilter `showHideRows` was not properly recognizing that a filter could exist without any rules, which is what happens when a table filter button is hidden. Another simple change fixes that problem.
The parent issue correctly points out the problem for Column D in samples/Table/01_Table. However, that sample also has a problem with Column A - unlike columns B and C, there is a filter appled to column A, so its dropdown button should appear different than those in B and C. That problem is also corrected with the fixes above. I also added some formal tests based on that sample.
oleibman
Claudia Schmidt