Fix#4249. Technically speaking, only the 1.29 branch needs fixing, and only for TEXT. It was fixed for the other branches by PR #3898. However, in adding test cases for the fix, it became apparent that PhpSpreadsheet's parsing in TIMEVALUE (which is called from TEXT in the original issue) did not really match Excel's. There are probably still edge cases where it doesn't, but, in the absence of a spec for how it operates, this will do for now.
We do not usually backport fixes from the master branch. Because this is more of a forward port from the earlier branch, there is an equivalent PR for each active branch.
These are 5 closely related functions for manipulating arrays. Now that dynamic arrays are part of PhpSpreadsheet, this PR implements those previously-unimplemented functions. Documentation undergoes a very minor change, since they are re-categorized as "Lookup and Reference", which is how Excel classifies them, rather than "Math and Trig".
This is a partial response to issue #4282. The actual logic to implement GROUPBY is probably very complicated. And, even worse, Excel has thrown a whole new way of (internally) specifying one of the arguments into the mix. That argument is a function name, expressed not as a mapped integer (as SUBTOTAL does), nor even as a string, but as the unquoted function name prefixed by `_xleta.`. And, unlike its `_xlfn.` and `_xlws.` predecessors, it is difficult to figure out when the new prefix needs to be added, and when it needs to be ignored. I am not even going to attempt that task with this ticket.
So, what does this change do? Like earlier attempts to introduce limited functionality (such as with form controls), it is there so that using GROUPBY can be passed through - you can load a spreadsheet that contains it, and save it to a new spreadsheet, and the function and its results are preserved. Some cautionary notes. Dynamic arrays must be enabled (the function makes no sense without doing that). Changing any of the inputs used in the function may result in internal inconsistencies between PhpSpreadsheet and Excel; this is especially so if the dimensions of the returned array change as a result of changes to the input data. The programmer can avoid some of these problems by changing the formulatAttributes of the cell where the function is used; this may be difficult to do in practice. Oh, yes, using the GROUPBY cell as an argument in another formula will probably lead to problems. Finally, I confess that part of this solution looks awfully kludgey to me.
With its limitations and those cautions, is it worth proceeding with this change? My gut feel is that it is more useful to proceed than not. However, I will give others the opportunity to weigh in. I will wait at least a couple of weeks into the new year before proceeding with this.
A solution, at least in part, for issue #4280. Xlsx Reader is not handling shared formulae correctly. As a result, some cells are treated as if they contain boolean values rather than formulae.
After breaking up Xls Reader (PR #4118), it is a little easier to identify uncovered code. BIFF8 had no tests involving constant arrays. This PR adds some. Most of the work is in the tests, but some source code is modernized to use things like null coercion.
PR #2850 and PR #2851 added support for Worksheet Visibility to ODS. They work well when LibreOffice is used to open the spreadsheet. However, when Excel tries to open it, it reports corruption. Strictly speaking, this is not a PhpSpreadsheet problem, but, if we can fix it, we should.
It took a while to figure out what's bothering Excel. I'm not sure that all of what follows is necessary, but it works. It appears that it wants the content.xml `style:automatic-styles` definition for the `table` (i.e. worksheet) to include a `style:master-page-name` attribute. That attribute requires a corresponding definition in the `office:master-styles` section in styles.xml, and that attribute likewise requires a definition in `office:automatic-styles`.
The new entries in styles.xml can be used to specify things like header and footer. However, the ways that these are specified is distinctly different from what Excel (and therefore PhpSpreadsheet) does. Implementing that will be a good future project. However, for now, they will remain unsupported for Ods.
Ods Writer currently will write something like `<table:table-cell table:number-columns-repeated="1023" />` at the end of each row. This is not necessary. In eliminating that, I have made the code a bit more efficient and (hopefully) more readable.
Fix#4261. Ods does nothing with the `indent` property of Alignment. The fix is easy, but not as easy as it should be. Excel treats indent as an int in some unspecified unit. Ods, on the other hand, specifies it as a float with unit (inches, ems, etc.). Since MS does not reveal what an indent unit is, I resorted to some experimentation. It appears that 1 unit is equal to 0.1043 inches. That is not a guaranteed relationship - the conversion might be non-linear, or it might be affected by external factors like font size. I think multiplying by 0.1043 is adequate for now, and unquestionably better than what we're currently doing (ignoring the property). If it breaks down at some point, we'll look at it again. BTW, Shared\Drawing uses some conversion values of 9525, which, reciprocating and sliding the decimal point along yields a value of 0.10498..., which is close but not close enough for me to use.
The other property used in conjunction with indent is `text-align`, and here we have not been doing the right thing. If that property has the default value (`General`), it is treated as `start` (i.e. `left` on LTR docs and presumably `right` on RTL). This will align numbers as if they were text, which does no harm (they're still usable as numbers), but is not how LibreOffice handles them by default. Ods writer is changed to omit `text-align` when it is set to `General`, and let LibreOffice choose the appropriate alignment.
These changes are for Writer only. Ods Reader support for styles remains severely lacking.
Fix#4248. PhpSpreadsheet has used what appear to be default attributes and tags when they are missing from Fill patterns and colors. However, Excel handles their absence a little differently from what the "default" would require. PhpSpreadsheet is changed to omit the attributes and tags in question when missing. This change is mostly targeted towards Xlsx read and write, but minor changes for Xls and Html write are also included.
This seems like it could be a breaking change, but I don't think it is. One test (DefaultFillTest introduced by PR #2050) must change, but the change is internal - loading and then saving the spreadsheet used in that change will appear the same after this change as it did before. Other differences are very likely to be bug fixes rather than breaks.
This commit fix two issues that happened when the a cell was formatted as text
* When the cell contains a number prefixed with dollar sign, this number is getting replaced with 0. The replacement happens due to the preg_replace function.
* When the cell contains quotes, the quote would be removed.
Having addressed several security advisories, one evident *theoretical* problem remains. This is an attempt to future-proof our code against similar vulnerabilities. It all begins with our implementation of libXmlLoaderOptions, which uses as a default LIBXML_DTDLOAD. This unfortunate choice opens us to XXE problems, many recently solved. I do not believe that there is a legitimate use case for allowing this, and will therefore ignore and deprecate that option.
Although this might seem to be a breaking change, it is not. The setting is used only after the Xml has been subject to a security scan, and the security scan throws an exception if it detects the use of `<!DOCTYPE` within the Xml. Therefore, the setting will be effective only on Xml which does not contain that tag, and will consequently have no effect on most Xml. The only exception would be Xml which has been crafted to avoid detection by the security scanner in a manner which has not been disclosed to us. Although we hope that we've now blocked all such avenues, this provides additional protection just in case.
With this change in place, we could relax certain restrictions, e.g. the use of EBCDIC or even UTF-7. For now, these will remain in place. I will need to be convinced that there is a legitimate use case for easing the restrictions before doing so. We might even consider the elimination of the Security Scanner altogether. However, it does allow for early detection, and, in any case, provides a method to correct Xml which most Xml readers would fail but which Excel accepts.
My plan is to merge this within the next few days, and tag a new release immediately after. It will also be backported to all active branches.
* Security Patch
* Throw Exception for EBCDIC Encoding
* Mixed UTF-8 and UTF-16
Further mischief. I don't know if the examples truly are valid Xml, but PhpSpreadsheet is letting them sneak through.
Fix#4213. Early versions of Php 8.4 caused problems for some Excel functions, among them ROUNDDOWN and ROUNDUP. New code was added which seemed to work for 8.4 and all prior releases (back to 7.4). However, as the issue shows, there is a problem with the new logic that had not been a problem for prior releases (PhpSpreadsheet 1.29.1 with Php 8.3 or earlier).
As it happens, some time after 8.4 broke the existing functionality, new features were added to it which enabled much cleaner logic. The code will now take this cleaner path when available, and revert to the older logic when not. This appears to work for all relevant Php releases, at least till the next non-match between PhpSpreadsheet and Excel is reported.
And, having made the change to ROUNDDOWN, it occurred to me that ROUNDDOWN always returns the same result as TRUNC (they differ only in number of required arguments). So I changed TRUNC to call ROUNDDOWN, and that didn't break anything. So I will keep that change. This probably means I should change Calculation so that ROUNDDOWN is called automatically, and deprecate TRUNC. That can happen at a later time.
See #4161. The best way to future-proof is to set the escape character to null string, and set testAutoDetect to false before reading. However, depending on the file being read, this may lead to different results than expected. This will be unavoidable because Php itself will change. This PR adds a new static method `affectedByPhp9` to Csv Reader. This can be used to identify in advance whether an input file will be affected by the changes. This will allow users to identify problems in advance, and prepare for how they might be handled.
Fix#2157. Excel 2003 format allows some things which a real Xml parser would reject. In particular, it permits Html entities, and leading whitespace. Change Xml Reader to do likewise.
* Validate Mime Type for Images
* Extend Change to Xlsx Reader
* Attach Drawing to Sheet Only If Valid
* Suppress Some Theoretical Warning Messages
* Minor Tweak
This supersedes PR #607 by @christian-forgacs, who deserves all the credit for reporting the problem and devising the solution. The PR went stale in 2018, and it is just easier to resubmit a clean version rather than clean up the old one. Among the suggestions in the PR was that you should try to create a spreadsheet from scratch to demonstrate the problem rather than supply one. However, my attempts to match the failing spreadsheet do not have a problem when they are read. So, a supplied spreadsheet it is.
Fix#1570. No sample spreadsheet was supplied with that issue, but I am almost certain that this is another example of the same problem. I am removing the stale label from that issue; it will be closed properly when this PR is merged.
Fix#4125. Currency and Accounting Wizards generate styles for ISO codes, but these are incorrect and cause a problem when Excel tries to open a spreadsheet containing these styles. Debugging that problem, other problems with Wizards came to light:
- Currency Wizard should permit four different styles for negative numbers (as Excel does) - minus sign, minus sign and red font, paretheses, and parenthese and red font. It currently uses only minus sign.
- Accounting Wizard should use parentheses for negative numbers (as Excel does). It currently uses minus sign.
- Accounting Wizard should always use SYMBOL_WITH_SPACING (as Excel does). It currently permits the use of SYMBOL_WITHOUT_SPACING. What WITH_SPACING really does is to ensure decimal-point alignment among adjacent cells in a column with the same format.
- Currency Wizard should always use SYMBOL_WITHOUT_SPACING (as Excel does). It currently permits the use of SYMBOL_WITH_SPACING.
I am correcting these problems by:
- renaming Currency Wizard to CurrencyBase
- adding a `negative` property with setter to it and its constructor.
- adding a new Currency which extends CurrencyBase, always using SYMBOL_WITHOUT_SPACING when formatting.
- having Accounting extend CurrencyBase rather than Currency, always using SYMBOL_WITH_SPACING and NEGATIVE_PARENS when formatting.
- CurrencyBase can be used if the restrictions on Currency and Accounting are not desired (e.g. the suggested accounting constant from [this unimplemented PR](https://github.com/PHPOffice/PhpSpreadsheet/pull/1576)).
Excel does some funny stuff with these formats. In particular, it might try to guess if you have a particular Accounting format in mind. So the Accounting wizard for dollar sign generates a format which (a) matches FORMAT_ACCOUNTING_USD, and (b) Excel (correctly) interprets as an Accounting format for symbol $. On the other hand, the Accounting wizard for euro sign generates a format which (a) matches FORMAT_ACCOUNTING_EUR, but (b) Excel interprets as a custom code rather than an Accounting format. This in itself is not a particularly big deal, but it has made it impossible for me to see exactly what format Excel uses for trailing currency symbols for negative numbers. I can't get them to decimal-point align with positive numbers if I put any kind of space between the trailing parenthesis and the currency symbol, so I omit that. It doesn't look terrible, and it keeps everything aligned, but it might not be what people are used to.
I've also changed the formatting to use spaces rather than non-breaking spaces. They seem to work just fine, and the constants mentioned above use them rather than nbsp.
Fix#4124. Currency formats that contain an ISO currency code which contains one of the characters used to recognize a date format (hmsdy), e.g. [$HUF], are being formatted by PhpSpreadsheet as dates rather than currencies. Code is changed to recognize open bracket followed by dollar sign followed by 3 Latin alphabetic characters followed by close bracket as a non-date.
Prevent XEE by hiding custom entities by using single quote to
declare a non-UTF-8 encoding.
XML standard, https://www.w3.org/TR/xml/#NT-EncodingDecl, allows single
quote to declare encoding, but we did not support it. Instead, we
incorrectly fell back on the default of UTF-8. That incorrectly kept the
XML as non-UTF-8, and thus prevented our regexp-based custom entity
detection mechanism to work.
Fix#4113. TRUNC isn't always producing the expected result. There was a promising algorithm at https://stackoverflow.com/questions/4668628/truncate-float-numbers-with-php from user Juan. It works through Php8.3, but failed in Php8.4 (more on this later). User Savageman on the same page has a solution that needs work, but, once the work had taken place, it works on Php8.1-8.4.
The ROUNDUP and ROUNDDOWN functions were adversely affected by Php8.4, probably for the same reasons as Juan's TRUNC suggestion. I put a kludge in place for them some time ago, but I wasn't happy with it. The solution used for TRUNC here suggested a change to the ROUNDUP and ROUNDDOWN code that would no longer require the kludge. The change to those functions now works more cleanly on Php8.1-8.4.
oleibman
Everton Barbosa
Adrien Crivelli