Six years ago, my PR #1557 was merged. It was written mainly to improve code coverage in Xml Reader, but some changes and enhancements were made to the Reader code. I noted in that ticket: "File templates/Excel2003XMLTest.xml, used in some tests, is not
readable by a current version of Excel. I have substituted a new file
excel2003.xml to be used in its place. I have not deleted the original
in case someone in future (possibly me) wants to see what it needs to
make it usable." That time has finally arrived.
Excel could not open the file. LibreOffice could, and did a pretty decent job, but something seemed wrong with how it handled formulas. PhpSpreadsheet could not read it, because a field identified as a date/timestamp was wrong - it specified a time-of-day hour as `288`. I made a copy of the file, changed the invalid hour to something valid. It turns out that was not Excel's problem - it still could not read the file. But, for the first time that I can recall, Excel produced decent diagnostic messages! They were in a location that you can't get to through Windows Explorer, even with "hide system files" turned off. However, you could get to it through Windows Command Prompt. There were 56 problems, all involving formulas (which was probably LibreOffice's problem). The formulas were all of the form:
```
of:=[.B1]+[.C1]
```
Now, XML spreadsheets use RC format rather than A1 format for its cells. The formula above is of a style that LibreOffice uses, but Excel does not understand it. It is looking for something like:
```
ss:Formula="=RC[-6]+RC[-5]
```
Excel2003Xml doesn't understand the formula in the file, but PhpSpreadsheet does. But, before it could do so, we had to eliminate the Exception when we tried to parse the invalid date/time. So, the first order of business was to wrap the Xml Reader date conversion logic in a try/catch. This allowed PhpSpreadsheet to read the file, and then to save it as an Xlsx file. Then we could open the Xlsx file in Excel, and save it as Xml. Voilà. We have converted the unreadable Xml file to a readable one. LibreOffice can also read the new file, and the formulas are now correct. BTW, I have no idea what was intended on sheet "Report Data" cells G17:G29. They all subtract a value from an empty cell, so the result is always negative, and try to format it as a date (which Excel doesn't like for negative numbers). Whatever that problem is, there is no need to fix it at this time.
At this point, a little cleanup was still needed. The Xml file was using some strange named styles, e.g. `Medium Date`. I was able to get a list of these with the google query `excel 2003 ooxml named number formats`. However, the list came only in the AI portion of the response; I could not find a link to anything permanent. So I went with what it showed me.
The earlier PR suggested that handling UTF-16 for this format would be difficult. However, saving the now-usable UTF-8 Xml file as UTF-16 with BOM results in a file which PhpSpreadsheet (and Excel and LibreOffice) can read without any code changes.
`simplexml_load...` can return false, and callers check for it. `dom->loadXML` can do likewise but, till now, it hasn't been checked. Only user of `dom->loadXML` is Ods Reader. Change it to throw if Xml is invalid.
GAMMA.INV silently returned the first bisection midpoint for alpha in
~[143, 171.62] (e.g. GAMMA.INV(0.5, 143, 1) gave 358.0 instead of
142.667) and #NUM! above that, because the Newton-step pdf evaluates
Gamma(a), b**a and value**(a-1) in linear domain, all of which overflow
even though the density itself is a small representable number. The same
pattern breaks the GAMMA.DIST, CHISQ.DIST and F.DIST densities and
GAMMALN, which computed log(Gamma(x)) through Gamma(x).
Evaluate these in log domain via the existing logGamma, and scale the
incomplete-gamma series/continued-fraction iteration cap as O(sqrt(a)),
which both expansions need to converge near x ~ a once the shape is
large (GAMMA.INV drifted from the true quantile above alpha ~5000 and
returned alpha+1 by alpha=10000; same for CHISQ.INV at high df).
calculateInverse() treated a CDF value of exactly 0.0 as "the guess is the
root" and collapsed the bracket with $b = $a, ending the search. inverse()
already rejects probability <= 0, so a CDF of 0.0 inside the bisection can
only be a float64 underflow at a guess far below the root - never an exact
hit. It is just an ordinary "guess too low" and belongs in the existing
else branch.
The underflow is reached whenever a probe lands many standard deviations
from the mean, so the search collapses after two or three iterations and
returns whichever midpoint it was holding. BETAINV(0.5, 5000, 5000) gave
0.25, though Beta(a, a) is symmetric and its median is exactly 0.5;
BETAINV(0.5, 20000, 3) gave 0.5 against a true 0.99986. Onset is around
alpha = 1080 for beta = 1, where the closed form 0.5 ** (1 / alpha) is
available to check against.
That test also happened to stop the search for shapes where incompleteBeta
declines to evaluate at all and returns 0 for every x, so inverse() now
rejects alpha + beta above that documented limit up front: the CDF is
identically zero there, so no quantile exists to search for.
Tests cover a shape x probability grid against scipy reference values plus
three checks that need no external oracle: the symmetric median, the
Beta(alpha, 1) closed form, and the BETADIST round trip / mirror identity.
The incomplete gamma primitive used a fixed 32-term power series with no
convergence test, so GAMMA.DIST, GAMMADIST, CHISQ.DIST(.RT), GAMMAINV and
CHISQ.INV were grossly wrong once the series argument reached ~32
(e.g. CHISQ.DIST.RT(80, 4) returned 0.806 instead of 1.74e-16).
Replace it with the standard convergence-tested regularized incomplete
gamma: series P(a,x) for x < a+1, continued fraction Q(a,x) for x >= a+1.
CHISQ.DIST.RT now uses Q directly so the right tail stays free of
1 - P cancellation. Consolidates the duplicate copy that already existed
privately in ChiSquared onto the shared primitive.
Introduces a first-class, read-only representation of pivot tables that
already exist in a loaded Xlsx file (issue #4534). Previously pivot tables
were completely inaccessible from the object model.
New model classes under Worksheet\PivotTable:
- PivotTable: name, location, source cache definition and field layout,
with axis helpers (getRowFields/getColumnFields/getPageFields/getDataFields).
- PivotCacheDefinition: cache id, source worksheet/range and cache field names.
- PivotField: field index, name, axis placement, and data-field aggregation.
Reader\Xlsx\PivotTableReader parses a pivotTableDefinition part and its
associated pivotCacheDefinition part into that model. The main Xlsx reader
discovers pivot parts via worksheet relationships and wires them up (skipped
under setReadDataOnly). Worksheet gains getPivotTableCollection(),
getPivotTables(), addPivotTable(), getPivotTableByName(), getPivotTableNames()
and removePivotTableCollection().
This is inspection-only: it does not create, modify, recalculate or render
pivot tables, and does not change what the writer emits. A minimal pivot-table
fixture and functional tests cover reading name, location, cache source,
fields and axis placement, plus the read-data-only path.
Add tests for previously-uncovered SparklineGroup setters (axis types,
manual limits, additional display options and colours) and a hand-crafted
Xlsx fixture exercising the reader's edge cases (non-sparkline ext, group
without sparklines, sparkline with empty sqref, missing colour elements).
Harden Reader/Xlsx/Sparklines against malformed input: guard the
sparklineGroup and sparkline iterations against a null children set so an
empty <x14:sparklineGroups> or <x14:sparklines> element no longer emits a
PHP warning.
Building on PR #3508 and PR #4393, which allowed ignoring 5 different cell 'errors'. Adding this new one should have been dead easy, but MS, in its own inscrutable way, decided to complicate this by making the xml attribute require its very own namespace, unlike any of its relatives.
There are supposed to be several ways to activate this tag. One is to enter a date in cell A1, then in cell B1 enter the formula `=A1`, and set B1's numeric format to `General`.
A follow-up to PR #4837. PR #4827, to which I am not necessarily committed, shows that there could be a use case for letting a user perform customized logic to convert a non-UTF8-encoded CSV. In addition, several other CSV Reader properties require the use of static or locale properties, which is somewhat problematic; this PR allows them to be set as instance variables, falling back to static/locale only when unset.
Fix#4823. When writing a spreadsheet to Html, a data Uri can be used if `embedImages` is true. Reading such an Html spreadsheet and attempting to write it to Xlsx results in an Exception. It should be noted that Excel itself cannot open the Html properly; none of the images are present.
The PhpSpreadsheet problem arises not with the inclusion of the image, but rather with attempting to include the appropriate entry in `[ContentTypes].xml`. This PR corrects that problem.
For the record, Xls Writer does not have a problem with this situation. Just to demonstrate that, a parallel test for Xls Writer is added in addition to the new Xlsx Writer test.
We have identical constants defined in several places, and use literals in others. We aren't consistent in checking limits. This PR makes the use of the constants in Cell/AddressRange the "official" source, deprecates all other constants, and substitutes the constants wherever literals are used. A number of different edge case tests are added.
During testing, I discovered that `columnIndexFromString` correctly throws an exception for 4-character string, but allows `XFE` through `ZZZ`, all of which are also invalid. There are similar inconsistencies with related routines, and this PR attempts to make them operate consistently. One suprise is that throwing for `row=0` causes serious regression problems, so it continues to be permitted (but the high row limit is enforced).
Further, Reference Helper sometimes dips into negative numbers, resulting in totally unexpected results (-1 affects column Z, -2 column Y, etc.). It is changed to ignore rows and columns outside the limits.
Continuing the work of PR #4810 and PR #4813 and PR #4814. According to the spec for LibreOffice, the xml for both coluns and rows allow a `default-cell-style-name` attribute. However, my testing indicates that using that attribute with a row doesn't result in the results I would expect; furthermore, LibreOffice doesn't seem to use that attribute in association with a row (it does use it for columns). For rows, it fills the unpopulated rows in the cell with a repeated table cell with the appropriate style. These filler cells have been a source of problems for us, and I'm not willing to add them at this time (maybe in future). So, if a style is applied to a row in PhpSpreadsheet, it will be reflected in all cells in that row until `highestDataColumn`, but no further.
Since LibreOffice does support adding a style to a column both in the UI and the Xml, it should be a little easier. But its support is imcomplete as well - when reading a spreadsheet, the UI will apply the style only to the rows that are defined in the xml. Again, this requires filler rows for the rows beyond `highestDataRow`, and, again, I am not willing to add those. However, it is important to process the column style when reading because, unlike the case with rows, the cells which are defined in the column may omit style data altogether if the style matches the column style.
There has till now been no support for `ReadDataOnly` (default false) and `ReadEmptyCells` (default true) for Ods Reader. These are now added. ReadEmptyCells is especially important and it would be sensible to usually set it to false. The reason is that LibreOffice usually supplies a width for all the columns mentioned in its row fillers, and PhpSpreadsheet needs to create ColumnDimensions for each of those as a result, resulting in much larger xml than one would expect when the spreadsheet is saved. This arose when I was testing this change - it has always been the case, but nobody has raised an issue about it. We avoid this problem by ignoring the extra columns for ReadEmptyCells false. But this requires a little redundancy in the code. We need to define styles for an entire column before we define styles for the cells in the column, but we don't need to define column widths beforehand. So, if ReadEmptyCells is false, we need to process the columns for styles, then process the cell data (so that we know what `highDataColumn` is), then process the columns again but this time for widths (ignoring columns beyond highDataColumn). If ReadEmptyCells is true, we can process column styles and widths in a single pass.
Continuing the work of PR #4810.
Ods Reader changes still to come
- Default border, alignment, protection
- Font
- subscript
- superscript
- Borders (up next)
- Style applied to entire row
- Style applied to entire column
- Parent styles?
Ods Writer changes still to come
- Default border, alignment, protections
- Style applied to entire row
- Style applied to entire column
- Parent styles?
Fix#2622. Fix#1191 (stale as of 2020 but just reopened). A long overdue addition. Ods Reader has till now not supported any styles (although PR #4806 added some support for number formats). This will be the first of several PRs to rectify this situation.
Ods Reader Changes
- Default font
- Font
- autoColor
- bold
- color
- italic
- font-family (name)
- size
- underline
- Fill (type=Solid only)
- startColor
Ods Writer Changes
- Default font
- Font
- strikethrough (fix#4808)
- size currently always writes to xml with one decimal point, will now eliminate `.0`
Ods Reader changes still to come
- Default fill, border, alignment, protection
- Font
- subscript
- superscript
- Fill
- types other than Solid
- Alignment
- Borders
- Protection
Ods Writer changes still to come
- Default fill, border, alignment, protections
Redo of PR #4799 due to failed attempt to resolve merge conflict.
Fix#4798. Partially address #3961. Ods Reader supports very little related to styling. Ods Writer supports many styling details, but has not heretofore supported Number Formatting. 3961 addresses both issues; I created 4798 specifically for the Writer side.
Writing number formats to Excel is pretty simple - you just supply a string and that is used directly in the Xml. Ods is much more complicated - it requires Xml nodes that give a complete description of the styling. For that reason, it is difficult and painstaking to convert from the string that Excel (and PhpSpreadsheet) uses to what Ods requires.
This PR provides code to support almost all the styles defined as constants in Style/NumberFormat. It also allows the user to add code to handle otherwise unhandled styles. New Sample55_DefinedStyles demonstrates the use of all the constant styles, plus the addition of a couple of custom styles. I may be amenable to adding some unsupported styles to the built-in list, but the custom style option will always be around in case I am being slow or unreasonable.
This PR does not fully support Ods Reader handling of styles. However, based on the new Writer output, it will often be able to guess the true type of numeric items and assign an appropriate style for that type. So, for example, if it can identify the field as a date, it will assign a date style. It will not always match the style in the sheet being read, but it is a big advance from just formatting the data as a generic number.
When using a read filter with the ODS reader, cells with
table:number-columns-repeated containing data were incorrectly handled.
The read filter was only checked for the first column of a repeated
group - if it didn't pass, the entire group was skipped, even if
other columns within the range should have passed the filter.
This caused column data misalignment where filtered columns would
either be missing or contain data from wrong positions.
The fix ensures that:
1. When the first column of a repeated group fails the filter, the
reader checks if ANY column in the group passes before skipping
2. During data writing, each repeated column is individually checked
against the read filter
FixesPHPOffice/PhpSpreadsheet#4802
Fix#4061. CONCATENATE, which has slightly different behavior than CONCAT, and which MS has deprecated for that reason, behaves in an unexpected way when a cell range is presented to it and the spreadsheet does not allow for array results. This would almost certainly occur only for Legacy spreadsheets, but such is what was presented in the issue. The code is changed so that when an array of cells is presented to CONCATENATE, and RETURN_ARRAY_AS_VALUE is in effect, the array will be treated as if it were wrapped in the SINGLE pseudo-function (which is what Excel does by somewhat mysteriously prefixing the cell range with `@`).
This is a niche case. This one stands out because of its deprecation and replacement function. It is possible that other functions exhibit this behavior. I have made no attempt to identify others. A similar approach can probably be applied if issues are raised for others.
Loading external images from a spreadsheet can subject the caller to security exploits. For those who think they need something between not loading any external images (the default and our strong recommendation) or loading them all unconditionally (the current alternative), a callback can now be provided to indicate whether a specific image is okay to load. We continue to urge caution in using this option.
Fix#4776. DefaultValueBinder treats strings that look like scientific notation as floats. However, if Php evaluates them as `INF`, this can cause problems for Excel. A one-line change to DefaultValueBinder avoids this situation. Tests are added for DefaultValueBinder, and also for StringValueBinder and AdvancedValueBinder, although those extend DefaultValueBinder and needed no code changes.
Fix#1319, which went stale in 2020, and is now reopened. Rowspan is not showing up when saving some spreadsheets as Html. There is some logic concerning property `isSpannedRow` which indicates that this is deliberate, but I have no idea what purpose it serves. It leads directly to this issue, and also adversely affects (in a minor way) some samples (e.g. 17a_Html), where merged cells maintain their text but lose their top and bottom padding. After discarding this logic, the reported problem goes away, the samples look better, and no tests fail. So, whatever its purpose, it wasn't tested for. This PR permanently removes the property, and adds a little additional bulletproofing. A minor accommodation is needed for Tcpdf in one place.
This PR probably deletes more lines from Html Writer than it adds. It would not surprise me if this is one of those unusual cases where Coveralls reports a decrease in coverage despite the fact that all the new code is covered.
oleibman
^^
Vincent Gao
saifulislamferoz
QDenka
Indrek Haav
Thomas Lallement
Thomas Lallement