@dkulyk noticed a case which wasn't handled properly. I believe the fix works for everything ... except `COLUMNS` and `ROWS`, where Excel seems to ignore the sheet, and I just can't figure out how to do it.
Fix#5042. `Shared\File::realpath` is used to normalize file name references in a zip file. It can be used for non-zip file names as well, but zip members is its primary use, and `file_exists` does not work with zip file members. Such usage leads to the issue to which this PR is linked, and there have been other instances in past. In particular, issue #3553 was addressed by editing it out a leading slash in such references; it is not needed, and neither Excel nor PhpSpreadsheet generate one, but it's legal. The fix for 3553 addressed only loading a spreadsheet; but `listWorksheetInfo` and `listWorksheetNames` are subject to the same problem, which is this issue.
The normalization that `realpath` does is already built into `File::realpath` using a well-documented algorithm. So neither `filepath` nor `file_exists` is needed there, and they are removed. There are other problems with the current implementation. It looks only for Unix, not Windows, file separators. And a very peculiar problem arises when the zip entry name matches a file path in the current working directory. These problems all go away with this PR.
It is very likely that some manipulation of the zip entry names can be removed entirely from `Reader\Xlsx`. I may look into this in future, but I'm not going looking for trouble now. The fix for 3553 remains in place, even though no changes are made to match it in `listWorksheet...`, so, chances are the 3553 change could be dropped. As I say, maybe some day.
The Ods Writer skipped every Worksheet\Table. LibreOffice keeps a table
as a named table:database-range, and from its header row it tags the
header cells of an exported PDF as /TH /Scope /Column (LO master).
The writer now writes one per table, with contains-header and the filter
buttons, after the named expressions as the schema orders them; a table
with no name gets one no other table has. The reader reads a named
database range back as a table and, as LibreOffice does, drops one whose
name is taken or cannot be a table. LibreOffice's anonymous ranges stay
a sheet filter, which now lands on the sheet its address names instead
of the last sheet; a range of a sheet that was not loaded is skipped.
LibreOffice writes the link of a cell that is not plain text (a number,
a boolean, a date, a formula) as style:hyperlink and loext:hyperlink in
the style:table-cell-properties of the cell's automatic style, which is
how it saves a Xlsx file with such links as Ods. The reader looked only
for text:a, so those links were lost.
The reader now maps each cell style to its link and uses it when the
cell has no text:a. A cell with no style of its own takes the default
cell style of its column, as LibreOffice does for A1 in the test file.
The Ods Writer wrote no direction for a sheet set right to left, so it
opened left to right in LibreOffice. The Ods Reader did not read the
direction either, so a right-to-left sheet saved by LibreOffice came back
left to right.
The Writer now puts style:writing-mode="rl-tb" in the table properties
of such a sheet, as LibreOffice does, and the Reader reads rl-tb or rl
from the table style next to its visibility.
The Ods Writer turned sheet://Sheet2!A1 into #Sheet2!A1, keeping the
Excel separator, while ODF and LibreOffice write #Sheet2.A1. The Ods
Reader only swapped the prefix, so a link written by LibreOffice became
sheet://Sheet2.A1, and a later save to Xlsx wrote location="Sheet2.A1",
which does not resolve.
The Writer now writes a dot after the sheet name, quoted or not. The
Reader turns that dot back into "!" when a cell address follows it, and
still accepts "!", which older releases wrote and which LibreOffice keeps
when the address after it also parses as R1C1 (#'My Sheet'!C3). A named
range, with no sheet, stays as it is.
Six years ago, my PR #1557 was merged. It was written mainly to improve code coverage in Xml Reader, but some changes and enhancements were made to the Reader code. I noted in that ticket: "File templates/Excel2003XMLTest.xml, used in some tests, is not
readable by a current version of Excel. I have substituted a new file
excel2003.xml to be used in its place. I have not deleted the original
in case someone in future (possibly me) wants to see what it needs to
make it usable." That time has finally arrived.
Excel could not open the file. LibreOffice could, and did a pretty decent job, but something seemed wrong with how it handled formulas. PhpSpreadsheet could not read it, because a field identified as a date/timestamp was wrong - it specified a time-of-day hour as `288`. I made a copy of the file, changed the invalid hour to something valid. It turns out that was not Excel's problem - it still could not read the file. But, for the first time that I can recall, Excel produced decent diagnostic messages! They were in a location that you can't get to through Windows Explorer, even with "hide system files" turned off. However, you could get to it through Windows Command Prompt. There were 56 problems, all involving formulas (which was probably LibreOffice's problem). The formulas were all of the form:
```
of:=[.B1]+[.C1]
```
Now, XML spreadsheets use RC format rather than A1 format for its cells. The formula above is of a style that LibreOffice uses, but Excel does not understand it. It is looking for something like:
```
ss:Formula="=RC[-6]+RC[-5]
```
Excel2003Xml doesn't understand the formula in the file, but PhpSpreadsheet does. But, before it could do so, we had to eliminate the Exception when we tried to parse the invalid date/time. So, the first order of business was to wrap the Xml Reader date conversion logic in a try/catch. This allowed PhpSpreadsheet to read the file, and then to save it as an Xlsx file. Then we could open the Xlsx file in Excel, and save it as Xml. Voilà. We have converted the unreadable Xml file to a readable one. LibreOffice can also read the new file, and the formulas are now correct. BTW, I have no idea what was intended on sheet "Report Data" cells G17:G29. They all subtract a value from an empty cell, so the result is always negative, and try to format it as a date (which Excel doesn't like for negative numbers). Whatever that problem is, there is no need to fix it at this time.
At this point, a little cleanup was still needed. The Xml file was using some strange named styles, e.g. `Medium Date`. I was able to get a list of these with the google query `excel 2003 ooxml named number formats`. However, the list came only in the AI portion of the response; I could not find a link to anything permanent. So I went with what it showed me.
The earlier PR suggested that handling UTF-16 for this format would be difficult. However, saving the now-usable UTF-8 Xml file as UTF-16 with BOM results in a file which PhpSpreadsheet (and Excel and LibreOffice) can read without any code changes.
`simplexml_load...` can return false, and callers check for it. `dom->loadXML` can do likewise but, till now, it hasn't been checked. Only user of `dom->loadXML` is Ods Reader. Change it to throw if Xml is invalid.
GAMMA.INV silently returned the first bisection midpoint for alpha in
~[143, 171.62] (e.g. GAMMA.INV(0.5, 143, 1) gave 358.0 instead of
142.667) and #NUM! above that, because the Newton-step pdf evaluates
Gamma(a), b**a and value**(a-1) in linear domain, all of which overflow
even though the density itself is a small representable number. The same
pattern breaks the GAMMA.DIST, CHISQ.DIST and F.DIST densities and
GAMMALN, which computed log(Gamma(x)) through Gamma(x).
Evaluate these in log domain via the existing logGamma, and scale the
incomplete-gamma series/continued-fraction iteration cap as O(sqrt(a)),
which both expansions need to converge near x ~ a once the shape is
large (GAMMA.INV drifted from the true quantile above alpha ~5000 and
returned alpha+1 by alpha=10000; same for CHISQ.INV at high df).
calculateInverse() treated a CDF value of exactly 0.0 as "the guess is the
root" and collapsed the bracket with $b = $a, ending the search. inverse()
already rejects probability <= 0, so a CDF of 0.0 inside the bisection can
only be a float64 underflow at a guess far below the root - never an exact
hit. It is just an ordinary "guess too low" and belongs in the existing
else branch.
The underflow is reached whenever a probe lands many standard deviations
from the mean, so the search collapses after two or three iterations and
returns whichever midpoint it was holding. BETAINV(0.5, 5000, 5000) gave
0.25, though Beta(a, a) is symmetric and its median is exactly 0.5;
BETAINV(0.5, 20000, 3) gave 0.5 against a true 0.99986. Onset is around
alpha = 1080 for beta = 1, where the closed form 0.5 ** (1 / alpha) is
available to check against.
That test also happened to stop the search for shapes where incompleteBeta
declines to evaluate at all and returns 0 for every x, so inverse() now
rejects alpha + beta above that documented limit up front: the CDF is
identically zero there, so no quantile exists to search for.
Tests cover a shape x probability grid against scipy reference values plus
three checks that need no external oracle: the symmetric median, the
Beta(alpha, 1) closed form, and the BETADIST round trip / mirror identity.
The incomplete gamma primitive used a fixed 32-term power series with no
convergence test, so GAMMA.DIST, GAMMADIST, CHISQ.DIST(.RT), GAMMAINV and
CHISQ.INV were grossly wrong once the series argument reached ~32
(e.g. CHISQ.DIST.RT(80, 4) returned 0.806 instead of 1.74e-16).
Replace it with the standard convergence-tested regularized incomplete
gamma: series P(a,x) for x < a+1, continued fraction Q(a,x) for x >= a+1.
CHISQ.DIST.RT now uses Q directly so the right tail stays free of
1 - P cancellation. Consolidates the duplicate copy that already existed
privately in ChiSquared onto the shared primitive.
Introduces a first-class, read-only representation of pivot tables that
already exist in a loaded Xlsx file (issue #4534). Previously pivot tables
were completely inaccessible from the object model.
New model classes under Worksheet\PivotTable:
- PivotTable: name, location, source cache definition and field layout,
with axis helpers (getRowFields/getColumnFields/getPageFields/getDataFields).
- PivotCacheDefinition: cache id, source worksheet/range and cache field names.
- PivotField: field index, name, axis placement, and data-field aggregation.
Reader\Xlsx\PivotTableReader parses a pivotTableDefinition part and its
associated pivotCacheDefinition part into that model. The main Xlsx reader
discovers pivot parts via worksheet relationships and wires them up (skipped
under setReadDataOnly). Worksheet gains getPivotTableCollection(),
getPivotTables(), addPivotTable(), getPivotTableByName(), getPivotTableNames()
and removePivotTableCollection().
This is inspection-only: it does not create, modify, recalculate or render
pivot tables, and does not change what the writer emits. A minimal pivot-table
fixture and functional tests cover reading name, location, cache source,
fields and axis placement, plus the read-data-only path.
Add tests for previously-uncovered SparklineGroup setters (axis types,
manual limits, additional display options and colours) and a hand-crafted
Xlsx fixture exercising the reader's edge cases (non-sparkline ext, group
without sparklines, sparkline with empty sqref, missing colour elements).
Harden Reader/Xlsx/Sparklines against malformed input: guard the
sparklineGroup and sparkline iterations against a null children set so an
empty <x14:sparklineGroups> or <x14:sparklines> element no longer emits a
PHP warning.
Building on PR #3508 and PR #4393, which allowed ignoring 5 different cell 'errors'. Adding this new one should have been dead easy, but MS, in its own inscrutable way, decided to complicate this by making the xml attribute require its very own namespace, unlike any of its relatives.
There are supposed to be several ways to activate this tag. One is to enter a date in cell A1, then in cell B1 enter the formula `=A1`, and set B1's numeric format to `General`.
A follow-up to PR #4837. PR #4827, to which I am not necessarily committed, shows that there could be a use case for letting a user perform customized logic to convert a non-UTF8-encoded CSV. In addition, several other CSV Reader properties require the use of static or locale properties, which is somewhat problematic; this PR allows them to be set as instance variables, falling back to static/locale only when unset.
Fix#4823. When writing a spreadsheet to Html, a data Uri can be used if `embedImages` is true. Reading such an Html spreadsheet and attempting to write it to Xlsx results in an Exception. It should be noted that Excel itself cannot open the Html properly; none of the images are present.
The PhpSpreadsheet problem arises not with the inclusion of the image, but rather with attempting to include the appropriate entry in `[ContentTypes].xml`. This PR corrects that problem.
For the record, Xls Writer does not have a problem with this situation. Just to demonstrate that, a parallel test for Xls Writer is added in addition to the new Xlsx Writer test.
oleibman
Dmytro Kulyk
Jonathan Goode
Guy Tomer
youzixu
^^
Vincent Gao
saifulislamferoz