xmlstarlet-notes news

Link: https://martin7th.github.io/xmlstarlet-notes/

Version 2025-03-12

repology.org links

^(link) Added links to the versions pages at repology.org for xmlstarlet, libxml2, and libxslt.

Correction: --doc-namespace reads from first input file

^(link)(link) Clarified that by default (--doc-namespace being in effect) namespaces declared in the root element of the first input file need not be declared with -N options. select and edit both support multiple input files.

Visualize XML document structure

^(link) New entry: process output from xmlstarlet elements with awk and graphviz’ dot.

Omissions fixed (select section)

^(link)xsl:sort was missing in the intro.

^(link)select="«value»" was missing in the --var header.

Removed extract subtree example (“null-ns hack” in select section)

(It seemed a good idea at the time.)

XSLT function generate-id() fixed in libxslt 1.1.38

A long-standing issue with libxslt has been fixed in version 1.1.38: The result of generate-id() is now deterministic across multiple transformations fixing many issues with reproducible builds.

This means that xmlstarlet’s select and transform commands can do variable-based grouping without using xsl:key, axes, or extensions, and even group across multiple files in one pass. For examples see postings by G. Ken Holman on stackoverflow.com and xsl-list.

Attempt to load network entity «name»

^(link) Added error message. Triggered by missing --net option.

Entity ‘«name»’ not defined

^(link) Added error message. Only select, transform, and canonic seem to understand an entity reference.

Invalid expression: ‘«pathname»’

^(link) Added error message. select fails with a run-time error if input file pathnames on the command line contain single quotes (known bug).

Correction: func:function (Appendix A)

^(link) Deleted phrase: despite element-available("func:function") returning false.

element-available isn’t supposed to return true for func:function as this is a top-level element, not an instruction, per the XSLT 1.0 recommendation sections 14.1 and 15.

The cautionary note about the missing listing of {http://exslt.org/functions}function has been removed because libxslt since version 1.1.39 registers and lists it as a top-level extension element.

Version 2023-08-15

XML parsing and serialization

^(link) Added example: CDATA can protect against default serialization.

Numeric representation

^(link) New entry, with examples of libxml2 numbers in scientific notation.

-L (--inplace) - edit input file(s) in-place

^(link) Added:

• formats the input file(s) unless option -P (--pf) is given
• updates the timestamps of the input file(s) even if output is identical to input

Add if «condition»

^(link) New entry, demonstrating if-then-else logic with xmlstarlet edit.

--xinclude - do XInclude processing on document input

^(link) The entire entry on XInclude has been rewritten.

xmlSAX2Characters: huge text node: out of memory

^(link) Added error message.

Patch adds --huge and --big-lines options to xmlstarlet

Recently the author of the xmlstarlet-notes document has provided a patch to make xmlstarlet use ‘huge’ nodes and ‘big lines’. It requires the program to be rebuilt from source code.

Quoting from the README:

This patch adds 2 global options to xmlstarlet, available in the subcommands elements, select, edit, format, canonic, validate, and transform:

• --huge
  Load XML files with libxml2’s XML_PARSE_HUGE parser option. Without this option the parser will fail with a “xmlSAX2Characters: huge text node: out of memory” error when loading a text node larger than 10 MB. (xmlstarlet’s pyx subcommand uses the SAX API which has no such limitation.) In libxml2 the XML_PARSE_HUGE option is disabled by default to prevent denial-of-service attacks.

• --big-lines
  Load XML files with libxml2’s XML_PARSE_BIG_LINES parser option. This allows line numbers larger than 65535 to be reported correctly in error messages and (for select and transform) in output from the saxon:line-number extension. There is currently an open issue on this option suggesting it’s limited to text nodes, however, it appears to have been resolved by now as line numbers are output as expected for all node types except the root node (/) which is fixed at line -1.

Version 2023-01-29

Special characters

^(link)edit newline variable example changed:

--var nl 'substring-before('"$(printf '"\nA"')"',"A")'

XML parsing and serialization

^(link) Added:

The xmlEscapeEntities function in libxml2’s xmlsave.c serialization module gives special treatment to characters &<> (output as &, <, and >) but neither apostrophe nor double quote ('"). xmlstarlet has no option to override this.

Back reference $prev variable

^(link) Changed:

The $prev variable refers to a nodeset (changed from ‘node’).

PYX: xmlstarlet pyx, depyx

^(link) Added:

a pyx | depyx pipeline does not guarantee an accurate roundtrip

Namespace prefix «name» … is not defined

^(link) New entry, dealing with : (colon) in a name when not indicating a namespace prefix.

Context node and current node

^(links): select -t (--template) | select -m (--match) | ex4: external lookup table | extract subtree example | xmlstarlet edit | tree height example
Changed:

The current node – part of the XSLT processing model – was referred to as context node – part of the XPath evaluation context. Changed to ‘current node’ as this is more in line with XSLT 1.0 terminology.

Version 2022-10-31

SourceForge now accepts the HTTPS protocol so all remaining HTTP links (xmlstar.sourceforge.net and saxon.sourceforge.net) have been changed to HTTPS.

-q (--quiet): suppress error output

^(link) Changes:

Removed: ‘Caution: this option is ignored by xmlstarlet edit.’

Added: Link to format -Q (--quiet) local option.

-Q (--quiet) [undocumented] - suppress error output

^(link) New entry:

Does what the -q (--quiet) global option does.

-b (--break) ends current container element

^(link) Changes:

Added to the --var without = list item: any nested --vars are local to the enclosing --var but must have unique variable names.

Version 2022-09-30

User’s guide, examples, source code, forums

^(link) 2nd and 3rd list item replaced with:

• xmlstarlet on SourceForge: homepage | docs | user’s guide | news | source | files | discussion | bugs
  Caution: the doc/xmlstarlet.txt there is not the latest version as it doesn’t mention $prev, --var, -L (--inplace), and -E (--embed) – the user’s guide is still silent on these
• xmlstarlet on Fossies – an accessible presentation of source, examples, and more:
  xmlstarlet-1.6.1.tar.gz contents | xmlstarlet user’s guide (1-page) | doc/xmlstarlet.txt (latest version)

Links to doc/xmlstarlet changed to the latest version at Fossies here: $prev, --var, -L (--inplace), and -E (--embed).

Global options and parameters

^(link) Added after 1st paragraph:

An input filename starting with - (dash) – unless it’s short for stdin – must be prefixed with ./ (dot slash) otherwise it will be parsed as an option, possibly causing select (Caution) to ignore the file.

Beware of known bugs for filenames containing (#123 ) ' (single quote), or (#110) urlencoded characters, e.g. %20.

See also: couldn’t read file | failed to load external entity

-L (--inplace) - edit input file in-place

^(link) Added as last list item:

• has a known bug for filenames containing urlencoded characters, e.g. %20, cf. Global options and parameters

couldn’t read file

^{(new entry)}

See failed to load external entity (re stdin)

failed to load external entity

^(link) 3rd list item changed to:

• missing or inaccessible resource
  • if reading from stdin add a - (dash) to the command line to work around parsing issues, e.g. if format’s -e (--encode) «encoding» is the last option (explanation: «encoding» mistaken for filename in src/xml_format.c#foProcess(), similarly in src/xml_validate.c#valParseOptions())

Version 2022-09-05

Initial release.