All notable changes to this project will be documented in this file.
Warning: Features marked as alpha may change or be removed in a future release without notice. Use with caution.
- The
manifestcommand now infers thepageBreakMarkersaccessibility feature when--inspect-htmlis combined with--infer-a11y(mergedorsplit). The publication's (X)HTML documents are scanned through their guided navigation representation for page break markers (epub:type="pagebreak"orrole="doc-pagebreak"), stopping at the first one found. The feature is added tometadata.accessibilityinmergedmode, or to the inferred accessibility object insplitmode
- The
jwt-bondingandjwks-bondingmodes no longer tie the bonding JWT to the browser'sAccept-Languageheader, only to itsUser-Agent. Safari leavesAccept-Languageout of the request that follows the bonding redirect, so publications failed to open in Safari with a 403 "browser integrity mismatch"
- Added the
--inspect-htmlflag to themanifestcommand. It converts the publication's (X)HTML documents into guided navigation documents and uses them to find where each image is used. Every image link inreadingOrderandresourcesthat is referenced from a document gains areferencesproperty: an array with one object per usage, carrying the image'sdescription(its accessible name, e.g. thealttext), the guided navigationrolevalues of the referencing element, and anhrefthat locates that element in its document, either through its fragment id (chapter.xhtml#par1) or, when it has none, acss()fragment with a unique CSS selector (chapter.xhtml#css(body%20%3E%20p:nth-child(3))). Decorative images (role="presentation", or an emptyaltwith no other accessible name) are not referenced. It can be combined with--inspect-images, which is unchanged
- The go-toolkit has been updated to v0.16.0. For the detailed list, refer to the go-toolkit's changelog; the highlights as they apply to the CLI:
- The guided navigation service (alpha) has been split in two, which changes the manifests
serveproduces for publications with media overlays. A dedicated media overlay service only converts SMIL media overlays, and surfaces solely through the alternates it swaps intoreadingOrderandresourcesitems, which now point at~readium/media-overlay.json?ref=...links. The guided navigation service (~readium/guided-navigation.json{?ref}, advertised in the manifest'slinks) now only converts (X)HTML documents, including ones that also have a SMIL overlay, and no longer adds alternates toreadingOrderorresourcesitems. Both services now cover documents inresourcesas well as the reading order. If you consume either of these endpoints fromserve, review your integration - The HTML to guided navigation converter can now locate every object in its source document through
textreflocators, which is what powers the new--inspect-htmlflag - Fixed a panic in the media overlay service when a manifest link has an alternate without a media type, and a panic in the CSS selector generator (shared by the content service and the
textreflocators) when its escaping produced a selector that could not be parsed back
- The guided navigation service (alpha) has been split in two, which changes the manifests
- The image inspection code behind
--inspect-imageshas moved from thepkg/helperspackage into a newpkg/inspectorpackage, built around anInspectorinterface so that multiple inspectors (currently image and image usage) run in a single pass over the manifest. Inspection now covers the whole manifest instead of onlyreadingOrderandresources. If you importhelpers.ImageInspectorfrom Go code, useinspector.Imageinstead - Go and third-party dependencies updated, along with the pinned Docker base images and GitHub Actions
- Docker builds use a local
mime.typesand pinned container hash for better security
- Fixed
session:scheme URLs for the web server not passing on query parameters to the session's HTTP request. Session URLs likesession:https://example.com/data.json?t=abcwill work now - Hash list for manifest command is now validated
- This has probably never shown up in the real world, but requesting the byte range
bytes=0-0for a resource would have caused issues in the web server - Further confine local FS reads on Windows. The long term solution is to is os.Root or such, but that requires go-toolkit changes
- Improved bonding reliability to deal with edge cases
- Improved open publication cache logic to deal with rare edge cases
- Standalone Readium Web Publication Manifests (e.g. a bare
manifest.json,.audiobookor.divinafile) can now be served. The publication's resources are resolved relative to the manifest's location, whether it lives on the local file system or on a remote source (http(s), s3, gs), and absolute HTTP(S) links in a manifest are fetched as-is. Previously, trying to open one would fail - Added the
--audio-parsing-cache-retainflag (enabled by default): the data fetched while parsing a remote audiobook (the container header and embedded chapter titles) is kept in memory, and requests for those byte ranges are served from it. Browsers request exactly these ranges before starting playback of e.g. an M4B, so with a many-chapter audiobook on a remote source, playback now starts several times faster. Disable the flag to save memory (it costs roughly one cache block per chapter plus the file headers, so a few MB per audiobook). Or just use a better audio format... - Publications opened from a WebPub manifest now get a positions service, and a guided navigation service (MVP) when they contain HTML content
- The go-toolkit has been updated to v0.15.1. Together with changes to the webserver, this dramatically reduces the latency and excess data transfer of streaming publications from remote (http(s), s3, gs) sources. For the detailed list, refer to the go-toolkit's changelog; the highlights as they apply to the CLI:
- The webserver now streams remote media to the client as it arrives whenever the range can be fetched efficiently (bare files and uncompressed archive entries), instead of buffering the entire requested range in memory first. Time to first byte no longer depends on the size of the requested range, memory use is bounded, and a client disconnecting cancels the remote transfer
- Streaming has been optimized to unlock the ability for data transfers from a local filesystem source to the network to happen in-kernel with
sendfile - Seeking through media inside a remote archive no longer re-downloads parts of the archive over and over (in the worst case, the whole archive tail on every request), and data is fetched in far fewer, larger range requests
- Opening the same publication only queries the remote source for a file's size once, instead of once per request while streaming. For audiobooks in particular, this removes a remote round trip from every range request a browser makes
- Fixed ranged reads of S3 resources downloading the entire object
- Fixed a path traversal issue: a crafted link inside a publication could address content outside the publication's own location (e.g. a sibling S3 key or URL path) on the remote source it is served from
- Clients disconnecting over HTTP/2 mid-response (e.g. a browser aborting range requests while probing or seeking media, which is routine) were logged as streaming errors such as
http2: stream closed, polluting the logs. They are now ignored like their HTTP/1 counterparts (broken pipe, connection reset)
- Added the
--audio-embedded-chapters,--audio-parsing-concurrency,--audio-parsing-cache-block-sizeflags for more control over audiobook parsing with regards to speed/latency/memory tradeoffs
- The go-toolkit has been updated to v0.14.0. This adds improved audiobook parsing. For a more detailed list of changed, refer to the go-toolkit's changelog. While best effort is made to minimize the latency when reading audiobooks from remote sources, there can still be an excess of reads that are necessary to fully parse metadata. The new settings provide a way to make tradeoffs depending on the situation
- v0.7.0 was tagged to the wrong commit, this is otherwise identical to v0.7.0 in terms of changes.
- Updated dependencies
- HTTP requests made by
servenow have a better, more descriptiveUser-Agentheader
- A new
sessionURL scheme has been added toserve, which allows for retrieval of a Reading Session API document. This is an experimental way of having a JSON object located at a remote URL (http, https) that returns the ebook's real location URL, as well as metadata to override, and rights (limitations) on the session. If you're trying to enforce restrictions seriously, you should use this together with one of the bonding modes below - Two new modes have been added to
serve,jwt-bondingandjwks-bonding. The difference between the two mirrots thejwtandjwksmodes. Bonding (which must be combined with the Reading Session API) lets you more strongly tie a reading session (reader URL) to a web browser, through the use of a device cookie and a bonding JWT. Be sure to check out the new params available to tweak settings for sessions/bonding - The webserver now strives to return problem details in JSON, which should helps clients better understand what the cause of an error is
- A CORS setting (
--cors-allowed-origin) is available forserve, for setups requiring CORS - An HTTP authorization setting (
--http-host-authorization) has been added, which will let you add theauthorizationheader to specific hosts you're making requests to. This should help authenticate the reader for the Reading Session API endpoint (or whatever other needs you have like auth for your ebook storage domain) - H2C (plaintext http2) support has been added to the webserver, for reverse proxies that support it
- Updated the go-toolkit, which fixes a race condition (error) that could occur reading local files
- Now, OPUS files has the mimetype
audio/opusin manifests, but remainaudio/ogg; codecs=opusin webserver responses for better compatibility
- Updated the go-toolkit, which fixes incorrect checking of whether a publication conforms to a specific profile based on its contents. This may have caused mistaken cases of a11y inferrence or Divina/Audiobook/EPUB/PDF profile detection in manifest outputs
- The go-toolkit has been updated, which fixes not being able to retrieve resources at paths containing special characters, such as spaces. This applies to any archived as well as exploded publications which contained such files. Retrieval of local files also had a bug
- Go and third-party dependencies updated
- The go-toolkit has been updated, and the version key present in WebPub manifests has changed from
https://github.com/readium/go-toolkit/releasestohttps://github.com/readium/go-toolkit#version. This was done to reduce confusion as go-toolkit releases have been superseeded by cli releases
- Problems with remote streaming of compressed multimedia have been addressed. This was due to a combination a bug in the implementation of partial resource reading, as well as special logic to make the kinds of requests browsers send for audio/video stream lead to smaller reads of the original resource. It's better just not to compressed multimedia in an EPUB though
- This update also fixes the issues with image inspection (failure to decode images) that occured due the bug in the partial resource reading, which was triggered by image decoding libraries reading fragments of the image files
- Go and third-party dependencies updated
- Docker container build fixed
- When using the serve command, a new
-mflag allows for authenticated access to publications using a JWT in the route to the publication instead of the encoded path. The subject (sub) of the JWT will instead be used as the path to the publication. The first new mode isjwtmode, which uses the HS256 method of authentication and a shared secret that is either provided using--jwt-shared-secretor autogenerated at startup. The second mode,jwks, is combined with the--jwks-urlflag that points to JWKS file, which can contain multiple keys used to validate the JWT, allowing for key rotation and other algorithms using public/private keypairs - The path of a publication with no resource specified now redirects to the manifest file
- The GOAMD64 value for release builds has been changed from
v3tov2. The discussion regarding this is here. This allows execution of the built binaries on older x64 CPUs - The HTTP client configuration used for streaming of remote publications has been changed to require, at minimum, TLSv1.2 for HTTPS connections
- The serve command's routes are now prefixed with
/webpub. So<domain>/<path>/manifest.jsonis now<domain>/webpub/<path>/manifest.json - Go and third-party dependencies updated
- The
/list.jsonroute in the serve command's webserver has been removed. It is not compatible with the new authenticated access schemes, and was only intended to be temporary. It may be replaced in the future by an OPDS2 feed
- When a manifest has no links (such as when using the
manifestcommand), thelinksproperty is now omitted instead of set tonullin the JSON output - The
--infer-page-countworks again, and general issues with services in themanifestcommand have been fixed due to a revamp in how they work in the toolkit
- Go and third-party dependencies updated
- The
--http-host-whitelistflag has been added to the serve command, to which a list of hosts can be passed. If at least one host is passed, access to streamed HTTP/HTTPS publications will be restricted to the provided hosts. A host like example.com can be further restricted to a "folder", such as example.com/the/path/ - The
--http-unsafe-requestsflag has been added. It disabled restrictions that are enabled by default to prevent access to private IP addresses (such as internal infrastructure or localhost), and should be used with caution
- The software version displayed when using the
--versioncommand was incorrect due to a bug. Thanks @tefkah for the fix
- Dependencies were updated
The WebPub data the go-toolkit parses and provides has been updated to more closely match the latest WebPub spec. Pay close attention to these changes if you depend on the WebPub output in reading systems/libraries that use an older version of the spec!
- Major changes to
readium servecommand syntax, to improve ease of use - Docker container no longer includes any test publications, an only enables http/https publication serving by default
- Improvements to documentation
- Updates to dependencies
- Links to services (positions list etc.) are removed from the manifest when running
readium manifest
- Add
--s3-use-path-styleflag to enable broader support for S3-like services that use the deprecated path-style bucket access (especially useful in docker environments)
- The version output from the CLI was incorrect, consisting of the go-toolkit version. It is now a combination of the CLI and go-toolkit version
- Add
--infer-a11y-ignore-image-hashesand--infer-a11y-ignore-image-dirflags tomanifestcommand to ignore certain images when inferring a publication's a11y access mode
- Upgraded go-toolkit from v0.9.0 to v0.10.1
- Default hashing algorithms for inspected images changed from SHA-256 + MD5 to just SHA-256
- Created this project based on the command-line utilities from the [https://github.com/readium/go-toolkit](Go Toolkit)