Repository navigation
📦 3.1.0 - #920
Merged
Merged
📦 3.1.0#920
Conversation
Bumps and [js-yaml](https://github.com/nodeca/js-yaml). These dependencies needed to be updated together. Updates `js-yaml` from 4.1.0 to 4.1.1 - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@4.1.0...4.1.1) Updates `js-yaml` from 3.14.1 to 3.14.2 - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@4.1.0...4.1.1) --- updated-dependencies: - dependency-name: js-yaml dependency-version: 4.1.1 dependency-type: indirect - dependency-name: js-yaml dependency-version: 3.14.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…i-75e6bc5210 build(deps): bump js-yaml
tslint is archived and ESLint 8 is end of life. The repo-level flat config also stops ESLint from picking up configs from parent directories.
Meteor 3.2 is the oldest release the dependencies resolve on, and 3.5.2 is the latest.
ostrio:cookies 3.0.0 needs fetch@0.1.6, which first ships with Meteor 3.2. Tests now run on mocha only, so the Tinytest helpers moved to tests/helpers.test.js. BREAKING CHANGE: Meteor 3.0 and 3.1 are no longer supported. Upgrade the app to Meteor 3.2 or newer.
…mits
Any client could delete any file record through _Abort, take over another user's upload by its id, overwrite or delete other files through a chosen file name, and send chunks of any size. Uploads now belong to the user who started them, files are created exclusively and checked by device and inode, paths must stay inside storagePath, and sizes, chunk ids, and request bodies are bounded.
Range parsing, Content-Disposition encoding, route matching, link() URL encoding, and several write-stream bugs are fixed in the same change because they share the upload and serve code paths.
BREAKING CHANGE: HTTP upload errors use { error: <status>, reason, isClientSafe? }. chunkSize is at most 16 MiB, Start bodies at most 1 MiB, EOF bodies at most 64 KiB. Uploads started on 3.0.x must restart. writeAsync() and loadAsync() reject with 409 for an existing fileId. Start rejects size 0 with 400.
The upload could send duplicate chunks after a reconnect, stall when Start or EOF failed, report failures as aborts, and leak workers and timers. It now sends one request at a time, retries network errors with backoff, surfaces server reasons, and cleans up on every exit path. Server-side sync cursor methods returned Promises or wrong values, so they now throw. BREAKING CHANGE: a failed upload emits only error and end, not pause, abort, or onAbort. A chunk is retried at most 10 times in a row on network failures. FileUpload pipes must not grow a chunk. Sync FilesCursor and FileCursor methods throw on the server; use the *Async methods.
The declarations sat inside a declare module wrapper that exported nothing, and many options and signatures did not match the code. index.test-d.ts covers them with tsd.
Several samples threw or used APIs removed in Meteor 3 and MongoDB driver 4, and constructor.md listed wrong defaults. The security guide explains the defaults that expose files, such as protected: true and allowClientCode.
Linux reuses an inode number right after unlink, so a file recreated at the upload path passed the device and inode check. The identity now includes the birth time in nanoseconds when the file system reports it.
…ests Replace `no-native-reassign` with `no-global-assign` and drop `no-extra-semi`, so the config keeps working after these rules are removed. Each helper test had its own `describe` with one `it`.
- Send Content-Length on 206 responses and never with Transfer-Encoding: chunked. A response with both is invalid, and some clients and proxies reject it. - Answer ?play=true without a Range header with 200, as RFC 9110 requires. - Add charset=utf-8 to the default Content-Type of text types, so text files display correctly inline. - Replace % with _ in the ASCII filename fallback (RFC 6266). - Treat ENOENT as success in unlinkAsync() and removeAsync(). - On _Abort without a stream in memory, remove the partial file when its identity matches. Before, the file stayed on disk. - An EOF that races the first EOF (DDP reconnect) waits for it and gets the same result instead of 408.
The `end` callback of a rejected upload gets a file object without `_id`, and link() turned it into `/undefined/original/undefined.txt`. A stored route with `?` or `#` would change the query or fragment of the URL, so it is no longer treated as safe.
Exceptions thrown by user callbacks went to Meteor._debug, which logs at the same level as other debug output, so they were easy to miss. The retry limit error blamed pipes even on a plain network outage, and it fired after 11 failed attempts while the docs said 10.
The demo upload form stayed on "Uploading" after abort() because the docs did not say that abort() emits no `end`.
serve() can not know how many bytes a caller's readableStream sends. A stream that is not cut to the requested range, or a compressed stream, then sends more or fewer bytes than Content-Length says, and the extra bytes corrupt the next response on a keep-alive connection. Responses from a custom stream are sent chunked again, as in 3.0 for 206. A 200 from a custom stream no longer gets Content-Length from the stored size. Responses that serve() reads from disk keep Content-Length.
…loads An anonymous owner's racing EOF takes the null === null owner path, and an abort after restart must keep the file of a record without a stored file identity.
…ckslashes URL parsers decode `%2e%2e` to `..`, so an encoded route could still contain a dot segment.
The docs say 5 attempts, but the check allowed a 6th.
Users who pass their own stream to serve() need to know that it should hold exactly the requested range, and that Content-Length from serve() does not fit a gzip-encoded body.
3.0 sent Content-Length with the stored size on a 200 from a caller's stream, and download progress in browsers relies on it. Only 206 from a custom stream stays without Content-Length, because a stream that is not cut to the range would send more bytes than the header says.
Feature/debug function
Await storagePath in upload start, writeAsync, and loadAsync. Sync functions and strings keep returning a string. An async function returns a Promise, so callers must await it. At startup the directory is created best effort when the async function resolves for an empty object. Based on #909. Co-authored-by: Surya Rath <18166257+codeonprod@users.noreply.github.com>
v3.1.0: secure uploads, fix the client upload flow, support Meteor 3.2 to 3.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's new
This release closes several upload and download security holes, fixes the client upload state machine, and supports Meteor 3.2 to 3.5. Read "Major changes" before you upgrade. See the new security guide.
Major changes
ostrio:cookies3.0.0 needsfetch@0.1.6, which first ships with Meteor 3.2.403or410for upload records created by 3.0.x.{ error: <status code>, reason, isClientSafe? }instead of{ error: "<text>" }.5xxresponses have a genericreason.chunkSizeis at most 16 MiB. HTTP Start bodies, includingmeta, are at most 1 MiB. HTTP EOF bodies are at most 64 KiB. HTTP chunk bodies are at most the Base64 size ofchunkSizeplus 4 KiB.FileUploadpipes that grow a chunk. A pipe must return Base64 that decodes to the same number of bytes as its input, so per-chunk encryption with an IV, a tag, or padding fails on the first chunk. Move such transforms toonAfterUploadon the server.errorandendwhen an upload fails.pause,abort, andonAbortnow fire only whenabort()is called.Meteor.Error 404from the synchronousFilesCursormethods (get,fetch,first,last,next,previous,hasNext,count,forEach,each,map,current,remove) and fromFileCursor#with()andFileCursor#remove()on the server. They returned Promises or wrong values before. Use the*Asyncmethods.writeAsync()andloadAsync()with409whenopts.fileIdalready exists. Before, they overwrote the existing file.file.size0with400. An empty upload has no chunk to send and never finished. The client already refused empty files.Other Changes
Fixed
_Abortfrom removing finished files. It removes only the unfinished upload record and its partial file, and answers404for unknown or foreign upload ids.409if the file id, the target path, or a pending upload path already exists, and creates the file exclusively.410or409when its file was removed or replaced on disk. The server compares device, inode, and birth time on every reopen.namingFunctionoutput per path segment and require the final path to stay insidestoragePath._id,fileId,path,_storagePath,_downloadRoute,_collectionName,versions,userId,public, extension, mime, and type flags.chunkId, and chunk length.413before buffering them.check()on HTTP Start as on DDP Start.500.Content-Dispositionwith an ASCIIfilenameand an RFC 8187filename*. The ASCII fallback replaces%with_._id, version, extension, and collection name inlink()and thefileURLhelper, and never produce protocol-relative URLs.link()and thefileURLhelper for a file without_id, for example the file passed toendafter a rejected upload. It returned/undefined/original/undefined.txtbefore.x_mtokcookie assecureon https pages.__proto__,constructor, andprototypeinfixJSONParseandfixJSONStringify.protectedresult to400-599.408.responseHeaders.bytes=-N) and the last byte, clamp the range end to the file size, and return200for multi-range requests. Unsatisfiable ranges return416whenstrictistrueand200when it isfalse.Content-Lengthon206responses thatserve()reads from disk, instead ofTransfer-Encoding: chunked. A response never carries both headers.206responses from a customreadableStreamchunked, withoutContent-Length, becauseserve()can not know how many bytes the stream sends. AContent-Lengthset inresponseHeadersis kept. A200still getsContent-Lengthfrom the stored size, as in 3.0.502,503, or504, after 5 failed attempts, as documented. Before, it took 6.400response reachable again.loadAsync()from throwing inside a timer. It opens the file only after a successful fetch, takessizefrom disk, and removes only partial files it created.writeAsync()create the directory and always close the file handle.503when a chunk write fails.WriteStream#end()returnfalsefor aborted or incomplete streams, and stopabort()from deleting a finished file.namingFunctionandstoragePathonce per upload, at Start.onInitiateUploadafter the server checked that the target does not exist.-.extensionWithDotto an empty string for files without extension.responseHeadersfunction that returns nothing as{}.unlinkAsync()andremoveAsync(). The server logs one debug line without a stack trace.502,503, and504retry with backoff from 500 ms up to 10 s.endanduploadedonly after a successful EOF.continue()follows a pause before Start.abort()from causing an unhandled rejection when the server answers404.reasonof HTTP4xxresponses toerror,onError, andend.errorandendonce per failed upload. A throwing callback is reported withconsole.errorand does not stop the others.x-mtokheader when there is no session instead of sending the string"null".insert()andinsertAsync()from changing the config object passed by the user.metaDates as Dates after an HTTP upload.beforeunloadlistener, timers, and trackers only afteronBeforeUploadpasses, and always clean them up.new FilesCollection().x_mtokcookie after reconnect and login without overwriting the app'sonReconnecthook.removeAsync()with401whenallowClientCodeisfalse.null,undefined, andFileCursorinlink().countDocuments()as infind(), including ObjectID.limitandskipinhasNextAsync()andlastAsync().estimatedDocumentCount()from failing in debug mode.index.d.tsto the runtime API. Many options and method signatures were missing or wrong.debugfunction throws. The error goes to the console._dataToSchema()whenstoragePathresolves to a Promise that was not awaited, so a Promise never lands in a stored document.Added
allowedCordovaOriginsoption on Server and Client, passed toostrio:cookies. Cordova and Meteor-Desktop need it for query-string cookies. The Server default is the value ofallowedOrigins.uploadIdleTimeoutServer option (milliseconds, default900000). It closes file handles of idle uploads, and the next chunk reopens them.nosniffServer option. It setsX-Content-Type-Options: nosniff(defaultfalse, planned to default totruein v4).allowClientCodeis on andonBeforeRemoveis not set.userAsync()to the client_getUser()result.debugoption to be a function. The server and client pass log arguments to it instead of the console. Thanks to @jankapunkt, Feature/debug function #907.storagePathfunction to beasync.collection.storagePath(fileObj)returns a Promise in this case, soawaitit. Thanks to @codeonprod, async support added for storagePath() function #909.Changed
_downloadRouteand_collectionNameinlink()when they are safe, otherwise the collection's own values. A stored route with?,#, or an encoded.,/, or\(%2e,%2f,%5c) is not safe.?play=truerequests without aRangeheader with200and the whole file, as RFC 9110 requires. Requests withRangestill get206.; charset=utf-8to the defaultContent-Typeoftext/*,application/json,application/javascript, andimage/svg+xmlfiles without a charset, so text files display correctly inline. Text files in another encoding, such as a Latin-1 CSV, need their charset set throughresponseHeaders.chunkSize(for example from a pipe) as one possible cause.index.d.tsat the top level. The olddeclare modulewrapper exported nothing.FileUploadpipes in reverse order of registration. This changes in v4.docs/audits/in git.Docs
migration-to-v3.md.@aws-sdk/client-s3v3), Dropbox, Google Cloud Storage, GridFS, andsharpthumbnail guides with async APIs and correct Range handling.updateAsync,countDocuments,estimatedDocumentCount,serve,download,allow/deny, the exported helpers, and theprogressevent arguments.about-transports.md.abort()emitspause, thenabort, and noend. A failed upload emitserror, thenend. Document thatabort()does not cancel an EOF or HTTP Start request that is already sent.readableStreampassed toserve()for aRangerequest should contain exactly the requested bytes, and whenserve()setsContent-Length.CLAUDE.mdwith commands, architecture notes, and invariants for contributors and coding agents.CONTRIBUTING.md: Meteor 3.2 requirement, links to the security and migration guides, andIsomorphicinstead ofAnywherein API labels.debugfunction and the asyncstoragePathoption indocs/constructor.md, andawaitstoragePath()in thesharpthumbnail guide.Tests
tests/security.test.jsfor upload ownership, path safety, request limits, Range parsing, and access checks.Content-Lengthon200and206for files and custom streams,HEAD,?play=true, default charsets, racing EOFs (also anonymous), abort after restart (also for records without file identity),link()without_id, encoded routes, and unlink of missing files.tscandtsd(npm run typecheck).debugfunction option and asyncstoragePath(Promise return, rejection, startup,writeAsync,loadAsync)..eslintrcwith an ESLint 9 flat config, without the deprecatedno-extra-semiandno-native-reassignrules.Dependencies
ostrio:cookies3.0.0eventemitter35.0.4meteortesting:mocha3.4.0,chai6,sinon22 for testsjs-yaml, ESLint 9, TypeScript 5.9,tsd) and the packagenpm-shrinkwrap.json