Repository navigation
Conversation
…#11042) Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.56.0 to 1.57.0. - [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/cznic/sqlite/compare/v1.56.0...v1.57.0) --- updated-dependencies: - dependency-name: modernc.org/sqlite dependency-version: 1.57.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…fs#11045) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.8 to 4.37.9. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.8...v4.37.9) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5 to 6. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](actions/setup-java@v5...v6) --- updated-dependencies: - dependency-name: actions/setup-java dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…eaweedfs#11046) Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.43.5 to 1.45.1. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@v1.43.5...v1.45.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-version: 1.45.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…eedfs#11044) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.293.0 to 0.294.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.293.0...v0.294.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.294.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…3437-b6eac429d7b6 to 1.85.0-dev (seaweedfs#11043) build(deps): bump google.golang.org/grpc Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.84.0-dev.0.20260723093437-b6eac429d7b6 to 1.85.0-dev. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/commits/v1.85.0-dev) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.85.0-dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
…seaweedfs#11048) * sts: session duration no longer clamped to the web identity token exp The assumed-role session lifetime is governed by DurationSeconds and the configured tokenDuration/maxSessionLength, matching AWS. Clamping to the already-verified token's exp made short-lived id_tokens (GitLab issues ~2-minute ones) yield unusable sessions regardless of configuration. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * sts: cover session duration against short-lived web identity tokens The mock OIDC provider now carries the token exp through to the identity like the real provider, so the integration test would catch the clamp. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
…aweedfs#11050) * filer_pb: walk a re-delivered directory only once in TraverseBfs A directory handed back twice by a listing (a page-boundary race with concurrent renames, or a store whose ordering misbehaves) was enqueued twice; the second walk re-lists the same subtree and can keep the traversal from ever terminating. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * filer_pb: fail a directory listing whose pagination stops advancing A full page ending on the very name the cursor started from re-fetches the same page forever; a store whose listing order does not advance past the cursor turns any full-directory read into a silent infinite loop. Return an error naming the stuck cursor instead. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * shell: skip foreign-collection manifests in fs.mergeVolumes Every manifest chunk in the namespace was resolved, downloading its manifest needle, even when the merge plan only touches one collection. Sub-chunks live in the manifest's own collection, so a manifest on a volume outside the plan's collections cannot reference a source volume; skip it and spare a cluster-wide download pass that looks like a hang after the real moves finish. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
* s3: resolve admin bucket subresources to their specific S3 actions Encryption, requestPayment, publicAccessBlock and ownershipControls requests reached the policy engines as s3:*, so only a policy granting all of s3 could authorize them. Map each subresource to its AWS action, with DELETE sharing the PUT permission as AWS does. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * s3: authorize CreateBucket as s3:CreateBucket in the policy engine A plain bucket-level PUT is registered with ACTION_ADMIN, which resolved to s3:*, so no attached policy short of s3:* could match it. Federated sessions whose policy explicitly allowed s3:CreateBucket were always denied while the same policy worked for object operations. Resolve it to s3:CreateBucket, like DeleteBucket already resolves. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ
noLock says "this invocation changes nothing" -- volume.balance, volume.move,
volume.copy, volume.merge and volume.fix.replication all set it for a dry run,
and none clears it. The CommandEnv is created once and reused by both
dispatchers, the interactive shell and the master's maintenance script runner,
so a simulation left every later command unlocked:
volume.balance -noLock # changes nothing
volume.move ... # mutates, and skips its lock
Reset before dispatch in both, where the invocation begins. forceNoLock is
untouched: that is set once, deliberately, for a trusted path.
…aweedfs#11058) * master: scope the startup capacity shed to a truly empty topology The retryable "no volume server capacity registered yet" shed checked capacity for the requested disk type, so a cluster serving only other media -- where that capacity will never register -- shed every assign until the client's deadline instead of failing fast. An unsteered write to such a cluster hung for its full HTTP deadline and surfaced "context deadline exceeded" in place of "No writable volumes". Shed only while no disk type has any registered capacity, and name the unserved medium in the fast failure. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R * master: name the unserved medium for every fail-fast caller The diagnostic sat in the growth-initiator block, so a follower joining an in-flight growth and a growth-disabled master failed the same way with only the generic pick error. Wrap at the fail-fast break instead, which every caller reaches, and cover all three paths in the test. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R
* volume server: open volume files with O_NOATIME Nothing reads the atime of .dat, .idx, .sdx, or EC files, but every needle read still dirtied the inode: even relatime writes atime on the first read after each write, so an actively written volume paid a metadata write per read/write cycle, and strictatime mounts paid one per read. Open the serving handles with O_NOATIME, falling back to a plain open when the file belongs to another owner (EPERM). Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * seaweed-volume: mirror the O_NOATIME volume file opens Same change as the Go volume server: serving handles for .dat, .idx, .sdx, .ecx, .ecj, and shard files open with O_NOATIME on Linux, with a plain-open fallback on EPERM. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * route the tier-down and recreate .dat opens through the no-atime helper Review caught the Rust tier-down swap opening the local .dat directly. The Go swapToLocalDatBackend and the zero-length read-only .dat recreate in maybeWriteSuperBlock had the same gap: all three install long-lived serving handles. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD
Three dispatchers reuse one CommandEnv: the interactive shell, the master's maintenance script runner, and the plugin worker's admin script handler. The first two were fixed; this is the third. It changes nothing today -- ForceNoLock already exempts this path -- so it is here to keep the rule the same everywhere rather than resting on that exemption staying in place.
…s in (seaweedfs#11060) * master: keep the periodic growth scan to data centers hosting the layout The rack-aware scan planned growth for every data center in the topology, so a collection pinned to one DC (fs.configure -dataCenter) sprouted volumes in all the others within one scan cycle. Plan only for data centers already hosting the layout's volumes; an empty DC gets its volumes from the DC-constrained assign that first asks for them. The lastGrowCount divisor likewise counts only the racks the scan can plan for. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: pin periodic must-grow growth to a single-DC layout's data center The must-grow and crowded paths of the periodic loop grow with no DataCenter, so even with the scan fixed a pinned collection's volumes could still land in any DC once lastGrowCount demands more writables. Stamp the grow request with the layout's data center when its volumes all live in one; layouts spanning DCs keep unconstrained growth. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: never pin growth of a cross-DC-replicated layout A layout whose replication spans data centers cannot legitimately live in one DC; observing a single hosting DC there means the other DCs are down. Do not encode that outage as a placement constraint. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: bound the hosting-DC walk by the answer it needs listVolumeDataCenters walked every location of the layout under accessLock — ~190ms for a million volumes, twice per layout per cycle, stalling assigns behind the read lock. Stop once enough distinct DCs answer the caller's question: two for the single-DC check, the topology's DC count for the scan. A spanning million-volume layout now finishes in microseconds; only a layout truly confined to fewer DCs still pays a full walk, the same cost class as the under-replication count this loop already takes each cycle. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5
seaweedfs#11069) * filer.sync: count received, processed, and failed events and export in-flight jobs The metadata processor admits at most -concurrency jobs and blocks the subscription stream past that, so the backlog lives in the source filer's metadata log and cannot be counted here. What can be measured honestly: events read off the stream, replication outcomes, and worker saturation. in_flight_jobs pinned at the concurrency limit means the sync itself is the bottleneck; near zero means it is caught up or starved by the source. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: export replication lag in seconds Lag is now minus the freshest of the processed watermark and the last idle heartbeat: the watermark stops at the last real event, so a quiet caught-up stream would otherwise show phantom lag. A ticker drives the gauge because the offset callback only fires while events flow and freezes exactly when the workers are saturated. Until the first event or heartbeat the gauge stays unset rather than reporting lag against a zero or stale resume offset. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: track replicated data sizes alongside event counts An event count hides that 32 in-flight jobs can be 32 KB or 300 GB. Byte counters mirror the event counters, and in_flight_bytes pairs with in_flight_jobs. An event's size is the chunk delta - new chunks the old entry does not already have - so deletes, renames, and attribute-only updates count zero and byte rates reflect data movement, not metadata churn. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * grafana: chart the new filer.sync metrics The lag panel reads lag_seconds directly instead of deriving it from sync_offset, and the sync row gains event rate, throughput, and the in-flight jobs and bytes gauges. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: a pinned failure keeps showing as lag An idle heartbeat means the stream is consumed, not that every event replicated. While a permanent failure pins the watermark, letting the heartbeat advance lag_seconds or the sync_offset gauge would report a caught-up stream with an unreplicated event in it, so both now ignore heartbeats until a restart replays the failure. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: in-flight gauges survive subscription retries A subscription retry builds a new processor sharing the gauge children while the old processor's jobs may still be draining, so setting the gauge from either side's local count clobbers the other. Each job now increments and decrements for itself, keeping the total truthful across generations. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw
…eaweedfs#11067) The .versions lookup retried every error through the full backoff ladder, so a missing key spent 12.7s (8 attempts, 100ms..6.4s) before the pre-versioning fallback could answer. NotFound is an answer, not a transient failure: gate the retries on isRetryableFilerErr, the same classifier retryFilerOp already uses, which also stops retrying for callers whose context is canceled or past its deadline. GetObject already treats NotFound on .versions/ as definitive; this brings the retention/tagging/ACL/attributes/delete/copy paths that go through getLatestObjectVersion in line with it. Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
…eaweedfs#11068) * s3api: drop two redundant filer round-trips per listed version entry ListObjectVersions paid two avoidable getEntry calls while walking a bucket, both re-fetching data the walk already held: - getObjectVersionList re-read the .versions directory entry that every caller had just received from listing the parent directory (or from its own sibling probe). Pass the entry down instead: one RPC saved per object listed. - getObjectOwnerFromVersion, on a version with no stamped owner, re-fetched the same version entry its OwnerID had been extracted from. The refetch cannot answer differently, so data written before owners were stamped cost one futile RPC per listed version, forever. All round-trips on this path are sequential, so on large versioned buckets (Veeam-style workloads) they add up to a visible share of per-page latency and gateway CPU. Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4 * s3api: treat a nil .versions entry as an empty version list filer_pb.GetEntry's contract permits (nil, nil) for an absent entry, and the old internal lookup answered that case with an empty list. Keep that answer now that the entry arrives from the caller. Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
…fs#11071) * volume: an already-deleted EC needle is not a delete failure Deleting a needle that is already gone is what the caller asked for, and the non-EC paths have always said so: BatchDelete reports StatusNotModified when DeleteVolumeNeedle finds nothing to do, and DeleteHandler answers 404 from its ReadVolumeNeedle pre-check. The EC branches had no such case, so ErrorDeleted fell through to a generic failure -- 500 from both, and DeleteHandler also counted it in VolumeServerFileWriteFailures, inflating a failure metric on a replayed or duplicated delete. The filer already tolerates this by string-matching "already deleted" on the result, which leaves an error message load-bearing; the status is now right at the source instead. Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1 * volume: close the EC fixture's disk location Close stops the location's disk-space goroutine and releases the mounted EC volume's file handles, which otherwise live until the test binary exits. Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
seaweedfs#11070) The version walk listed every directory with no prefix, transferring all 1024-entry batches over gRPC and filtering gateway-side - and kept paging past the point where names can no longer match. On wide directories (many sibling orgs/jobs next to the requested prefix) that is most of the transfer, decode, and CPU cost of every page. Derive the next path component of the requested prefix per directory level and hand it to the filer listing. A name holds no slash, so a directory whose name does not start with the component cannot contain a matching key and a file that does not cannot be one; stores with native prefixed listing (sql, leveldb) turn this into a range scan and stop the stream at the end of the prefix zone. Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4
…edfs#11072) * s3: share the per-key object authorization across copy and delete AuthorizeCopySource and AuthorizeObjectDelete both authorize a key the request URL does not name by evaluating the bucket policy and IAM against a synthetic per-key request; only the method and action differed. Extract that into authorizeObjectKeyAction and make the two callers thin wrappers. No behavior change. Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5 * s3: route POST Object uploads through the shared object authorization POST Object uploads (presigned-POST / HTML form) authorized the write with only the coarse per-identity Write action, unlike the other write paths which also check the resolved object against the bucket policy and IAM. Route POST through authorizeObjectKeyAction via a new AuthorizeObjectWrite so it is authorized like the equivalent PUT. Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5 * s3: test POST Object per-key authorization Drives a signed POST upload and checks the per-key authorization decision for a denied, permitted, and admin caller. Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5
…eaweedfs#11073) * mount: remember the entry of a directory removed while still referenced A directory removed while a descriptor is open on it keeps its inode until the kernel's final forget, but unlike a file it has no handle to live on through: OpenDir hands out only a listing cursor. Keep the last-known entry in memory, keyed by inode, from rmdir until that forget. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK * mount: serve metadata ops on a removed open directory from its remembered entry fchmod, futimens, and the f*xattr calls on a descriptor whose directory was removed failed with ENOENT: maybeReadEntry resolved the inode to a path, and rmdir had already dropped it. Fall back to the remembered entry the same way an unlinked file falls back to its open handle. Mutations publish a changed copy back rather than editing in place, so a concurrent reader never sees a half-applied change, and the empty path keeps nlink 0 in every reply. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK * mount: stash the entry the delete itself returned, not an earlier snapshot A chmod landing between Rmdir's entry load and the delete RPC would be resurrected pre-change: the remembered entry was the earlier local snapshot. The filer serializes the delete against updates under the path lock and hands the entry back in the delete event, so prefer that, keeping the local load for the sticky-bit check and as fallback when no event comes back. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK * mount: drop a remembered entry whose insert lost to the final forget The forget's cleanup runs between RemovePath and the insert when the kernel evicts the inode concurrently, finds nothing, and the entry would sit in the map for the life of the mount. Re-check the inode after inserting and take the entry back out; every interleaving now ends with the map empty. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK * mount: insert the remembered entry under the inode table lock The post-insert HasInode re-check could be fooled by inode number reuse: a lookup landing between the forget and the check makes the number look alive and the stale entry stays, keyed to someone else's inode. Do not check after the fact — RemovePath now runs the retention callback inside its critical section, where the forget that releases under the same lock cannot have run and cannot be missed. Publishes need no such fence: their open descriptor keeps the kernel from issuing the final forget in the first place. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
…ectories (seaweedfs#11075) The POSIX suites skirt this: pjdfstest's unlink/14.t covers only fstat and pread on an unlinked descriptor — its driver has no fchmod at all and never opens a directory. Pin the full rule in the FUSE integration suite: ftruncate, fchmod, futimes, fstat, and the f*xattr calls keep working between the removal of the last name and the final close, for a file after unlink and a directory after rmdir, with nlink 0 and the changes visible to a following fstat. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK
Membership was a linear scan over a slice, and ensureCorrectWritables runs it for every volume on every heartbeat, so the master's steady-state cost per volume server is quadratic in that server's volume count. BenchmarkSyncDataNodeRegistration, median of 3: 1000 volumes 565.7us -> 535.0us -5.4% 100000 volumes 1.665s -> 55.3ms -96.7% Allocations are unchanged at both sizes, so the difference is the scan. Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
…eaweedfs#11077) The replace pinned thrift to a post-v0.23.0 master commit so 32-bit GOARCHes would compile. That fix shipped in v0.24.0, so the replace was only overriding the v0.24.0 require back to the pseudo-version and holding the build below the CVE-2026-43871 fix. Claude-Session: https://claude.ai/code/session_01C5BpSeYD3yULWmfVXwPRmB
…st/kafka (seaweedfs#11078) build(deps): bump google.golang.org/grpc in /test/kafka Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.81.1 to 1.83.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.81.1...v1.83.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.83.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
mini hardcoded the per-directory volume limit to 0, so the volume server always auto-sized it as free disk space divided by the volume size. That sizing reserves a whole volume size for every writable volume, so a workload spreading small objects over many buckets runs out of slots long before the disk fills and assign starts failing with "no free volumes left". Same name and semantics as the flag weed server already carries, and still 0 (auto) by default. Claude-Session: https://claude.ai/code/session_01BiQLeBvZLzG8XitjiypDKu
ValidatePassword compared the stored and supplied passwords with subtle.ConstantTimeCompare, which returns 1 for two zero-length slices. A user provisioned for public-key-only auth has an empty stored password, so an empty supplied password authenticated as that user whenever "password" was among the enabled auth methods (the default). Treat an empty stored or supplied password as a non-match. Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
…seaweedfs#11096) recursivelyCheckLocksWithClient tested EntryHasActiveLock only on non-directory entries, so a directory-marker object (an S3 key ending in "/") that carries retention or a legal hold was recursed into but never lock-checked. DeleteBucket then saw no locks and removed the bucket, destroying an object under active Object Lock along with the rest of the bucket. DeleteObject already enforces the lock on the same key, so the two paths disagreed. Check the directory entry for an active lock before recursing. Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY
…11359) fix: flush metadata log before closing filer store Serialize sealed-batch handoffs with shutdown, reject late appends, and wait for log-buffer workers before closing the filer metadata store. Cover queued writes, interval and explicit flushes, late-write rejection, and pending persistence with shutdown tests.
* s3: separate object lock configuration permission * test: synchronize manifest cancellation setup
…#11347) * fix(ec): encode drops tombstoned needles, last-wins replay * fix(ec): drop zero-offset rows in encode, match readNeedleMap
…1363) Serve can return when its listener closes while HTTP requests are still draining. The main path could then close the metadata store before those requests finish. Make signal, context, and Serve-exit paths join one shutdown sequence. Drain gRPC and HTTP concurrently with 15-second default limits, then close the store. Test both completion orders.
…weedfs#11348) * fix(ec): BatchDelete cookie fail-closed via locate_data geometry * fix(ec): honor skip_cookie_check, require full cookie header * fix(ec): retry short cookie header reads, still fail closed on EOF * chore(ec): trim cookie validation comments --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>
admin: replace Font Awesome with MIT icons
s3: enforce dedicated object lock actions
* s3tables: hide inaccessible table buckets * s3tables: hide inaccessible namespaces * s3tables: hide inaccessible tables * s3tables: hide inaccessible resources in rename and namespace delete RenameTable/RenameView denied on the source now report the same not-found as a missing source, and the destination name conflict is checked only after destination authorization so a denied caller cannot distinguish an existing destination namespace or name from a missing one. DeleteNamespace denials use the same formatted message as a missing namespace.
… Windows (seaweedfs#11342) * rust volume: one positional read helper; never seek a dup'd handle on Windows Positional read-exact was hand-rolled four times: the complete cross-platform version in needle_map/sorted_file.rs, a Windows-only half in volume.rs whose unix half was inlined as a cfg(unix)/cfg(windows)/compile_error! triple at three call sites, a byte-identical Windows-only copy in ec_volume.rs, and read_full_at in ec_bitrot.rs. Three more sites -- EcVolumeShard::read_at, EcLocalShard::read_at and ec_encoder::read_at_most -- hand-rolled the short-read-permitted variant with a cfg(not(unix)) arm that try_clone()s the handle and seeks it. That last arm is wrong. A duplicated descriptor shares one kernel file offset with the original, so seek-then-read is two syscalls against state another thread can move in between: a concurrent reader or an append repositions the offset and the read returns bytes from somewhere else entirely. EcLocalShard::read_at documents that it must never seek, one line above the seek. Windows seek_read carries its own offset in a single call, so that window does not exist. All seven now go through storage::io::{read_exact_at, read_at}, whose module doc records why duplicating a handle is not a way to get a private file position -- opening the file again is, as Volume::dat_scan_plan already does. read_at_most keeps its own fill-until-EOF loop; only the per-iteration positional read changes. Behaviour on unix is unchanged: every unix arm was already FileExt::read_exact_at or FileExt::read_at. The one exception is ec_bitrot::verify_shard_blocks, which now retries on EINTR (std's read_exact_at does; the loop it replaces did not) and, on unix, reports the standard "failed to fill whole buffer" text instead of "short read on shard block". The Windows arm still says "unexpected EOF in seek_read"; both carry ErrorKind::UnexpectedEof, as before. NeedleStreamSource::read_exact_at and Volume::read_exact_at_backend keep their signatures; only their bodies shrink. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: retry Interrupted in Windows read_exact_at Unix std's FileExt::read_exact_at ignores ErrorKind::Interrupted and retries, but the Windows seek_read loop propagated it, so the shared exact-read contract differed by platform. seek_read can surface ERROR_OPERATION_ABORTED, which std maps to Interrupted. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…rom 1.44.0 to 1.45.0 (seaweedfs#11372) build(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace Bumps [go.opentelemetry.io/otel/exporters/otlp/otlptrace](https://github.com/open-telemetry/opentelemetry-go) from 1.44.0 to 1.45.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.44.0...v1.45.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace dependency-version: 1.45.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* security: require go 1.26.6 and bump vulnerable deps A dependency scan of the 4.47 release flagged the bundled toolchain and modules: - github.com/golang/go < 1.26.6 (CVE-2026-39821, CVE-2026-56853, CVE-2026-56859, CVE-2026-56862, CVE-2026-56864, CVE-2026-56865, CVE-2026-33818, CVE-2026-46600): raise the go directive to 1.26.6 so every built artifact requires the fixed toolchain. - google.golang.org/grpc (CVE-2026-84445, CVE-2026-84304): move to the fixed dev pseudo-version; released tags through v1.85.0-dev remain in the affected range. - github.com/pelletier/go-toml/v2 <= v2.4.2 (unbounded parser recursion): v2.4.3. - alpine libcrypto3/libssl3 < 3.5.8-r0 (CVE-2026-75803, CVE-2026-63073, CVE-2026-63075, CVE-2026-63076, CVE-2026-63072, CVE-2026-54874, CVE-2026-18798, CVE-2026-14456, CVE-2026-14457): the release images already apk-upgrade the final stage; extend the same to the telemetry and admin-integration images. Same bumps applied to the test/kafka, test/sftp, kafka-client-loadtest, and telemetry/server modules. * telemetry: send integration test report above the 10 GiB floor The collect endpoint keeps reports only when TotalDiskBytes >= proto.MinDiskBytes, but the integration test still sent 1 GiB, so the server counted the report and skipped storing it. No cluster_id series was ever created and /metrics lacked seaweedfs_telemetry_volume_servers. Send just above the floor (via proto.MinDiskBytes so it cannot silently drift again) so the expected per-cluster metrics are exported.
….0 to 1.45.0 (seaweedfs#11373) build(deps): bump go.opentelemetry.io/otel/exporters/zipkin Bumps [go.opentelemetry.io/otel/exporters/zipkin](https://github.com/open-telemetry/opentelemetry-go) from 1.36.0 to 1.45.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.36.0...v1.45.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/exporters/zipkin dependency-version: 1.45.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
…tlptracegrpc from 1.44.0 to 1.45.0 (seaweedfs#11371) build(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc Bumps [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc](https://github.com/open-telemetry/opentelemetry-go) from 1.44.0 to 1.45.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.44.0...v1.45.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc dependency-version: 1.45.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
…weedfs#11376) * s3api: map ambiguous filer transport errors to retryable 503 Canceled, DeadlineExceeded and Unavailable can be returned after the filer applied the write, so the outcome is ambiguous. Reporting them as a 4xx tells the client not to retry; report ServiceUnavailable instead. * s3api: verify entry existence before deleting orphaned chunks A failed CreateEntry can still have landed on the filer when the error is a transport failure, and entryCreated=false would tombstone chunks a live entry references, leaving a dangling pointer that survives only because reads pass readDeleted=true until vacuum reclaims the needle. Before deleting, look the entry up: if it is stored with the same chunks, the write succeeded; if the lookup cannot be answered, keep the chunks for vacuum to reclaim; only a confirmed absence still cleans up. * s3api: regression tests for ambiguous CreateEntry outcomes Covers the three post-create-failure cases in putToFiler: the entry landed despite the error (treat as success, keep chunks), the entry is confirmed absent (delete orphans), and the outcome is unverifiable (keep chunks, return error). * volume: count reads served from deleted needles A readDeleted read succeeding on a tombstoned needle is the signal that metadata still points at deleted data. Count it under a readDeletedNeedle handler label in both the Go and Rust volume servers so the condition is visible before vacuum turns it into a 404. * s3api: never delete chunks on an ambiguous create error Review feedback on the first fix showed verification could still go wrong in both directions: a stale or lagged lookup could report not-found for a committed entry, a prefix object stores its chunks on a directory entry, and filer-side manifestization rewrites the top-level chunk ids the comparison relied on. Rework the rule so the outcome classes are asymmetric: - A transport-level error (anything filerErrorToS3Error maps to a retryable 503) is ambiguous and never deletes chunks; the lookup can only upgrade the write to success. - Any other error is a definitive filer refusal and still cleans up. confirmCreateLanded asks the write owner first, resolves the stored entry through chunk manifests, requires an exact match of the uploaded file ids, and on success runs the finalize callback the failed create skipped (under the object write lock, with the same rmObject undo the create path uses). Zero-chunk writes stay ambiguous since they cannot be told apart by chunks. * s3api: cover definitive refusals and stale entries in put tests The confirmed-failure case now uses a definitive refusal so it still exercises orphan cleanup, and a new case keeps chunks when the stored entry belongs to an older object rather than this PUT. * volume: count deleted-needle reads once per request Streamed Go reads ran the deleted check in readNeedle and again in readNeedleDataInto, and non-streamed Rust reads in stream_info and the full-read fallback, double-counting one request. Count at the single entry probe each implementation takes per GET: readNeedle in Go, read_needle_stream_info in Rust. * s3api: run recovered-write rollback under the object lock Two follow-ups from review: ResolveChunkManifest returns traversed manifest blobs in its manifestChunks output, so requiring it empty rejected every manifestized landing; and the rmObject undo ran after the object write lock was released, so a concurrent newer write could be deleted between finalize failure and rollback. Compare only the resolved data chunks and keep the undo inside the lock. * s3api: verify, finalize and roll back recovered creates in one lock A lookup done before the object write lock let a concurrent PUT replace the entry between the chunk comparison and the finalize/rollback section, so a failed afterCreate could rmObject a newer write. Run the owner lookup, manifest resolution, chunk comparison, afterCreate and the conditional undo inside a single withObjectWriteLock section.
* telemetry/server: tidy module dependencies * ci: make telemetry deploy work on Oracle Linux 7 * ci: install telemetry unit and logrotate on every deploy * ci: abort telemetry deploy on install failures
…false (seaweedfs#11370) * mount: keep a deliberate bucket allow-empty-folders setting * shell: s3.bucket.allowEmptyFolders toggles the empty folder cleaner * shell: guard allow-empty-folders toggle with expected extended attrs * filer: drop cached empty-folder policy on bucket entry update * mount: guard allow-empty-folders write with expected extended attrs * filer: skip caching a stale cleanup policy read across an update * mount, shell: snapshot the full extended attributes for update preconditions * filer: fail closed and invalidate on all bucket entry events for cleanup policy * filer: key the cleanup policy generation by bucket * filer: requeue cleanup when the bucket policy cannot be loaded * filer: expire idle cleanup policy generations * filer: skip requeueing cleanup after the cleaner stops * filer: bound cleanup retries on repeated policy failures * filer: cover cleanup requeue on repeated policy failures * filer: keep cleanup policy generations while reads are in flight * filer: exercise the cleanup queue lifecycle in the retry-cap test
…pleted uploads metadata-only (seaweedfs#11375) * s3: commit versioned multipart upload in one transaction CompleteMultipartUpload wrote the version file, flipped the .versions pointer, then removed .uploads/<id> metadata-only as a best-effort post-commit step. A filer error or gateway crash in that window left the upload directory referencing the same chunks as the published object, and the next s3.clean.uploads run purged it with data -- corrupting a committed object. Put the version file, remove the upload directory metadata-only (its chunks are the object's chunks), and recompute the latest pointer in one ObjectTransaction under the object's per-path lock on the owner filer. The mutation order keeps every partial state safe: the chunks stay referenced at all times, and a published object never coexists with the upload directory the cleaner would purge. Unused part entries are freed before the transaction, since the metadata-only directory delete would otherwise leak their chunks. * s3: remove upload directory inside the multipart object PUT The same committed-object/stranded-upload window existed on the suspended and non-versioned paths: writeMultipartObject committed the object, then a best-effort rm dropped .uploads/<id>. Ride the metadata-only removal on the routed PUT itself so the two land in one transaction; the unrouted mkFile fallback keeps post-commit cleanup. * shell: purge completed uploads metadata-only in s3.clean.uploads A leftover .uploads/<id> can outlive a committed object when the completion's metadata-only delete fails or the gateway dies in between; its part entries then share chunks with the live object, and a recursive purge frees them out from under it. Before purging a stale upload, check whether it completed: the object entry or any version file under <key>.versions carrying the upload id. If so, delete with skipChunkDeletion. If the lookup fails, skip the upload for this run rather than risk live chunks. * s3: abort multipart completion when unused part cleanup fails Deleting the upload directory metadata-only erases the only metadata pointing at part entries whose deletion failed, orphaning their chunks. Propagate the error so the completion fails while the upload directory still exists and the request remains retriable. * s3: require the upload directory to exist at multipart commit A delete that does not take the object lock (abort, lifecycle, s3.clean.uploads) can remove .uploads/<id> and its chunks between the prepare step and the commit transaction. The commit now carries an IF_EXISTS precondition on the upload directory so the race fails the request with NoSuchUpload instead of publishing an object over freed chunks. * s3: keep the version file when the upload directory is gone The finalize transaction has no rollback, so a failure at the latest-pointer recompute leaves the version written and .uploads/<id> removed. Deleting the version then destroys the only remaining record of the upload, making a retried CompleteMultipartUpload return NoSuchUpload while the version's chunks leak. Roll back only while the upload directory survives; otherwise keep the version, which a retry resolves through SeaweedFSUploadId and the version reconciler promotes. * s3: keep manifests when a routed object write partially commits For non-versioned and suspended completions the object PUT precedes the upload-directory DELETE, so an error can mean the object entry exists while the response reports failure. Freeing this attempt's manifest chunks then destroys the committed object. Keep them when the object entry survived, and after a failed null-marker finalize which always follows a committed write. * s3: skip the keep-version path on precondition failure A rejected precondition means no mutation ran, so there is no version file to preserve and this attempt's manifests are orphans the error cleanup should free. * s3: keep manifests when the object-existence check itself fails A transient lookup error previously read as absent, letting the error cleanup free manifest chunks a committed object still references. * s3: keep the upload directory when post-commit part cleanup fails Removing it metadata-only after a failed entry delete erases the only reference to the leftover chunks. Leave the directory so the entries keep their chunk references for s3.clean.uploads or manual recovery. * pb: fix filer list entry counting on 32-bit int(limit) wraps to -1 on 386 when limit is math.MaxUint32, so the beyond-limit check discarded every streamed entry. Compare in uint64 instead; the semantics are unchanged on 64-bit platforms. * shell: resolve trailing-slash object keys in s3.clean.uploads Completion stores a key ending in / inside the directory it names (<bucket>/dir/dir), but FullPath+DirAndName on the normalized key looked one level too high. Deriving dir and name with path.Dir and path.Base mirrors getEntryNameAndDir so the completed-upload check finds the entry instead of purging its chunks. * s3: heal a suspended completion hidden behind a delete marker Removing .uploads/<id> inside the commit transaction means a failed finalizeSuspendedNullWrite leaves nothing to retry against: the object entry is committed but the marker still makes the key read as deleted, and a retried CompleteMultipartUpload can only report NoSuchUpload. When the upload directory is gone, check the regular path for an entry carrying the upload id and re-run the marker finalize, so the retry both succeeds and repairs the key. Only suspended buckets can hold this state; anything newer owns the key. * s3: report store errors when resuming a committed multipart object
* s3api: test that a session token must not reveal its credential * sts: derive secret access key with HMAC keyed on the signing key * s3api: stop accepting STS session tokens as bearer credentials * security: reject STS session tokens on filer and admin gRPC auth * test: sign s3/iam framework requests with the session credential * s3api: exercise the real auth pipeline in the end-to-end harness
…eaweedfs#11381) * s3api: fail closed when S3 Tables signature verification fails * s3api: avoid nil Account dereference in S3 Tables auth log * iceberg: return auth error instead of falling back to DefaultAllow * lance: return auth error instead of falling back to DefaultAllow * s3api: stop trusting client-supplied s3-account-id The header is set by the server after successful authentication; scrub inbound values alongside the other internal headers, and apply the same admin guard to the header fallback branch of getAccountID that the identity branch already has. * test: cover table-catalog auth wrappers and principal resolution * test: configure anonymous identity where catalog clients do not sign * s3api: scrub s3-account-id after signature verification
* s3: abort a completed upload's leftover directory metadata-only A .uploads/<id> directory can outlive the object it completed into when the commit's metadata-only removal failed or the gateway died in between; the restored part entries then share chunks with the published object. AbortMultipartUpload deleted the directory recursively, chunks and all, so aborting such a leftover destroyed a committed object (seaweedfs#11382). Run the same check s3.clean.uploads gained in seaweedfs#11375 before deleting: when the object entry or a version file under <key>.versions carries the upload id, remove .uploads/<id> metadata-only and answer the abort; when the lookup cannot decide, refuse with InternalError rather than risk live chunks. * s3: apply the completed-upload check to lifecycle MPU abort lifecycleAbortMPU ran the same destructive recursive delete on .uploads/<id>. Reuse uploadCompleted so a leftover whose object entry or version file carries the upload id is removed metadata-only, and an undecidable lookup retries later instead of freeing live chunks. * s3: serialize abort's upload-dir delete with the object's commit The completed check alone leaves a race: abort can read completed=false, then an in-flight completion publishes the object over the same part chunks before the recursive delete frees them. Run the check and delete inside the object write lock, which non-routed completions hold for their whole finalize. With an owner, send the data delete as an ObjectTransaction on the object's lock key — a routed commit then either loses its upload-exists precondition after our delete or has already stamped the object, which the transaction's IF_EXTENDED_NOT_EQUAL condition detects and falls back to a metadata-only remove. lifecycleAbortMPU shares removeUploadDir so both callers get the same ordering. * s3: check for an empty object before resolving its write owner * s3: check completion at the abort's resolved object key An upload record missing ExtMultipartObjectKey skipped the completed check entirely even though the request's Key names the object.
…fs#11384) * fix(filer): leave reader cache unbounded without an explicit budget NewReaderCache silently installed a 256MiB ReaderCacheBudget when the caller passed none. Only weed mount opts into a budget; every other caller (S3 gateway, WebDAV, query engine, mq logstore) inherited the cap. Under ~90 concurrent S3 GETs of medium objects, prefetch wants far more than 64 chunk buffers, so reserve() serialized chunk fetches, clients timed out and retried, and the retry re-downloaded chunks the cancelled request had already fetched. A nil budget now means unbounded, restoring the pre-4.47 behavior for callers that never asked for a memory cap; reserve/complete/release are nil-safe. The mount path is unchanged and still enforces -readerCacheSizeMB. Fixes seaweedfs#11380 * feat(s3): expose -s3.readerCacheSizeMB reader buffer budget Operators who want the S3 gateway read path memory-bounded can now opt in: -s3.readerCacheSizeMB on weed filer/server/mini and -readerCacheSizeMB on standalone weed s3, matching the mount flag. The default 0 keeps the unbounded pre-4.47 behavior; a positive value installs a shared ReaderCacheBudget across in-flight and retained chunk buffers for all S3 GETs. * fix(filer): validate chunk size before consulting the reader budget A nil budget returned early and skipped the negative chunkSize check, letting a corrupted size reach mem.Allocate and panic. Also drop the command-specific flag prefix from the S3 validation error since standalone weed s3 exposes the option as -readerCacheSizeMB. * filer: drop chunk buffers once fully consumed ReaderCache retained every completed chunk buffer in the downloaders map until the slot limit evicted it, so buffers lingered after all readers finished with them. Track attached readers on each SingleChunkCacher and remove the cacher when the last reader consumes the buffer to its end. In-flight download deduplication and the prefetch handoff are unchanged: a buffer always survives until fully read, partial reads keep it available, and an attached reader pins a consumed buffer until it detaches. Repeat reads now go through the chunk cache where enabled, or refetch. * filer: drop consumed buffers on last detach, rechecked under cache lock Two review findings on the drop-on-consume change: - Removal only fired when the detaching reader itself reached the chunk end. If the end-reaching reader finished first and the last remaining reader did a partial read or cancelled, the consumed buffer and its budget reservation lingered until eviction. Track a persistent consumed flag instead, so any end-reaching read marks the buffer and the last detach drops it. - remove() checked only map identity, so a reader attaching between the reader count hitting zero and removal could attach to a cacher that was then deleted underneath it. removeConsumed() re-checks identity, readers == 0, and consumed under the ReaderCache lock; a raced attach keeps the cacher and its own detach retries the removal.
…seaweedfs#11389) * s3api: test for chunks deleted under an entry the filer committed Issue seaweedfs#11387: the filer can report a create failure after inserting the entry (e.g. a parent-directory creation failing post-insert). The error arrives in the response rather than as a transport status, so it maps to a definitive error and putToFiler deletes the chunks of the live entry. * s3api: confirmCreateLanded also reports a confirmed-absent entry The verification a failed create runs can answer both directions: the entry matching the uploaded chunks proves the write landed, and an authoritative not-found proves the uploaded chunks are orphaned. Return both outcomes so the cleanup path can gate on the fact rather than the error class. An empty upload can never prove a landing, so a zero-chunk entry match no longer upgrades the outcome. * s3api: delete orphaned chunks only when the entry is confirmed absent A failed create no longer skips verification based on the error class: the filer can fail after inserting the entry (issue seaweedfs#11387) and a partially-applied routed transaction can leave it behind too, both surfacing as definitive errors. Every failed create now resolves the entry's fate, and the uploaded chunks are deleted only when the entry is confirmed absent; anything unverifiable keeps them for vacuum. * s3api: confirm absence on every filer the create could have committed on A lock-path create fails over across filers, so the entry can live on a replica the routed owner has not caught up to; one not-found does not prove absence. The confirmation now queries the owner, the prior owner, and the failover set, declaring absent only when none of them has the entry. * s3api: bound the reconciliation lookups confirmCreateLanded runs The lookups ran on context.Background() under the object write lock, so a connected filer that never replies could stall the write path. One timeout now covers the whole enumeration; an expired budget fails the remaining lookups as uncertain, which keeps the chunks.
…#11390) * pb: stop exiting the process on malformed server addresses ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf when hostAndPort could not parse the port, which os.Exit(255)ed the whole process. A caller-supplied copy or tail source address reached this path synchronously in the serving goroutine, so one anonymous VolumeCopy with a non-numeric port terminated the volume server. Log the parse error and return the input unchanged instead: the dial or request that consumes the address then fails as an ordinary error. * volume: validate copy and tail source addresses before dialing VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a caller-supplied source address (SourceDataNode / SourceVolumeServer) with no endpoint validation, so an anonymous caller could aim the volume server at loopback, link-local (cloud metadata) or other unintended destinations and read dial behavior back as a connectivity oracle. Apply the same peer-target deny list FetchAndWriteNeedle uses for replica targets: the source must be a bare host:port whose host is not loopback, link-local or unspecified; cluster peers stay reachable on private networks, and -volume.allowUntrustedRemoteEndpoints opts out. The loopback-using copy tests set the flag to keep exercising the copy path in process. * rust volume: validate copy and tail source addresses before dialing Mirror the Go guard on the Rust volume server: volume_copy, volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied source address, so run it through validate_replica_target first (bare host:port; no loopback, link-local or unspecified hosts; private peers stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test fixture and the Rust test-cluster launcher set it so loopback sources in tests keep working. * volume: pin validated copy/tail source addresses at dial time validateReplicaTarget resolves the source hostname once, but the gRPC client resolved it again at connect, leaving a DNS-rebinding window for hostname sources. The copy and tail source dials now run through the same guardedDialerPolicy the remote-storage path uses, so every resolved address is re-checked against the replica deny list (private peers allowed) immediately before the TCP connect. guardedDialerPolicy also moves to util.OutboundDialContext so the guarded path keeps the -ip.bind source binding the default gRPC dialer had. The Rust volume server mirrors this with connect_guarded, a tonic connector that resolves, re-checks each address, and connects to the first passing IP; handlers use it whenever the untrusted-endpoint opt-out is off. A handler-level test now exercises the enabled validation branches for all three source-taking RPCs. * pb: return empty server address for malformed grpc addresses GrpcAddressToServerAddress used to return the unparseable input on a hostAndPort failure, so a malformed raft address (e.g. "host:abc") flowed into admin dashboard master maps unchanged. Return an empty string instead, skip empty conversions at the two raft-cluster merge sites, and drop the now-stale comment about the fatal exit the earlier commit removed. * test: opt erasure-coding loopback clusters out of the remote endpoint guard The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between volume servers bound to 127.0.0.1, which the copy/tail source guard now rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the test volume launches, matching what the volume_server framework harnesses already do. * admin: only claim fallback master leadership on an empty raft response A nonempty RaftListClusterServers response whose entries were all rejected left masterMap empty, so the fallback marked the reachable current master as leader the same way a genuinely empty (non-raft) response does. Track whether the successful response returned zero servers and only promote the fallback master then.
… size (seaweedfs#11393) ReadNeedleBodyBytes sliced the needle body with the size from the needle header without checking it. A corrupted .dat header carrying size -1 still gets a positive body length (16 bytes on v3), so vacuum compaction read that body and panicked with "slice bounds out of range [:-1]". Writers never put a negative size in a .dat header: a delete appends a size-0 record, and TombstoneFileSize only lives in the .idx. Reject a size that is negative or leaves no room for the checksum/timestamp tail with an error wrapping ErrorCorrupted. ScanVolumeFileFrom already logs body read errors and moves on, so compaction now skips the record like any other corrupt needle. Fixes seaweedfs#6763
* s3api: add Snowflake s3compat API integration tests Run the upstream snowflakedb/snowflake-s3compat-api-test-suite against a local SeaweedFS server in CI. test/s3/snowflake/run.sh starts weed server with S3 (-s3.autoCreateBucket=false so missing-bucket PUTs return NoSuchBucket), prepares the fixtures the suite needs (versioned bucket, deny-all-policy bucket, >1000-object prefix), clones the suite, patches it to path-style addressing, and runs mvn -Dtest=S3CompatApiTest. The suite also exposed that GetBucketLocation returned 404 NoSuchBucket for a malformed bucket name; validate the name first and return 400 InvalidBucketName like AWS. * test: harden snowflake s3compat runner per review - Pin the upstream suite to a tested commit (SUITE_REV) instead of the moving default branch - Bind the test server to loopback only - Require the AccessDenied error code when verifying the denied bucket - Fix README so go install runs in a subshell - checkout with persist-credentials: false - Make the concurrency group unique per PR, and widen path filters to the storage/operation/wdclient/cluster/pb packages the S3 stack uses * test: advertise loopback ip for snowflake test server -ip.bind 127.0.0.1 alone left the volume server advertising the host's primary address, so chunk uploads were refused. Also set -ip 127.0.0.1 and disable the Iceberg/Lance listeners so the harness is loopback-only and does not collide with other local services.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem are we solving?
Adds x-amz-expiration header support.
How are we solving the problem?
Adding the header in responses.
How is the PR tested?
Checks
Checks for AI generated PRs