Attempt to load all the required sync functions and fail if any one of them fails.
This reintroduces a macro for optional loading of functions but keeps it separate from the fallback macro rather than having that do two different jobs.
Remove synchronization from Windows `hashmap_random_keys`
Unfortunately using synchronization when generating hashmap keys can prevent it being used in `DllMain`.
~~Fixes #99341~~
Support setting file accessed/modified timestamps
Add `struct FileTimes` to contain the relevant file timestamps, since
most platforms require setting all of them at once. (This also allows
for future platform-specific extensions such as setting creation time.)
Add `File::set_file_time` to set the timestamps for a `File`.
Implement the `sys` backends for UNIX, macOS (which needs to fall back
to `futimes` before macOS 10.13 because it lacks `futimens`), Windows,
and WASI.
Implement network primitives with ideal Rust layout, not C system layout
This PR is the result of this internals forum thread: https://internals.rust-lang.org/t/why-are-socketaddrv4-socketaddrv6-based-on-low-level-sockaddr-in-6/13321.
Instead of basing `std:::net::{Ipv4Addr, Ipv6Addr, SocketAddrV4, SocketAddrV6}` on system (C) structs, they are encoded in a more optimal and idiomatic Rust way.
This changes the public API of std by introducing structural equality impls for all four types here, which means that `match ipv4addr { SOME_CONSTANT => ... }` will now compile, whereas previously this was an error. No other intentional changes are introduced to public API.
It's possible to observe the current layout of these types (e.g., by pointer casting); most but not all libraries which were found by Crater to do this have had updates issued and affected versions yanked. See report below.
### Benefits of this change
- It will become possible to move these fundamental network types from `std` into `core` ([RFC](https://github.com/rust-lang/rfcs/pull/2832)).
- Some methods that can't be made `const fn`s today can be made `const fn`s with this change.
- `SocketAddrV4` only occupies 6 bytes instead of 16 bytes.
- These simple primitives become easier to read and uses less `unsafe`.
- Makes these types support structural equality, which means you can now (for instance) match an `Ipv4Addr` against a constant
### ~Remaining~ Previous problems
This change obviously changes the memory layout of the types. And it turns out some libraries invalidly assumes the memory layout and does very dangerous pointer casts to convert them. These libraries will have undefined behaviour and perform invalid memory access until patched.
- [x] - `mio` - Issue: https://github.com/tokio-rs/mio/issues/1386.
- [x] `0.7` branch https://github.com/tokio-rs/mio/pull/1388
- [x] `0.7.6` published https://github.com/tokio-rs/mio/pull/1398
- [x] Yank all `0.7` versions older than `0.7.6`
- [x] Report `<0.7.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0081.html
- [x] - `socket2` - Issue: https://github.com/rust-lang/socket2-rs/issues/119.
- [x] `0.3.x` branch https://github.com/rust-lang/socket2-rs/pull/120
- [x] `0.3.16` published
- [x] `master` branch https://github.com/rust-lang/socket2-rs/pull/122
- [x] Yank all `0.3` versions older than `0.3.16`
- [x] Report `<0.3.16` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0079.html
- [x] - `net2` - Issue: https://github.com/deprecrated/net2-rs/issues/105
- [x] https://github.com/deprecrated/net2-rs/pull/106
- [x] `0.2.36` published
- [x] Yank all `0.2` versions older than `0.2.36`
- [x] Report `<0.2.36` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0078.html
- [x] - `miow` - Issue: https://github.com/yoshuawuyts/miow/issues/38
- [x] `0.3.x` - https://github.com/yoshuawuyts/miow/pull/39
- [x] `0.3.6` published
- [x] `0.2.x` - https://github.com/yoshuawuyts/miow/pull/40
- [x] `0.2.2` published
- [x] Yanked all `0.2` versions older than `0.2.2`
- [x] Yanked all `0.3` versions older than `0.3.6`
- [x] Report `<0.2.2` and `<0.3.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0080.html
- [x] - `quinn master` (aka what became 0.7) - https://github.com/quinn-rs/quinn/issues/968https://github.com/quinn-rs/quinn/pull/987
- [x] - `quinn 0.6` - https://github.com/quinn-rs/quinn/pull/1045
- [x] - `quinn 0.5` - https://github.com/quinn-rs/quinn/pull/1046
- [x] - Release `0.7.0`, `0.6.2` and `0.5.4`
- [x] - `nb-connect` - https://github.com/smol-rs/nb-connect/issues/1
- [x] - Release `1.0.3`
- [x] - Yank all versions older than `1.0.3`
- [x] - `shadowsocks-rust` - https://github.com/shadowsocks/shadowsocks-rust/issues/462
- [ ] - `rio` - https://github.com/spacejam/rio/issues/44
- [ ] - `seaslug` - https://github.com/spacejam/seaslug/issues/1
#### Fixed crate versions
All crates I have found that assumed the memory layout have been fixed and published. The crates and versions that will continue working even as/if this PR is merged is (please upgrade these to help unblock this PR):
* `net2 0.2.36`
* `socket2 0.3.16`
* `miow 0.2.2`
* `miow 0.3.6`
* `mio 0.7.6`
* `mio 0.6.23` - Never had the invalid assumption itself, but has now been bumped to only allow fixed dependencies (`net2` + `miow`)
* `nb-connect 1.0.3`
* `quinn 0.5.4`
* `quinn 0.6.2`
### Release notes draft
This release changes the memory layout of `Ipv4Addr`, `Ipv6Addr`, `SocketAddrV4` and `SocketAddrV6`. The standard library no longer implements these as the corresponding `libc` structs (`sockaddr_in`, `sockaddr_in6` etc.). This internal representation was never exposed, but some crates relied on it anyway by unsafely transmuting. This change will cause those crates to make invalid memory accesses. Notably `net2 <0.2.36`, `socket2 <0.3.16`, `mio <0.7.6`, `miow <0.3.6` and a few other crates are affected. All known affected crates have been patched and have had fixed versions published over a year ago. If any affected crate is still in your dependency tree, you need to upgrade them before using this version of Rust.
Rewrite Windows `compat_fn` macro
This allows using most delay loaded functions before the init code initializes them. It also only preloads a select few functions, rather than all functions.
This is optimized for the common case where a function is used after already being loaded (or failed to load). The only change in codegen at the call site is to use an atomic load instead of a plain load, which should have negligible or no impact.
I've split the old `compat_fn` macro in two so as not to mix two different use cases. If/when Windows 7 support is dropped `compat_fn_optional` can be removed entirely.
r? rust-lang/libs
This allows using most delay loaded functions before the init code initializes them. It also only preloads a select few functions, rather than all functions.
Co-Authored-By: Mark Rousskov <mark.simulacrum@gmail.com>
Windows: Use `FindFirstFileW` for getting the metadata of locked system files
Fixes#96980
Usually opening a file handle with access set to metadata only will always succeed, even if the file is locked. However some special system files, such as `C:\hiberfil.sys`, are locked by the system in a way that denies even that. So as a fallback we try reading the cached metadata from the directory.
Note that the test is a bit iffy. I don't know if `hiberfil.sys` actually exists in the CI.
r? rust-lang/libs
Add `struct FileTimes` to contain the relevant file timestamps, since
most platforms require setting all of them at once. (This also allows
for future platform-specific extensions such as setting creation time.)
Add `File::set_file_time` to set the timestamps for a `File`.
Implement the `sys` backends for UNIX, macOS (which needs to fall back
to `futimes` before macOS 10.13 because it lacks `futimens`), Windows,
and WASI.
Implement ExitCodeExt for Windows
Fixes#97914
### Motivation:
On Windows it is common for applications to return `HRESULT` (`i32`) or `DWORD` (`u32`) values. These stem from COM based components ([HRESULTS](https://docs.microsoft.com/en-us/windows/win32/api/objbase/nf-objbase-coinitialize)), Win32 errors ([GetLastError](https://docs.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-getlasterror)), GUI applications ([WM_QUIT](https://docs.microsoft.com/en-us/windows/win32/winmsg/wm-quit)) and more. The newly stabilized `ExitCode` provides an excellent fit for propagating these values, because `std::process::exit` does not run deconstructors which can result in errors. However, `ExitCode` currently only implements `From<u8> for ExitCode`, which disallows the full range of `i32`/`u32` values. This pull requests attempts to address that shortcoming by providing windows specific extensions that accept a `u32` value (which covers all possible `HRESULTS` and Win32 errors) analog to [ExitStatusExt::from_raw](https://doc.rust-lang.org/std/os/windows/process/trait.ExitStatusExt.html#tymethod.from_raw).
This was also intended by the original Stabilization https://github.com/rust-lang/rust/pull/93840#issue-1129209143= as pointed out by ``@eggyal`` in https://github.com/rust-lang/rust/issues/97914#issuecomment-1151076755:
> Issues around platform specific representations: We resolved this issue by changing the return type of report from i32 to the opaque type ExitCode. __That way we can change the underlying representation without affecting the API, letting us offer full support for platform specific exit code APIs in the future.__
[Emphasis added]
### API
```rust
/// Windows-specific extensions to [`process::ExitCode`].
///
/// This trait is sealed: it cannot be implemented outside the standard library.
/// This is so that future additional methods are not breaking changes.
#[stable(feature = "windows_process_exit_code_from", since = "1.63.0")]
pub trait ExitCodeExt: Sealed {
/// Creates a new `ExitCode` from the raw underlying `u32` return value of
/// a process.
#[stable(feature = "windows_process_exit_code_from", since = "1.63.0")]
fn from_raw(raw: u32) -> Self;
}
#[stable(feature = "windows_process_exit_code_from", since = "1.63.0")]
impl ExitCodeExt for process::ExitCode {
fn from_raw(raw: u32) -> Self {
process::ExitCode::from_inner(From::from(raw))
}
}
```
### Misc
I apologize in advance if I misplaced any attributes regarding stabilzation, as far as I learned traits are insta-stable so I chose to make them stable. If this is an error, please let me know and I'll correct it. I also added some additional machinery to make it work, analog to [ExitStatus](https://doc.rust-lang.org/std/process/struct.ExitStatus.html#).
EDIT: Proposal: https://github.com/rust-lang/libs-team/issues/48
Usually opening a file handle with access set to metadata only will always succeed, even if the file is locked. However some special system files, such as `C:\hiberfil.sys`, are locked by the system in a way that denies even that. So as a fallback we try reading the cached metadata from the directory.
Windows: Iterative `remove_dir_all`
This will allow better strategies for use of memory and File handles. However, fully taking advantage of that is left to future work.
Note to reviewer: It's probably best to view the `remove_dir_all_recursive` as a new function. The diff is not very helpful (imho).
once cell renamings
This PR does the renamings proposed in https://github.com/rust-lang/rust/issues/74465#issuecomment-1153703128
- Move/rename `lazy::{OnceCell, Lazy}` to `cell::{OnceCell, LazyCell}`
- Move/rename `lazy::{SyncOnceCell, SyncLazy}` to `sync::{OnceLock, LazyLock}`
(I used `Lazy...` instead of `...Lazy` as it seems to be more consistent, easier to pronounce, etc)
```@rustbot``` label +T-libs-api -T-libs
Windows: No panic if function not (yet) available
In some situations (e.g. #97814) it is possible for required functions to be called before they've had a chance to be loaded. Therefore, we make it possible to recover from this situation simply by looking at error codes.
`@rustbot` label +O-windows
In some situations it is possible for required functions to be called before they've had a chance to be loaded. Therefore, we make it possible to recover from this situation simply by looking at error codes.
Call the OS function to set the main thread's name on program init
Normally, `Thread::spawn` takes care of setting the thread's name, if
one was provided, but since the main thread wasn't created by calling
`Thread::spawn`, we need to call that function in `std::rt::init`.
This is mainly useful for system tools like debuggers and profilers
which might show the thread name to a user. Prior to these changes, gdb
and WinDbg would show all thread names except the main thread's name to
a user. I've validated that this patch resolves the issue for both
debuggers.
Normally, `Thread::spawn` takes care of setting the thread's name, if
one was provided, but since the main thread wasn't created by calling
`Thread::spawn`, we need to call that function in `std::rt::init`.
This is mainly useful for system tools like debuggers and profilers
which might show the thread name to a user. Prior to these changes, gdb
and WinDbg would show all thread names except the main thread's name to
a user. I've validated that this patch resolves the issue for both
debuggers.
Make HashMap fall back to RtlGenRandom if BCryptGenRandom fails
With PR #84096, Rust `std::collections::hash_map::RandomState` changed from using `RtlGenRandom()` ([msdn](https://docs.microsoft.com/en-us/windows/win32/api/ntsecapi/nf-ntsecapi-rtlgenrandom)) to `BCryptGenRandom()` ([msdn](https://docs.microsoft.com/en-us/windows/win32/api/bcrypt/nf-bcrypt-bcryptgenrandom)) as its source of secure randomness after much discussion ([here](https://github.com/rust-random/getrandom/issues/65#issuecomment-753634074), among other places).
Unfortunately, after that PR landed, Mozilla Firefox started experiencing fairly-rare crashes during startup while attempting to initialize the `env_logger` crate. ([docs for env_logger](https://docs.rs/env_logger/latest/env_logger/)) The root issue is that on some machines, `BCryptGenRandom()` will fail with an `Access is denied. (os error 5)` error message. ([Bugzilla issue 1754490](https://bugzilla.mozilla.org/show_bug.cgi?id=1754490)) (Discussion in issue #94098)
Note that this is happening upon startup of Firefox's unsandboxed Main Process, so this behavior is different and separate from previous issues ([like this](https://bugzilla.mozilla.org/show_bug.cgi?id=1746254)) where BCrypt DLLs were blocked by process sandboxing. In the case of sandboxing, we knew we were doing something abnormal and expected that we'd have to resort to abnormal measures to make it work.
However, in this case we are in a regular unsandboxed process just trying to initialize `env_logger` and getting a panic. We suspect that this may be caused by a virus scanner or some other security software blocking the loading of the BCrypt DLLs, but we're not completely sure as we haven't been able to replicate locally.
It is also possible that Firefox is not the only software affected by this; we just may be one of the pieces of Rust software that has the telemetry and crash reporting necessary to catch it.
I have read some of the historical discussion around using `BCryptGenRandom()` in Rust code, and I respect the decision that was made and agree that it was a good course of action, so I'm not trying to open a discussion about a return to `RtlGenRandom()`. Instead, I'd like to suggest that perhaps we use `RtlGenRandom()` as a "fallback RNG" in the case that BCrypt doesn't work.
This pull request implements this fallback behavior. I believe this would improve the robustness of this essential data structure within the standard library, and I see only 2 potential drawbacks:
1. Slight added overhead: It should be quite minimal though. The first call to `sys::rand::hashmap_random_keys()` will incur a bit of initialization overhead, and every call after will incur roughly 2 non-atomic global reads and 2 easily predictable branches. Both should be negligible compared to the actual cost of generating secure random numbers
2. `RtlGenRandom()` is deprecated by Microsoft: Technically true, but as mentioned in [this comment on GoLang](https://github.com/golang/go/issues/33542#issuecomment-626124873), this API is ubiquitous in Windows software and actually removing it would break lots of things. Also, Firefox uses it already in [our C++ code](https://searchfox.org/mozilla-central/rev/5f88c1d6977e03e22d3420d0cdf8ad0113c2eb31/mfbt/RandomNum.cpp#25), and [Chromium uses it in their code as well](https://source.chromium.org/chromium/chromium/src/+/main:base/rand_util_win.cc) (which transitively means that Microsoft uses it in their own web browser, Edge). If there did come a time when Microsoft truly removes this API, it should be easy enough for Rust to simply remove the fallback in the code I've added here
