Port to Solaris

[psumbera]

Currently mio is using for Solaris epoll interface. But Solaris doesn't have this interface. Only Illumos has it.

There are two things to be done:

• To distinguish between Solaris and Illumos.
• Implement selector on Solaris (probably via even ports API)

[Thomasdezeeuw]

So you're saying that the Solaris port never worked?

@psumbera do you use Solaris and are you willing to contribute such a port? Otherwise we maybe should just drop support for Solaris if it never worked in the first place.

/cc @carllerche

[carllerche]

Well, we need some way to test it in CI... none of the mio devs have a solaris setup.

[psumbera]

Mio is used by Rust. And Solaris really needs Rust to build Firefox. From Rust version 1.38 something changed and it now fails to link on Solaris because of this issue (though I know Mio was used also with previous Rust versions without the issue).

I'm Solaris engineer so I have access to Solaris. Unfortunately I'm not very good with Rust. But anyway I will try to come with something or will seek for help.

[Thomasdezeeuw]

Can you try testing Mio (by running cargo test) and give us an output of the error, then we can try and fix it.

Also could you help with setting up the CI for Solaris? Going forward we would like to a bit stricter about what platforms we support and having a CI for it is one of them.

[psumbera]

Can you try testing Mio (by running cargo test) and give us an output of the error, then we can try and fix it.

error: linking with `gcc` failed: exit code: 1
  |
  = note: "gcc" "-m64" "-L" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.11v52vx52e6g1ahq.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.13gmn7nbr8fz58ef.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.1a0zdiqerop42dtc.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.1qz1pata0l5g4pvq.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.1vxfmru39gxz4ar3.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.1xciivo4r5pr1dsq.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.24zcpn44b12eo1td.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.25r1lhkoly7a984g.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.2l3vzd2qkdu4ithc.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.2rlwld8q5u3rzw4m.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.2zklttukxouc1zzb.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.348stzlifohl7ig9.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.375xutsz6sjyrar4.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.39xu1wql7p9t2w6.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.4ap6wzhjmmr89gni.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.4dgjk7iptqfgxwiu.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.4kqcii0nyyrp2t8o.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.4txpvbe1msmro5ki.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.4zymg0dpk7z686tz.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.503g90dslmwg1yit.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.58y8hj1dfye6cfhp.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.f7q89bdlkfageun.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.jtqk5a7j6hdi865.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.no53nngwdccutsq.rcgu.o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.wwpf7khpch25zyy.rcgu.o" "-o" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc" "/builds/psumbera/mio-test/target/debug/examples/udp_server-2bc75a83567111bc.2c8h1obrgnxnb2gw.rcgu.o" "-Wl,-zignore" "-nodefaultlibs" "-L" "/builds/psumbera/mio-test/target/debug/deps" "-L" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib" "-Wl,-Bstatic" "/builds/psumbera/mio-test/target/debug/deps/libmio-13bc6fc3f9b5e16b.rlib" "/builds/psumbera/mio-test/target/debug/deps/liblibc-96233ac8f4948085.rlib" "/builds/psumbera/mio-test/target/debug/deps/liblog-56b3bee101ff990c.rlib" "/builds/psumbera/mio-test/target/debug/deps/libcfg_if-f7c1f14feae8d52e.rlib" "-Wl,--start-group" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/libstd-8db4654fdc908441.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/libpanic_unwind-3bfcf78ba2f0a654.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/libbacktrace-7e6fbe207ebada1c.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/libbacktrace_sys-37dec1614f68e8aa.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/librustc_demangle-7a6232bee123d2d4.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/libhashbrown-4b510aad541d18fe.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/librustc_std_workspace_alloc-58508acde8e63d53.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/libunwind-766e9e902a9094b7.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/libcfg_if-2e9ddb3347410ede.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/liblibc-ca073c74dcf43c77.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/liballoc-92a9c38060cc44cf.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/librustc_std_workspace_core-953de5af53a4fd9f.rlib" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/libcore-5fd40412d92f6541.rlib" "-Wl,--end-group" "/builds/psumbera/rustc-1.37.0/lib/rustlib/sparcv9-sun-solaris/lib/libcompiler_builtins-ba711d886e9bdfdf.rlib" "-Wl,-Bdynamic" "-lsocket" "-lposix4" "-lpthread" "-lresolv" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread"
  = note: Undefined            first referenced
           symbol                  in file
          epoll_wait                          /builds/psumbera/mio-test/target/debug/deps/libmio-13bc6fc3f9b5e16b.rlib(mio-13bc6fc3f9b5e16b.5d4rzgaw272n8cti.rcgu.o)
          epoll_ctl                           /builds/psumbera/mio-test/target/debug/deps/libmio-13bc6fc3f9b5e16b.rlib(mio-13bc6fc3f9b5e16b.5d4rzgaw272n8cti.rcgu.o)
          epoll_create1                       /builds/psumbera/mio-test/target/debug/deps/libmio-13bc6fc3f9b5e16b.rlib(mio-13bc6fc3f9b5e16b.5d4rzgaw272n8cti.rcgu.o)
          ld: fatal: symbol referencing errors
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

I'm trying to add 'src/sys/unix/selector/evport.rs' which would add support via Solaris even ports API.

My question is also whether it wouldn't be enough to add support for other platforms via select()?

Also could you help with setting up the CI for Solaris? Going forward we would like to a bit stricter about what platforms we support and having a CI for it is one of them.

This is different problem we are aware of. As for Solaris right now there is for free available only old image of Solaris 11.4. So if you have virtual environment where to run it (e.g. VirtualBox) I can do some setup. But as you can see see I'm now locked with Rust 1.37 till we solve Mio problem...

[Thomasdezeeuw]

I think the problem is that rust's libc wrapper has the epoll function defined here: https://github.com/rust-lang/libc/blob/bf877ba7123d8d1f990b70a8dda306942c9a93b0/src/unix/solarish/mod.rs#L2171-L2196. So currently Mio builds against these functions, but as you said (and comment in the libc code) these aren't present.

If you remove target_os = solaris from the targets here:

mio/src/sys/unix/selector/mod.rs

Lines 1 to 5 in ac0515c

	#[cfg(any(target_os = "linux", target_os = "android", target_os = "solaris"))]
	mod epoll;
	#[cfg(any(target_os = "linux", target_os = "android", target_os = "solaris"))]
	pub use self::epoll::{event, Event, Events, Selector};

. And create a new file in that directory for Solaris using select(), or whatever functions are appropriate, you can use it with the following target #[cfg(target_os = "solaris")]. Much like the mod and pub use statements in that file. Something like:

#[cfg(any(target_os = "linux", target_os = "android"))]
mod epoll;

#[cfg(any(target_os = "linux", target_os = "android"))]
pub use self::epoll::{event, Event, Events, Selector};

#[cfg(target_os = "solaris")]
mod evport;

#[cfg(target_os = "solaris")]
pub use self::evport::{event, Event, Events, Selector};

// kqueue...

In src/sys/unix/selector/evport.rs you would need to define the correct types and module as exported above.

Also could you update

mio/src/poll.rs

Line 168 in ac0515c

/// | Solaris | [epoll] |

.

[psumbera]

I have started my work (see here):
psumbera@f6284af

But now I'm getting:

error[E0277]: `*mut std::ffi::c_void` cannot be sent between threads safely
  --> tests/poll.rs:16:5
   |
16 |     assert_send::<Events>();
   |     ^^^^^^^^^^^^^^^^^^^^^ `*mut std::ffi::c_void` cannot be sent between threads safely
   |
   = help: within `libc::unix::solarish::port_event`, the trait `std::marker::Send` is not implemented for `*mut std::ffi::c_void`
   = note: required because it appears within the type `libc::unix::solarish::port_event`
   = note: required because of the requirements on the impl of `std::marker::Send` for `std::ptr::Unique<libc::unix::solarish::port_event>`
   = note: required because it appears within the type `alloc::raw_vec::RawVec<libc::unix::solarish::port_event>`
   = note: required because it appears within the type `std::vec::Vec<libc::unix::solarish::port_event>`
   = note: required because it appears within the type `mio::event::events::Events`
note: required by `util::assert_send`
  --> tests/util/mod.rs:110:1
   |
110| pub fn assert_send<T: Send>() {}
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Any idea about this one? Thank you!

[Thomasdezeeuw]

I guess port_event contains a *mut c_void field? In rust raw pointers don't implement the Send trait, which is the trait that defines that a type may be send across threads. You'll need to define your own Event type that wraps the port_event type and implement Send and Sync on it:

pub struct Event(libc::port_event);

unsafe impl Send for Event {}
unsafe impl Sync for Event {}

Note however you must manually ensure that the libc::port_event may be send over thread bounds. If this is not allowed we'll need to discuss if we want to drop the Send guarantee from Event.

[psumbera]

I guess port_event contains a *mut c_void field?

You are right:

    pub struct port_event {
        pub portev_events: ::c_int,
        pub portev_source: ::c_ushort,
        pub portev_pad: ::c_ushort,
        pub portev_object: ::uintptr_t,
        pub portev_user: *mut ::c_void,
    }

portev_user are user data. It doesn't have to be pointer (man page says: "user defined cookie"). It's set via port_associate(). I do set there token. Is that right?

I tried quick hack by modifying it to ::c_ulong in libc rust wrapper and now I get only following error:

error: `core::slice::<impl [T]>::len` is not yet stable as a const fn
  --> tests/unix_datagram.rs:21:26
   |
21 | const DATA1_LEN: usize = DATA1.len();
   |                          ^^^^^^^^^^^
   |
   = help: add `#![feature(const_slice_len)]` to the crate attributes to enable

Any clue?

In rust raw pointers don't implement the Send trait, which is the trait that defines that a type may be send across threads. You'll need to define your own Event type that wraps the port_event type and implement Send and Sync on it:

pub struct Event(libc::port_event);

unsafe impl Send for Event {}
unsafe impl Sync for Event {}

Sorry I'm no good enough with Rust yet. I tried:

--- a/src/sys/unix/selector/evport.rs
+++ b/src/sys/unix/selector/evport.rs
@@ -113,7 +113,11 @@ impl Drop for Selector {
     }
 }

-pub type Event = libc::port_event;
+pub struct Event(libc::port_event);
+
+unsafe impl Send for Event {}
+unsafe impl Sync for Event {}
+
 pub type Events = Vec<Event>;

 pub mod event {

And I get now:

error[E0308]: mismatched types
  --> src/sys/unix/selector/evport.rs:62:37
   |
62 |         syscall!(port_getn(self.ep, events.as_mut_ptr(), events.capacity() as u32, &mut nget as *mut u32, timeout))
   |                                     ^^^^^^^^^^^^^^^^^^^ expected struct `libc::unix::solarish::port_event`, found struct `sys::unix::selector::evport::Event`
   |
   = note: expected type `*mut libc::unix::solarish::port_event`
              found type `*mut sys::unix::selector::evport::Event`

error[E0609]: no field `portev_user` on type `&sys::unix::selector::evport::Event`
   --> src/sys/unix/selector/evport.rs:128:21
    |
128 |         Token(event.portev_user as usize)
    |                     ^^^^^^^^^^^

Can you please hint me little bit more?

Note however you must manually ensure that the libc::port_event may be send over thread bounds. If this is not allowed we'll need to discuss if we want to drop the Send guarantee from Event.

I hope there is no problem. Event port API is marked as thread safe.
Or am I missing something?

[Thomasdezeeuw]

portev_user are user data. It doesn't have to be pointer (man page says: "user defined cookie"). It's set via port_associate(). I do set there token. Is that right?

Yes then it means you can implement Send for it and it can be used in Token.

I tried quick hack by modifying it to ::c_ulong in libc rust wrapper and now I get only following error:

error: `core::slice::<impl [T]>::len` is not yet stable as a const fn
  --> tests/unix_datagram.rs:21:26
   |
21 | const DATA1_LEN: usize = DATA1.len();
   |                          ^^^^^^^^^^^
   |
   = help: add `#![feature(const_slice_len)]` to the crate attributes to enable

Any clue?

My guess it that you're using an older rust compiler version. Can you update to a newer version? Preferably 1.39. If not you can change those constants to literal values (that of the length of DATA1) for the time being.

error[E0308]: mismatched types
  --> src/sys/unix/selector/evport.rs:62:37
   |
62 |         syscall!(port_getn(self.ep, events.as_mut_ptr(), events.capacity() as u32, &mut nget as *mut u32, timeout))
   |                                     ^^^^^^^^^^^^^^^^^^^ expected struct `libc::unix::solarish::port_event`, found struct `sys::unix::selector::evport::Event`
   |
   = note: expected type `*mut libc::unix::solarish::port_event`
              found type `*mut sys::unix::selector::evport::Event`

error[E0609]: no field `portev_user` on type `&sys::unix::selector::evport::Event`
   --> src/sys/unix/selector/evport.rs:128:21
    |
128 |         Token(event.portev_user as usize)
    |                     ^^^^^^^^^^^

Can you please hint me little bit more?

The first problem is that now we're using different types. port_getn expects a pointer to an array of port_event structures, but we pass a pointer to the Event structs (which we defined). As these are different types Rust won't allow us to do that. I think the easier way forward is to remove the test in tests/poll.rs which caused the problem in the first place. We'll figure out later the best way to provide an Send implementation.

So revert the suggestion I made and go back to Event being a type alias for port_event. This should also resolve the second error.

Note however you must manually ensure that the libc::port_event may be send over thread bounds. If this is not allowed we'll need to discuss if we want to drop the Send guarantee from Event.

I hope there is no problem. Event port API is marked as thread safe.
Or am I missing something?

This is safe, see the first part of my comment

[psumbera]

Ok. With my modification for Rust libc: port_event item portev_user *mut ::c_void to ::c_ulong

and

--- a/tests/unix_datagram.rs
+++ b/tests/unix_datagram.rs
@@ -18,8 +18,8 @@ use util::{

 const DATA1: &[u8] = b"Hello same host!";
 const DATA2: &[u8] = b"Why hello mio!";
-const DATA1_LEN: usize = DATA1.len();
-const DATA2_LEN: usize = DATA2.len();
+const DATA1_LEN: usize = 16;
+const DATA2_LEN: usize = 14;
 const DEFAULT_BUF_SIZE: usize = 64;
 const TEST_DIR: &str = "mio_unix_datagram_tests";
 const TOKEN_1: Token = Token(0);

I have following test failures:

     Running target/debug/deps/mio-f560c53dfa705f78

running 3 tests
test poll::as_raw_fd ... ok
test sys::unix::uds::tests::abstract_address ... ok
test sys::unix::uds::tests::pathname_address ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/close_on_drop-3d9cbb23646a15fe

running 1 test
test close_on_drop ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/event_source-2b115c80abbcee13

running 1 test
test assert_event_source_implemented_for ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/interests-5ec5456f296032de

running 3 tests
test bit_or ... ok
test is_tests ... ok
test fmt_debug ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/poll-40df02e7a89b5311

running 9 tests
test is_send_and_sync ... ok
test poll_registration ... ok
test poll_erroneous_registration ... ok
test drop_cancels_interest_and_shuts_down ... ok
test registry_behind_arc ... ok
test zero_duration_polls_events ... ok
test poll_closes_fd ... ok
test add_then_drop ... FAILED
test run_once_with_nothing ... FAILED

failures:

---- add_then_drop stdout ----
thread 'add_then_drop' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 62, kind: Other, message: "Timer expired" }', src/libcore/result.rs:999:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

---- run_once_with_nothing stdout ----
thread 'run_once_with_nothing' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 62, kind: Other, message: "Timer expired" }', src/libcore/result.rs:999:5


failures:
    add_then_drop
    run_once_with_nothing

test result: FAILED. 7 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

[Thomasdezeeuw]

If you add RUST_BACKTRACE=1 (or full) to the env vars it logs a backtrace of the error. To run a single test you can add the test name after the command, for example RUST_BACKTRACE=1 cargo test add_then_drop.

For the error itself, I don't know what the source could be. It seems specific to the Solaris port though, as I've never seen the error for Mio before.

Also if you can't find the source of the error it's fine to ignore the test for a while by adding the #[ignore] attribute, see:

mio/tests/tcp_stream.rs

Lines 358 to 360 in d64a303

	#[test]
	#[ignore = "This test is flaky, it doesn't always receive an event after shutting down the write side"]
	fn shutdown_write() {

.

[psumbera]

I resolved the issue with 'add_then_drop' like this:
psumbera@e67a3da

But now I'm puzzled with 'register_deregister' test. port_dissociate() returns ENOENT at the end. The issue seems to be that I need to re-associate fd every time I get event for it from port_getn(). Man page says:

       PORT_SOURCE_FD events represent a transition in the poll(2) status of a
       given file descriptor. Once an event is delivered, the file  descriptor
       is  no longer associated with the port. A file descriptor is associated
       (or re-associated) with a port using the port_associate(3C) function.

And I don't do it now. Isn't this needed for epoll too?

[Thomasdezeeuw]

On epoll the registration is kept alive (kernel side) even when an event is delivered. It seems that ports doesn't do that. So you would have to reregister after each event to simulate level triggers, or after an operation returned would block or didn't fill a buffer to simulate edge trigger (default in Mio). We currently face a similar problem on Windows.

For now; on each I/O operation, e.g. read or write, call reregister to ensure another event will come. This would simulate level triggers but that is fine for now. You might need to ignore ENOENT errors however since the object may be used without it ever being registered. That should fix the register_deregister test.

If it does not you can ignore the ENOENT error in deregister, since the object is no longer registered in any case.

[psumbera]

For now; on each I/O operation, e.g. read or write, call reregister to ensure another event will come. This would simulate level triggers but that is fine for now.

How should I do it? Are there some mio read and write function I can modify?