Wasm `--no-docker` Builds + `wrap-exec` Musing

Proposal

I think Polywrap’s docker-based build system for wasm modules should be “opt-in” and not mandatory. This will help improve the speed of development for builders, since docker builds require additional setup of the virtual machine.

Rough Impl Details

In order to do this, we could:

  1. Divide the Docker.mustache file into 2 sections (ex: wasm-rust-setup & wasm-rust-build)
  2. > polywrap wasm build --no-docker would runCommand(...) for each step within the wasm-rust-build section.

Could be as easy as putting all build commands in a build.sh file, and in Dockerfile.mustache we just RUN bash build.sh

Consensus Check

Good idea?

  • Yes
  • No
  • Unsure

0 voters


Tangent (feel free to ignore)

^^^ it’s frustrating how there’s no good cross-platform scripting solution that isn’t node.js… maybe we should make a node.wrap executable to easily run things on the CLI like you can with node script.js. We’d need to better define Polywrap’s system-level interfaces, could replicate node’s like shown below…

Rough Impl Details

Someone could run this within their project’s directory:

wrap-exec .polywrap/packages/@polywrap/run-wasm-build-cmds/wrap.wasm -- --build-cmds .polywrap/build/wasm-rs/build-cmds.json
#import Executable from "node/executable"
#import Args from "node/args"
#import Process from "node/process"
#import ChildProcess from "node/child_process"
#import Path from "node/path"
#import FS from "node/fs"

type Module implements Executable {
  main: Int!

  # ignore for now, will be used below
  onStdout(data: String!): Void!
  onStderr(data: String!): Void!
}
export function main() {
  const cwd = Process.cwd();

  // --build-cmds .polywrap/build/wasm-rs/build-cmds.json
  const args = Process.args();

  // .polywrap/build/wasm-rs/build-cmds.json
  const buildCmdsArg = Args.extract("--build-cmds", "-b", args);
  const buildCmdsPath = Path.join(cwd, buildCmdsArg);
  const buildCmds = FS.readFile(buildCmdsPath, "utf-8");

  const projectManifest = Path.join(cwd, "polywrap.wasm.yaml");

  // TODO: craft the config for the build commands
  //       (i.e. variables in the Dockerfile.mustache)

  // Perform the build
  for (buildCmd of buildCmds) {
    ChildProcess.exec(
      buildCmd,
      callbacks: {
        stdout: onStdout,
        stderr: onStderr
      }
    );
  }
}

export function onStdOut(args: Args_onStdout) {
  Console.log(args.data);
}

export function onStderr(args: Args_onStderr) {
  Console.error(args.data);
}

The above wrapper could be run on windows & linux, the same way you can run node script.js on either OS.

Consensus Check

Good idea?

  • Yes
  • No
  • Unsure

0 voters

it’s frustrating how there’s no good cross-platform scripting solution that isn’t node.js

Sounds like a good case for CLI wrappers
Maybe add in workflows as well

My 2 cents on the docker builds being opt-in/opt-out:

Overall, I think it’s a good idea, I think the source verification system + auditing + profiting from the reproducibility of the docker builds is something that’s not being done or used; and I don’t know if we even have the need for it right now. So the performance + maintenance overhead makes us want to have a non-repro alternative and I think that’s fine. By making it opt-in, we allow users to only use it when it is actually a needed feature.

As to what I think this would entail, I think the .sh files approach is probably best and shouldn’t be too hard. But I think we should take into account that:

  • We would require users to install deps like toml-cli, wasm-snip, wasm-bindgen-cli, etc or install them for the user (we didn’t need to worry about this before because these deps lived only inside the docker environment)
  • We’d need to verify that failed attempts to run the build scripts are effectively “cleaned up”. I’m unsure if the build scripts leave behind autogen’d dirs or files, etc. Because, with docker we started with a “clean slate” every time and remnant files didn’t make it into the host machine