2022-02, 03 - Grant Proposal - NeuralField

Name: NeuralField

Requested Grant:

Wallet Stable Coins Hours Rate / Hr. Period
0x3193DE955136dF26Cd44E2C0e6ffb1717F9b402C 22,400 USDC 140 hours 160 USDC Feb/Mar 2022

Snapshot

Approved Proposal

Summary:

I’m Jure. You know me as one of the Polywrap core developers. Recently, I set up a dev shop called “NeuralField” and am requesting this grant on its behalf.
The team for this proposal consists of two part-time Senior (Web2) Developers, with myself contributing in my free time to guide the development effort. We’re looking to improve the Persistence Node to give a more stable IPFS resolution and wrapper persistence experience.

Deliverables:

Persistence node (ipfs.wrappers.io gateway)

  • Multi-network support
    • Rinkeby is the only supported network right now. This task aims to provide a general way of adding support for more evm based networks.
    • It will allow the node to persist wrappers registered on Mainnet and Ropsten (as well as any future network).
  • Better indexing (No more web sockets)
    • The way ENS registrations are fetched right now is through the use of web sockets. Unfortunately, this has proven to be unreliable in production.
    • A loop will be implemented that consistently queries for new events and records which blocks it has processed.
    • This will allow us to bypass both web socket and potential node.js single-threaded issues.
  • Background retries for unresponsive wrappers
    • When the node tries to fetch a wrapper from IPFS, and the wrapper is unresponsive, it adds it to a list of unresponsive wrappers. These wrappers can only be processed again (pinning can be retried), if you stop the node and execute a separate command.
    • The objective of this task is to move that process into the background so that it can be retried over and over again.
  • More robust retries for unresponsive wrappers
    • This task extends the previous one and focuses on developing a more sophisticated retrying method. Instead of blindly retrying over and over, a configuration file will determine the number and frequency of retries and whether or not a wrapper should be considered lost if it reaches a certain number of failed retry attempts.
  • New API endpoint to show a list of files in an IPFS directory
    • There is, currently, no convenient way to check if the gateway can resolve a wrapper. The only option is to try to download a file by going to “/api/v0/cat?arg={IPFS_HASH}/{FILE_NAME}”. The objective of this task is to implement a similar endpoint to the ipfs.io/ipfs/{IPFS_HASH} endpoint.
    • This endpoint will list the files in the directory of the wrapper. For every file, a link will be shown to the endpoint for that file.
  • Separate IPFS node option
    • The wrap persistence node has an integrated IPFS node. This is useful when the machine running it does not have an existing IPFS client installed. However, this also makes the two nodes dependent, which has already resulted in some issues.
    • This task will separate the two nodes, and while it will still be possible to run the IPFS node as part of the wrap persistence node, it will also allow for an external IPFS node URI to be specified.
  • Separate CLI
    • There are multiple commands (like “past” which processes past blocks and “unresponsive” which processes wrappers from the unresponsive list) that can only be executed while the node is offline.
    • The goal of this task is to separate the daemon from the CLI and make those two be able to function separately
  • Shallow dependency pinning
    • Only IPFS URIs registered with ENS are currently pinned. This means that if a wrapper has a dependency on another wrapper hosted only on IPFS, that wrapper will not be pinned.
    • Supporting recursive pinning of dependencies would increase the likelihood that the persistence node gets exploited by a malicious attacker
    • The short term solution is only to pin immediate IPFS dependencies (not dependencies of dependencies)
    • ENS dependencies will be ignored. It is assumed they are covered with the following issue: Multi-network support
    • Additionally, there should be a limit to the number of immediate dependencies that get pinned to reduce the chance of exploitation. Ten seems like a good short-term number.
  • New API endpoint to show a list of pinned wrappers
    • There is, currently, no convenient way to check if the nodes behind the gateway are pinning your wrappers.
    • This endpoint will list the wrappers that are being pinned by the persistence node.

Potential future tasks

The following are some tasks that the team could tackle in the future. However, at the moment, they are not a priority.

  • Build the Persistence Node as a wrapper in Rust
  • Deploy a dashboard at https://wrappers.io to show pinned and added wrappers
  • Deploy to two more servers (a total of four) to take advantage of CloudFlare’s zero-downtime failover (for https://ipfs.wrappers.io)
  • Use IPFS Cluster to replicate wrappers added to IPFS of one persistence node via Wrappers CLI across all of the nodes

Logs:

February

  • Hours: 13
  • Cost: 2080 USDC

March

  • Hours: 63
  • Cost: 10080 USDC

Approved for March payment.

Note:

  • This proposal was submitted before we implemented sponsorships for projects.
  • For NeuralField’s future proposals, there should be an assigned sponsor who reviews work and approves spending.
  • Project proposals starting April need to follow as closely as possible to our new template: