Oyster Enclave: Wicked problems worth solving

Marlin Oyster is an open platform that allows developers to deploy custom computational tasks or services over untrusted third-party hosts. It relies on specialty hardware/software such as Intel SGX or AWS Nitro Enclaves which provide trusted execution environments (TEE) for execution of code in isolation from remaining processes running in the host. Such enclaves ensure that neither the host nor any other application running in it can snoop into data or alter the integrity of computations that occur inside the TEE.

While originally designed for small networking tasks to implement custom networking routes, developers have since pointed out various use cases that Oyster could potentially cater to if certain missing features were implemented. Easy to say, but hard to build, this article provides a glimpse into some of the open problems worth solving to expand the potential role Oyster could play in the architecture of decentralized systems.

Reproducible builds

While it is easy to verify whether an enclave is running a given enclave image, it's not easy to verify if a particular enclave image was obtained from a given Dockerfile short of reverse engineering the final image. 

The ideal way is for builds to be byte-for-byte reproducible so people can produce images themselves and compare them, but it's very hard to ensure nothing changes between builds. Base Docker images change over time and so do package registries. Dependencies might not always resolve to the same version and even if they are made to do so via pinning, the upstream repository might allow packages to update artifacts for an existing version. Several tools, including Docker itself, insert timestamps and file paths into the build process which will not be the same across builds.

Persistent storage

Machines hosting Oyster enclaves may fail due to various reasons, such as power outages or system faults. As a result, it is crucial to store any important data or state in a way that allows it to be retrieved upon reboot. This can be challenging, as the trusted hardware that Oyster relies on for isolation guarantees does not come with persistent storage. Careful software design is necessary when communicating between trusted and untrusted modules, as errors could compromise the security of operations.

There are several possible solutions for providing persistent storage to enclaves: sealing, cloud-based persistent storage and enclave migration. Further discussion on these options deserves an article of its own. 

Serverless deployment

Oyster allows DAOs to provision enclaves directly using smart contract calls without the need for any particular DAO member to manage SSH or any other authentication keys. A benefit of using Oyster could be that the cost of instances is divided amongst several users with users only being charged for the period of consumption. Apart from cost considerations, DAOs should ideally only have to focus on their core service instead of having to start-stop instances and manage devops. 

Such time-sharing and server abstraction is now commonplace in cloud deployments, for example, AWS Lambda. However, there are security and performance challenges with implementing something similar for Oyster.

Security: Oyster is designed to provide a secure execution environment for sensitive workloads, so it is important to ensure isolation between processes sharing the same resources.

Performance: TEEs are often used to run performance-critical workloads, so it is important to ensure that the system has low overhead and is able to provide high performance. A system similar to Cloudflare Workers can be implemented within the enclave. Cloudflare Workers is a serverless platform that allows developers to run JavaScript code in a highly available and scalable environment by using the V8 JavaScript engine and isolates to prevent code from accessing memory outside of the isolate.

Monitoring and auto-scaling

Related to the points above, instances crash due to various reasons and workloads should ideally be migrated automatically or scaled across instances when situations demand. Designing a decentralized system with such features is difficult as it isn’t easy to detect failures or congestion reliably and trustlessly.

Certain obvious challenges include: 

1. Cloud providers can start any number of enclaves on their own, as long as they don't need secret materials or access to confidential information, but when an enclave requires a secret key or access to confidential information, the owner must be involved in the startup process for attestation and providing secrets.
2. In addition to the online application owner's functional needs, there are security challenges to automatically deploying enclaves in the cloud. For example, an adversary who gains control of the cloud infrastructure may attempt to run multiple enclaves of a given application in order to launch forking attacks.
3. The enclave replication service must continuously monitor the number and status of deployed enclaves for a particular application to prevent such attacks.
4. Recursive problem of detecting faults in the auto-scaling solution.

Diversifying hardware and cloud vendors

Oyster is currently built on AWS Nitro Enclaves which uses proprietary techniques to provide resource isolation. However, concentration of nodes to a single cloud provider is not a good steady state for a decentralized network. GCP and Azure both provide equivalent offerings to deploy secure enclaves. Moreover, both Intel and AMD have a line of products offering hardware enclaves allowing individual node operators with home servers to join the network.

To be widely adopted, the TEE must be compatible with various hardware and software platforms. Each vendor provides a different SDK to build and deploy enclaves. Therefore, building an abstraction layer atop these different primitives requires intricate knowledge of what guarantees each platform offers and how they can be unified such that application behavior is predictable and consistent.

Open source SDK & Frameworks: Creating a single unified enclave abstraction provides developers with an API set that allows them to build once and deploy on multiple TEE providers. Solutions like the Open Enclave SDK aim to standardize the development of enclave applications across TEEs from different hardware vendors, although the current implementation is limited to Intel SGX and ARM TrustZone.

Authentic Execution framework allows developers to easily build and deploy distributed, heterogeneous applications with strong integrity and authenticity guarantees as well as partial confidentiality. The input for the framework is the source code for each module of the application and a configuration file called the deployment descriptor which defines the topology of the application, including nodes, modules, and connections. Nodes are physical or virtual hardware resources with TEE capabilities, such as an Intel SGX machine or a Sancus board. Modules are the software components of the application and connections link the output of one module to the input of another. Developers can use a simple API in the form of annotations or macros to write module code, only defining the endpoints of each module.

HTTPS support

Oyster could be a suitable substitute for DAOs or apps that wish to decentralize their backend operations out of centralized cloud accounts. Gateways for IPFS, RPCs or other such bridges from web 2 to web 3, flashbots relays, payment gateways, captchas are some obvious use cases. However, implementing HTTPS for enclaves right now is insecure as if a certificate issuer issues a certificate for the enclave, the untrusted 3rd party host providing the enclave and having the same IP address can also get a SSL certificate issued for itself which can be misused to MITM traffic.

There’s an open PR on Let’s Encrypt to add support for enclaves by allowing certificates to be issued to a specified account (in this case, the enclave specifically). However, it requires more tests before the PR is possibly approved. Another solution is to not rely on HTTPS at all and use other secure communication protocols supported by the client.

There’s been a recent update and it seems like enabling HTTPS for enclaves should now be doable using Lets Encrypt as the said PR is now available in production. Reference: https://community.letsencrypt.org/t/enabling-acme-caa-account-and-method-binding/189588

Contribute

If solving problems such as these interest you on a design or implementation level, do drop by our discord channel #enclaves-research. We would also be more than happy to learn about any unique use cases Oyster could satisfy.

Follow our official social media channels to get the latest updates as and when they come out!

Twitter | Telegram Announcements | Telegram Chat | Discord | Website