Neon is Postgres in its most advanced form: cloud-native, serverless, with separation of storage and compute. Neon is the only transactional database to cleanly separate compute and storage, making it possible for enterprise customers to bring their own S3 to use as the storage layer.

If this is of interest, get in touch, or read on for more context.

A refresher on transactional workloads

What defines a transactional workload?

• Generally deemed as mission critical and requiring a min of three nines (99.9%) availability. 
• Performance requirements are providing a consistent end-user experience at scale (low latency transactions at high concurrency).
• Security requirements tend to be rigorous and certain types of transactional workloads need to comply with SOX or equivalent. For the most part, auditing access and changes tend to be common-place. 
• Operational requirements as availability, recoverability are stringent – Recovery Point Objective (RPO) and Recovery Time Objective (RTO) matter and are typically under a predefined SLA. 

If we are to generalize we could say

1. Highly available
2. Consistent end-user experience (defined latencies for transactions) 
3. In compliance with regulations. Access maybe audited and changes be traced 
4. Predefined SLA with respect to recovery from failures

For transactional workloads, Postgres is a proven choice to serve as the underlying database -it is stable, resilient and the core functionality meets most of the requirements out of the box. 

Separation of compute and storage, defined

What does it mean to separate storage and compute in a cloud product?

1. Ability to scale storage and compute independently
2. Ability to choose a vendor for each component or have multiple vendors per component
3. All with a seamless experience from an end-user perspective

To give an example, I should be able to run compute in Azure and GCP, storage in GCP (and maybe in AWS as well) and the control plane in AWS.  

Why storage-compute separation matters for transactional workloads

Separation has always been advertised as necessary for scale-out analytical workloads that require performance with high availability. With compute on-demand, cost management would be an additional benefit.

While compute and control plane are easily separated out, reliable shared remote storage had been a challenge until the introduction of Object Storage (aka S3). S3 vastly simplified the process of accessing remote storage using a simple protocol (https) albeit with some compromises on functionality vis-a-vis block storage or NFS.

S3 from the major cloud providers have certain features out of the box 

• Five nines (99.999%) availability 
• Strong security – encryption, access control, auditing and logging
• Versioning
• Replication between S3 buckets

S3 can solve several challenges with respect to transactional workloads. It can provide: 

• High availability for the storage layer
• Strong security at the storage layer to meet regulatory requirements 
• Meet RTO and RPO requirements by eliminating need for backups. Easily enable flashback (aka time travel) and with native replication make the data available to other consumers. 
• And lastly, with versioning, it allows for snapshots, branches, time travel and instant refreshes of data all leading to higher developer productivity and reduced DBA burden.

Happy Developers lead to less bugs and faster time to delivery for features. DBAs would love to be freed of mundane tasks such as backups, restores and related. All in all – it is a win-win.

However, native Postgres does not support S3 as a storage engine.

Where Neon fits in this picture

Neon Postgres is the first transactional database to use S3 as the underlying storage layer. This allows Neon Postgres to enable separation of compute and make available the operational benefits of S3 to end-users.

Neon Postgres was inspired by AWS Aurora.  

Neon’s goal is to bring Aurora’s capabilities to the masses without the cost and complexity that comes with AWS services, and with a much better development experience. And as an end-user, you should be able to benefit from the separation of compute and storage and the flexibility to run these components anywhere feasible. Neon Postgres is designed to be Developer and DBA First.

Aurora is geared towards the market for transactional workloads that requires extreme performance and scale at a premium. Though Aurora has an independent storage engine, it does not quite allow direct and full control of the underlying storage (it is not clear as to what is the underlying storage used by Aurora as the implementation is proprietary to AWS). Aurora being from AWS, the entire stack resides on AWS.

Neon Architecture 

Here’s a basic diagram of Neon’s component architecture. A control plane layers on top, serving the role of orchestration and API.

Control Plane – Hosted by Neon

We want to make this experience as seamless as possible with our control plane.

With our control plane, it is trivial to manage 100’s of  instances and databases. You can create as many branches as needed. With continuous backups, there is no need to manually schedule backups. See Neon Backups Documentation for more.

Compute on-demand

With the ability to spin up compute on demand, it is easy to distribute the load and power down when not needed. Neon does this for you automatically. Individual compute instances can be set to scale up based on usage, and read replicas can be pointed to the same storage to isolate workloads and scale out.

Storage – You bring your buckets or we host it for you

At Neon, we believe that the end-user should be able to have full control over their data.

So we will allow you to specify the s3 bucket of choice (can be any cloud provider or even hosted locally in your datacenter). In this respect, Neon is unique. 

Summing up

Postgres has come a long way from the Berkeley days. It is now the de facto database for new applications. With separation of compute and storage, Postgres has taken a giant leap forward.  We see a bright future for Postgres and hope to be an enabler towards its continued success.

If you’re interested in the ability to bring your own S3 account or S3-compatible storage to Neon, get in touch here, and we’d love to discuss your requirements and see if Neon is a good fit for your workload