Retroedge - System Architecture - Part 1

I’ve just released v0.1.0 of a Rust-based Backtesting engine on:

https://retroedge.io/

It is a complete rewrite of the previous Python implementation, with a focus on performance, maintainability, and future extensibility. The original implementation is documented in the a previous series.

Why bother rewriting it in Rust from Python?

#

“If it ain’t broke don’t fix it”, right? It wasn’t broken but could be better.

Python has an exceptional ecosystem for rapid prototyping and quantitative computing. However, for a latency-sensitive application where performance is a primary design constraint, Rust provides stronger control over memory, concurrency, execution costs, and native zero-cost abstractions without relying on Python-to-C FFI boundaries.

Why Rust and not C++ for a Trading application?

#

Rust offers performance comparable to C and C++ while providing stronger guarantees around memory safety and concurrency.

Although Rust’s ecosystem is younger than those of Python or C++, the language itself provides strong primitives for building the missing pieces when required.

I also considered other candidates like: Go, C#, Java, but needed a language that allowed for fine-tuned performance optimisations down the line if needed. Go and its garbage collection,

Before discussing the application architecture, it is worth looking at the overall system architecture.

Deployment Architecture

#

%%{init: {"flowchart": {"useMaxWidth": true}, "themeVariables": {"fontSize": "32px"}}}%%
flowchart LR

U[Users / Clients]

LB[Proxy Server & Load Balancer]

subgraph CI[CI & CD]
  C1["CI Server (Build & Test)"]
  C2["CD Server (Deploy)"]
end

subgraph APP[Application Servers]
  A1[App Server #1]
  A2[App Server #2]
  A3[App Sferver #N]
end

API[API]

DB[(Database)]

U --> LB

LB --> A1

LB --> A2

LB --> A3

A1 --> API

A2 --> API

A3 --> API

API --> DB

C1 --> C2
C2 -.->|Deploy| A1
C2 -.->|Deploy| A2
C2 -.->|Deploy| A3

style LB fill:#F28C28,stroke:#FFFFFF,stroke-dasharray: 5 5,color:#000000
style A2 fill:#F28C28,stroke:#FFFFFF,stroke-dasharray: 5 5,color:#000000
style A3 fill:#F28C28,stroke:#FFFFFF,stroke-dasharray: 5 5,color:#000000

The load balancer is not actually part of the architecture, but has been added to show where it would fit in. Same for the App Server #2 and #3.

Benefits

#

I’m fond of this structure because it encourages system-wide separation of concerns:

• load balancing;
• stateless application servers only serving client requests;
• stateful API responsible for business logic;
• simpler development, testing, and debuggin;
• scalability – can add more application servers if the load demands it;
• resilience – if one of the application servers is down another can pick up the slack (*);
• high availability through redundant application servers;
• security – the API is behind multiple layers

(*) this helps with fault tolerance and redundancy only partially, as it is all on the same machine. True redundancy requires deployment across multiple hosts.

Trade-offs

#

• increased operational complexity;
• more involved deployment;
• single point of failure in the load balancer.

The last point can be mitigated by making the load balancer itself redundant, but it’s similar to the resilience point: if it’s all on the same machine and it’s down for whatever reason, e.g. power outage, etc., the whole thing won’t work.

DevOps

#

The CI/CD pipeline is orthogonal to the request flow and exists solely to support development. New commits trigger the CI pipeline, which builds the project and executes the test suite. Successful builds can then be deployed through the CD pipeline.

Once the CI server green lights the changes, the CD flow can take over and deploy them to the live environment. Note: you can have Continuous Delivery without the delivery being automated. To mitigate risk I prefer to manually click the deploy button, even if the actual deployment is automated.

To container or not to container; that is the question

#

For this project I decided not to containerise the services. The deployment targets are well understood, and native deployments are simpler operationally while avoiding an additional abstraction layer. Containers can be valuable when portability, orchestration, or heterogeneous environments become requirements, but they would add complexity without solving a problem this project currently has.

This concludes the overview of the system architecture.

In the next article we will cover the application-level architecture.