SwimOS Fundamentals

Streaming Data Applications are an emerging architectural approach that affords application functionality to streaming data. The three foundational components of Streaming Data Applications are:

Stateful Objects

A Stateful Object is a simple object representation of a business entity that lives amongst your streaming data. It is very much like a Plain Old Java Object (POJO) equipped with some additional superpowers. A Stateful Object continuously consumes updates of synced state. For each update, it computes new state that automatically gets synced to listeners without explicit action or coordination.

SwimOS’s implementation of Stateful Objects are called Web Agents. They are web-addressable, distributed objects for business entities. Like a traditional object, they marry state with behavior. Instead of requests and queries, Web Agents subscribe to updates or have them pushed directly during ingestion whenever information for the underlying entity comes in. Web Agents subscribe to all the state needed to take action for its corresponding entity. We call this full-context business logic because it takes in first-order entity state, as well as the first-order states of requisite entities, and then the second-order state generated by stream processing, and finally, all applicable derived state from other Web Agents.

Streaming APIs

With polling, load scales with how often you ask. With streaming, load scales with how often data changes. True real-time is impossible with polling because you would have to poll infinitely fast. E-Tag caching has provide a big improvement, but cannot achieve the degree of low-latency compared to streaming.

Streaming APIs move data the moment it is available and keep it moving until the data has been sent to all subscribers. Producers are decoupled from consumers because the Streaming API implementation handles the routing and hopefully enforcing access controls. Using efficient routing, a stream’s journey can satisfy multiple subscribers along the way achieving efficiency gains. Rigid point-to-point methods preclude such gains.

With an optimal, massively scalable Streaming API implementation like that of SwimOS, the Streaming API will use multiplexing to batch potentially massive concurrent updates into a single stream.

Additionally, while events are compact, they are verbose compared to differential state sync methods. A recurring event may require zero update to an entity’s state based on business logic, or it might require advancing or flipping a bit or two, or anywhere less than the event’s payload. This is because business logic can determine the impact on state by executing its own very specific business logic. Furthermore, this underscores a point that can be easy to miss, when a system syndicates events, every recipient has upon itself the requirement to process that event. An event routinely reaching 100s or 1000s or more recipients within the network will dramatically compound the toll of event processing in the system. On the other hand, if the system impels execution of business logic and only transmits the resulting state change deltas, rather than compounding the toll of event processing, the beneficial derived state that results from event processing is advantageously compounded instead. For this reason, SwimOS uses differential state sync as opposed to simple event propagation for its Streaming API implementation.

On top of the dramatic scalability increases by orders of magnitude, a Streaming API will impart performance and efficiency gains outside of the processing network by exposing web-friendly, streaming APIs to web-enabled endpoints. As a result, though User Interfaces may be outside the network, they will no longer be left outside the real-time delivery pipeline. Now, with Streaming APIs, end-to-end, uninterrupted streaming at half-ping latency can fulfill the real-time imperative mandated by these historically underserved clients in the first place.

Real-Time UIs

Traditional UIs play out like a power point presentation or time lapsed photography video. A Real-Time UI offers the 60-frame-per-second experience, constantly but judiciously updating. Nstream’s Real-time UIs are designed for scalable real-time streaming, ensuring that the browser doesn’t get overwhelmed and that the most appropriate data is prioritized with respect to what is in view and the desired level of detail.

Simply connecting the browser to an array of web sockets is a recipe for disaster when the volume of data is too large and too fast. Beneath web-developer friendly typescript APIs, SwimOS’s Real-Time UI library includes battle-tested real-time data processing components that have proven themselves at massive scale with large enterprise customers.

SwimOS Concepts

Swim unifies the traditionally disparate roles of database, message broker, job manager, and application server, into a few simple constructs: Web Agents, Lanes, Links, Recon, and WARP.

Agents, lanes, and links

Web Agents

Swim applications consist of interconnected, distributed objects, called Web Agents. Each Web Agent has URI address, like a REST endpoint. But unlike RESTful Web Services, Web Agents are stateful, and accessed via streaming APIs.

Lanes

If Web Agents are distributed objects, then lanes serve as the properties and methods of those objects. Lanes come in many flavors: value lanes, map lanes, command lanes, and join lanes, to name a few. Many lanes are internally persistent, acting like encapsulated database tables.

Distributed objects need a way to communicate. Links establishes active references to lanes of Web Agents, transparently streaming bidirectional state changes to keep all parts of an application in sync, without the overhead of queries or remote procedure calls.

Recon

Communication only works if all parties understand each other. Swim natively speaks a universal, structured data language, called Recon. A superset of JSON, XML, Protocol Buffers, and more, Recon naturally translates into many languages. Declare, definine, and utilize Web Agents and their properties using these configuration files.

WARP

The Web Agent Remote Protocol (WARP) enables bidirectional links to streaming API endpoints, called lanes, of URI-addressed distributed objects, called nodes, that run Web Agents.

Continue reading to learn more about Web Agents, Lanes, Links, and Recon. Or dive into the tutorials to learn by doing.

SwimOS Concepts
Why SwimOS?