Distributed architectures and the cloud with Ingescape

– Ingescape platforms Part 1 –

Intro 1 – Distributed architectures and the cloud with Ingescape [EN|FR]

The Ingescape open source software library makes it possible to set up industrial and operational information systems around heterogeneous software using any programming language and any operating system. These software involving Ingescape are called agents and are modeled according to a simple but complete formalism to offer very targeted to very advanced services, without technological constraints.

Ingescape is therefore suitable for the creation of new information systems but also for the adaptation and total or partial migration of existing legacy software platforms. Ingescape’s small footprint, simplicity and self-contained structure make the transformation of existing software into Ingescape agents generally quick and easy, within hours to a few days at most. In practice, Ingescape imposes no constraints on software architecture or development techniques.

An agent modeled with Ingescape offers inputs and outputs named and typed (impulses, Booleans, integers, floating numbers, character strings or binary data) that can be linked to other agents to constitute data flows called mapping. In addition, each agent can expose services in the form of named functions, called services, accompanied by parameters that are also named and typed.

The following illustration summarizes the key notions of an agent model, the communication patterns, data types and events that agents know how to detect between themselves.

On the same computer or on computers connected in a network, as well as via the internet and the “cloud”, the agents discover each other automatically with a network configuration that does not require knowledge of the rest of the system. This discovery produces a fully decentralized network of meshed agents with little to no software coupling depending on the architecture schemes used.

This article presents the different architecture schemes offered by Ingescape, comparing them with the most advanced industrial and technological practices. Two related articles, pointed at the end of this one, then allow to explore on the one hand the concept of services (web services, RPC, etc.), and on the other hand the concept of data flows (software buses).

Same computer or same local network

Discovery on the network

Most often, an Ingescape platform is a set of software programs using the Ingescape library and connected on the same local network, i.e. jointly:

On IP addresses belonging to the same subnet,
With the same netmask and same broadcast address,
On the same port.

In a synthetic way, this is configured by the following line of code in any Ingescape agent:

This line of code allows, for a given agent, to select a network device (here “en0”), and therefore an IP address, a mask and a broadcast address – determined by the configuration of the chosen network device -, as well as a network port (here 5670) corresponding to the “common key” of the platform to be joined: all agents using the same network and the same port discover each other automatically, regardless of the language and operating system they use .

Ingescape also offers discovery through brokers which are dedicated software, responsible for connecting agents. This second, more centralized mode of discovery requires knowing at least the IP address of one of the brokers of the platform. If the brokers are turned off or an agent does not know the address of at least one of them, it will not be able to connect to the other agents. We therefore prefer to use the auto-discovery mode most often, relying solely on the network configuration of the computer hosting a given agent. It should be noted that even when using brokers, agents constitute a decentralized mesh network. Losing a broker will only prevent newcomers using that broker from connecting to other agents already present.

Decentralized meshed set

Whatever the discovery mode, Ingescape allows the creation of a meshed and decentralized network of agents: all the agents know each other and are able to communicate peer to peer. An Ingescape platform has no central point. An agent can connect or stop at will. Other agents are notified of the arrival and departure of other agents on the platform.

The meshed and decentralized aspect, coupled with an initial automatic exchange of configuration information, allows the agents to constitute a coherent whole, known to each of them. The agents are then able, in real time, dynamically, to establish data flows and service calls between them. This initial exchange of information for the creation of the mesh generally takes a few hundredths of a second with a few hundred bytes passing between each agent two by two. This allows the Ingescape platforms, on a classic 100BaseT computer network, to be able to cope without difficulty with several tens of thousands of agents. Generally, industrial and operational Ingescape platforms are made up of a few dozen agents, leaving plenty of room for increases if needed.

Automatic optimization of communication channels

For all meshed agents, information flows and service calls use very high performance peer-to-peer connections carried by ZeroMQ. These connections are able to take full advantage of the most efficient networks, with a very low communication overhead compared to the raw data to be transmitted. This allows Ingescape to offer very high communication performance in terms of throughput and latency, to meet the expectations of the most demanding systems, well beyond conventional TCP/IP connections.

When two agents are running on the same computer in two different processes/applications, Ingescape automatically uses peer-to-peer connections that are more efficient than TCP/IP. On Windows, loopback is used. On UNIX systems, these are IPC connections. The latency and throughput gains allowed by this completely transparent optimization for agent developers are very substantial and are suitable for very high frequency and very high throughput communications, sought in intensive computing or complex simulation systems.

Ingescape also makes it possible to group together sets of agents within the same process/application, whether in the same thread or in separated threads . Communication is then provided by shared memory offering even higher performance than IPC exchanges. Ingescape is of course thread-safe.

With this flexibility, the same models of mappings and exchanges of services remain applicable regardless of the location between two agents. It is possible – and usual in development, fine-tuning, diagnostics or prototyping phases – to relocate certain agents among different machines operated by several developers or maintainers, and then optimize software/hardware topology during commissioning. These urbanizations of the applications can range from an assembly in the same application to a set of machines, either physical, virtual or containerized.

Finally, for cases where every microsecond is counted with network topologies that do not allow TCP/IP to be bypassed, Ingescape allows the creation of specific communication flows, requiring additional development but greatly facilitated, relying on the same communication mechanisms as Ingescape using the very widely recognized performance of ZeroMQ.

Edge computing, distribution of computing resources and bandwidth

Unlike client/server solutions or communication systems based on brokers (Apache Kafka, RabbitMQ, MQTT, etc.) requiring static infrastructures to operate around centralization for the circulation of information, Ingescape is very suitable for dividing software systems into independent units that remain capable of communicating in an efficient, effective and controlled way on a peer-to-peer basis. As such, Ingescape greatly facilitates the architecture of distributed and heterogeneous information systems by making it possible to distribute the activities of calculation, storage/retrieval, user interactions, etc. optimally within the available hardware capabilities.

The dynamic nature of Ingescape, i.e. its ability to create and redirect information flows and services on the fly, makes it easy to deal with parallelization and edge computing issues to finely allocate – and adjust over time – information processing and storage resources, as well as the needs and necessities of information circulation on the networks. The fact that each Ingescape agent is a self-contained executable program, which can be developed in practically any language and for any operating system, including virtual machines and containers (docker, etc.), gives a very significant flexibility when compared to most advanced industrial solutions.

Developers using Ingescape benefit at any time from all the services and all the information allowing them, statically or on the fly, to adapt the topology of their platform, the computing resources, the information flows and redundancy and load balancing strategies.

Microservices and reuse

An Ingescape software agent can be seen as a well-identified features provider with a combination of input/output streams and services that can be requested. The simplicity of the model also allows the simplicity of the understanding of the features, even if they are advanced and rich. The experience of using Ingescape has shown the advantages of the Ingescape model, including in systems that were previously modeled with intrinsically more complex solutions: the translation of these third-party models into that of Ingescape was very easy by the teams involved, whether it’s web services, querying databases or specific protocols or flows.

Modeling a system in agents of complexity always adapted to the features they provide also makes it possible to maximize the reuse of these agents. It is common for Ingescape users to be able to reuse agents from one project to another because they provide well-defined features that are loosely coupled to the rest of the environment. This gain obtained over time is very significant in terms of costs and technical expertise, whether on products to be adapted or configured for customers, or for tailor-made projects in a given field.

Advanced data structures and serialization

With its input/output and services model, Ingescape already offers interesting information structuring capabilities suitable for most industrial and operational projects. It is easy to complete this structuring with on the one hand text-based formalisms (JSON, XML, etc.) and on the other hand with serialization solutions for binary data such as Protocol Buffers. Ingescape is completely open to exchanges of complex structured data in text and binary format, without creating the obligation to use them and therefore not imposing formal complexity often being a source of errors and complications when introducing evolutions of the system architecture.

In addition, the Ingescape visual editor allows display and even formal verification and validation of textual and binary third-party structured formats in order to validate the compliance of exchanges.

Resilience, redundancy and real-time system administration

It is simple on an Ingescape platform to launch several instances of the same agent, receiving and sending the same data flows and offering the same featuress. We do not describe here the capabilities of Ingescape to distribute information to perform parallel computing: this point is covered in the article Ingescape Platforms – Part 2 – Software Buses and Data Flows with Ingescape. Here, we introduce the notion of elections allowing several instances of the same agent – and more broadly any group of agents having an interest in it – to organize elections for a given role. This concept makes it possible in particular to easily set up main/backup type architectures, with one or more backups, offering hot redundancy possibilities in a very accessible way for developers.

In addition, the meshed and completely dynamic nature of an Ingescape platform allows any agent to be informed of the arrival or departure of any other agent. This therefore allows agents to adapt to the loss or arrival of another agent to react to these events, activate degraded modes, direct information flows differently, redirect accessible services, etc. It is also very easy to trigger system administration measures via agents watchdogs created specifically for a platform and capable of controlling process launches, system notifications, etc.

Remote platforms, cloud and internet – beyond local networks

Ingescape, although allowing the coordination of tens of thousands of agents within a platform, is primarily suited to linking agents on the same machine or the same local network. To produce geographically distributed architectures, the philosophy promoted by Ingescape aims to set up multiple interconnected platforms, with an edge computing approach. Ingescape then offers the possibility of linking these different platforms together to export or inject information flows or service calls from other platforms.

An Ingescape ecosystem distributed between several disjoint networks, and therefore several disjoint platforms, can be seen as groups of agents, each in a platform and offering access to these agents independently from the platform on which they are located. ‘execute.

In practice, when two platforms are linked, Ingescape makes it possible to “project” one or more agents from a source platform onto a target platform, making these agents accessible on the target platform, as if they were running natively there. A projected agent then exposes its inputs, outputs, and services to agents on the target platform.

Just as agents are meshed within a platform, it is possible to mesh more than two platforms together:

These platforms can operate on different sub-networks on the same geographical site, on different sites connected by private (VPN) or public (Internet) connections, or on hybrid platforms, local on the one hand and running in the “cloud” on the other hand, and on servers or containers hosted by third parties on separate infrastructures.

Link and merge remote platforms with Ingescape

To connect remote platforms, Ingescape offers proxies. An Ingescape proxy is a particular agent allowing to manage the connection between two platforms. It works in two modes:

A local mode allows to expose all or part of the agents of a source platform with a centralized point of contact to the outside, materialized by an IP address and a port whose access can be controlled by a firewall and/or a NAT, and be accessible by a VPN,
Symmetrically, a remote mode allows, from the target platform, to connect to a proxy in local mode to project the agent(s) exported from the source platform to the target platform.

When two proxies, one in local mode and the other in remote mode, are connected, agents exported from the source platform are exposed on the target platform as if they are executed directly on the latter. The agents of the target platform are then able to exchange information and to use the services of these agents exported from the source platform, exactly as if they were present locally. The only constraint to be taken into account is the quality of the connection between the two proxies to ensure that latency and bandwidth are adapted to the needs of exchanges between the platforms.

The Ingescape proxies allow flexible combinations:

A proxy can operate in both local and remote mode, so that two platforms can mutually project agents from one to the other. The ultimate use case is a full fusion between two remote platforms.
If bandwidth requirements or information segmentation require it, multiple proxies can be implemented to link the same source platform and the same target platform. Each proxy then consumes the bandwidth associated with the agents it relays on the connection allocated to it.
Several source platforms can be projected to the same destination platform and the same source platform can be projected to several destination platforms. It is therefore possible to link an unconstrained number of platforms together, to go as far as merging several platforms together if necessary.

The links between proxies are secure by default and can be supplemented by the use of specific certificates with asymmetric encryption. The level of security offered is completely suitable for use via the Internet or non-secure networks.

As in any geographically distributed network architecture, the use of proxies must be adapted to the required bandwidth and latency constraints. The raw performance of proxies ensures minimal overhead in terms of bandwidth and latency. The possibility of involving several proxies between two remote platforms also makes it possible to split the data streams and balance data loads to ensure a minimum quality of service when this is desirable.

Integrate web applications from web browsers or mobile applications

Beyond geographically distant platforms, some software applications cannot present themselves as “thick client” applications communicating on a local network. This is the case when:

An application runs in a web browser, effectively imposing a client/server scheme and limiting communications to HTTP requests and websockets,
An application, light, rich or heavy, in a web browser, on a tablet/telephone or on a classic computer, is located on a network different from that of the platform, including on the Internet or via a VPN, without the possibility of set up a target platform, i.e. control the target local network and its firewall.

To deal with these situations, Ingescape proxies also make it possible to provide access to a source platform for isolated agents on third-party networks. In order to allow wide compatibility of communications and easy use in web browsers (and for any other type of less constrained applications), Ingescape proxies allow the creation of a local mode alternative opening a websocket type server. This local connection offers the same control and configuration possibilities as the classic source/target connection presented in the previous section.

Client applications wishing to integrate a source platform in this way are qualified as pseudo-agents in the sense that they will be manipulated like classic agents on the source platform but connect using a client/server architecture corresponding to that of a websocket server with an address of the following format: wss://server_address:port. These pseudo-agents do not embed the Ingescape library directly but a simplified version currently available in Javascript, C++/Qt and Java, especially for Android.

The arrival and departure of pseudo-agents is detected as for classical agents. The simplified version of the Ingescape library used by pseudo-agents offers the same simplicity and quick learning capabilities as the original with a very similar programming interface. The simplification mainly concerns network control capabilities but the programming concepts around the Ingescape model and associated events are identical.

It is possible to connect large quantities (hundreds, thousands or even millions) of pseudo-agents to a source platform, but this requires the same precautions regarding sizing, bandwidth, security, etc. as for high-volume web-based client/server architectures. It is possible to launch several proxies simultaneously to provide several redundant access points to the same source platform or to achieve load balancing like LAN proxies web servers already allow. The strategies to be implemented are in fact the same as those very familiar to administrators of classic web platforms.

Security

For Ingescape platforms, security involves the following aspects:

Control and block connection attempts to a platform for unauthorized agents,
Encrypt communications between agents on a given platform,
Control, encrypt and if necessary block connections between platforms and from pseudo-agents in web and mobile applications.

The Ingescape library relies on the security layer offered by ZeroMQ. This security layer provides authentication and encryption by private and public certificates around the SASL mechanism, Curve25519 elliptic curves cryptography with 256-bit keys, via the libsodium (https:/ /libsodium.gitbook.io/doc/) library.

Authentication and encryption levels correspond to the State of the Art of 2020 technology. The security itself is based on the ability to preserve private certificates and distribute the associated public certificates. Given the industrial and distributed nature of the Ingescape platforms, we do not use a certification authority for the certificates, unlike the certificates used in HTTPS. A notable exception is however introduced for websocket connections using WSS for which it is of course possible to use certificates from recognized certification authorities.

With Ingescape, agents on the same platform must have access, each to their private certificate and all to the public certificates of other agents. Local and remote proxies must have reciprocal access to their own private certificate and to the public certificate of the other, which certificates may be different from those of the platform.

websocket connections can use the WS-Security (WSS) layer and its various possibilities according to the 1.0 standard established by OASIS-Open. Most generally with WSS, only the signature and encryption parts are used through X.509 certificates, similar to HTTPS connections using these same types of certificates.