- Snaplogic is a commercial software company that provides an Integration Platform as a Service (iPaaS) tools for connecting cloud data sources, SaaS applications, and on-premises business software applications.
- The Snaplogic Intelligent Integration Platform (IIP) uses AI-powered workflows to automate all stages of IT integration projects – design, development, deployment, and maintenance – whether on-premises, in the cloud, or in hybrid environments.
- With the Snaplogic platform, organizations can connect all of their enterprise systems quickly and easily to automate business processes, accelerate analytics, and drive transformation.
- It uses Snaps, our integration connectors, which allow you to easily connect any combination of SaaS and on-premises applications and data sources.
Introduction:
- SnapLogic Enterprise Integration Cloud as Integration Platform as a Service (iPaas)
- Its capability is best described by the 3A’s – Anything, Anywhere and Anytime.
- It enables you to connect anything.
- It enables you to execute this integration anytime.
- It supports execution based on a schedule and execution in real time.
FEATURES OF SNAPLOGIC:
1. Unified Data Handling:
The Snaplogic Elastic Integration Platform stands out with its ability to manage both structured and unstructured data. Snap endpoints consume hierarchical data in their native format and then pass it on to downstream Snaps in a pipeline. This process eliminates the need to flatten data into records or convert a JSON document into a string or BLOB type.
2. Advanced App and Data Integration:
The Snaplogic platform is ready to handle your data integration needs. Whether they involve streaming or batch applications.
3. Beyond Point-to-Point Integration:
Thirdly, Snaplogic goes beyond the limitations of point-to-point cloud integration. Snaplogic platform uses a hub and spoke integration architecture. This approach allows customers to integrate complex apps both in the cloud and on-premises incrementally and cost-effectively, without manual coding or re-work.
4. Hybrid Integration Use Cases:
As SaaS adoption grows and gains broader acceptance in IT organizations, enterprises need modern integration processes. They need solutions that cater to different scenarios. These scenarios include cloud-to-cloud, cloud-to-ground, ground-to-ground, and hybrid (cloud-to-cloud and ground).
5. Multi-Tenant, Software-Defined Cloud Service:
The Snaplogic Elastic Integration Platform builds on the principles of Software-Defined Networking (SDN). The “control plane” decides where and how data is processed based on user configuration and preferences. Meanwhile, the “data plane” (aka the Snaplex) processes the data as per instructions from the control plane.
SnapLogic supports all types of integration:
- Cloud to Cloud
- Cloud to Ground
- Ground to Ground
- Hybrid
- It also allows you to run these integrations anywhere.
- These allow you to move your application closer to where your data resides, thereby making integration more efficient by avoiding large amount of data across the network.
SnapLogic consists of three primary components:
- Platform
- Snaplexes
- Snaps
- Platform is a cloud-based application which is delivered as a multi-tenant cloud service.
- These applications allow you to design, monitor and execute applications.
- Snaplexes are data processing engines where pipelines are executed.
- It is available as a cloud plex which SnapLogic managers pour you, ground plex where you manage within your firewall or private cloud and a Heder plex which is a specific type of ground plex used in Hadoop environment.
- Snaps are computational units which enable you to perform data operations and connect to end points.
- SnapLogic provides more than 400 snaps.
PLATFORM:
Designer:
- The Snaplogic Designer is the user interface that enables you to develop pipelines.
- The Designer comprises the Canvas, the Snap Catalog or Asset Palette, and the toolbars.
- A Designer Canvas tab opens by default for every open Pipeline.
- It provides integration assistant for developing the pipelines.
- It provides data preview.
- We can test pipelines with manuals execution.
Dashboard:
It comprises of:
- Health wall: monitors pipeline and snaplex health.
- Pipeline wall: view pipeline execution statistics and details.
- Snaplex wall: view snaplex statistics.
- Insight wall: view time series data and logs.
- Project spaces and projects enable you to organize your assets. They also help to export and import all related artifacts.
- A project space consists of one or more projects.
- A project is used to store all related files, accounts, tasks, pipelines, snap packs and snaplexes.
- Snaplexes are the data processing engine of the snap logic platform.
- It is made up of nodes. Nodes are containers or virtual machines or ‘real’ hardware-based servers.
- Minimum of two nodes required per a snaplex for production use.
- A snaplexis available to one organization.
- It is horizontally and vertically scalable. To add more capacity, add more nodes or increase the size of the node.
- you can deploy one or many Snaplexes as required to run pipelines and process data.
- A snaplex can be a cloudplex or groundplex .
- For cloudplexes snaplogic will install and maintain the snaplex nodes.
- For groundplexes customers need to install the snaplex software on nodes in the customer data center.
Cloudplex:
- All cloudplex instances run inside the snap logic IIP.
- A Cloudplex is ideal if you require integrations that orchestrate across cloud applications (such as Salesforce, ServiceNow, and Workday) with no on-premises connections that do not require any software to run behind a firewall.
- Use SnapLogic Managerand Dashboardto administer and monitor your Cloudplex.
Advantages:
- No infrastructure overhead.
- Flexibility in deployment.
- Minimal administrative tasks required.
Disadvantages:
- No support for customization.
- Only endpoints reachable over the public internet are accessible from Cloudplexes.
Groundplex:
- If you need on-premises connectivity (such as SAP, Oracle, or Microsoft Dynamics AX) then you require a Groundplex that runs behind the firewall.
- Although Groundplex nodes run on private or virtual private data centers, Groundplex instances are managed remotely by the SnapLogic Platform’s control plane.
Advantages:
- Customized configurations supported
- Choice of security implementation and network communication
- Flexibility in choice of node resources
- Can access private endpoints which are not reachable over the public internet.
- Allows for data processing closer to the source/target endpoints which come with performance and security benefits.
Disadvantages:
- Requires administrative oversight
- Setup requires planning.
High-Level Architecture:
- The Data plan is where all the data transfers and data processing take place.
- Data plans consist of data sources and snaplexes.
- The control plan consists of the platform and snaps.
- The control plan only deals with meta data and doesn’t interact with any data.
- The meta data is sent to snaplexes for processing, snaplexes connect to end points, retrieve data and process it.
- After the process is completed snaplexes write data to end points.
- Snaplexes only process data and don’t store or cash it.
ADVANTAGES:
- SnapLogic is a unified platform for data and application integration.
- It can connect to both ligase and modern applications.
- SnapLogic provides self-service capability using simple design and a platform approach.
- SnapLogic uses modern architecture; it doesn’t use any ligase components that might prevent data and application integration services from running at clous speed.
- SnapLogic provides snaps for a vast range of applications.
- SnapLogic also enables you to create custom snaps.
- Unlike point-to-point integration tools, SnapLogic enables you to use multiple end points in a single workflow.
- SnapLogic also provides you with the iris to expertise the development process.
Iris and the Integration Assistant:
- SnapLogic iris is the industry’s first self-driving technology that applies ai to enterprise integration.
- Iris uses advanced algorithms from millions of meta data elements and billions of data flows.
- Iris provides integration assistant – which is the recommendation engine.
- It recommends snaps that are most likely to be used next.
KEY TERMS AND CONCEPTS:
- Snaps
- Snap is a computational unit that performs a single data operation.
- Snaps only process data and don’t store it.
- Snaps increase reusability by containerizing functionality.
- There are 6 types of snaps: read, parse, transform, flow, format and write.
- Read snaps – read data from end points including enterprise applications, analytics platforms and databases.
- Parse snaps – convert data from binary format to document format for further processing.
- Transform snaps – enables you to transform or manipulate data.
- Flow snaps – enables to route data based on certain configural conditions.
- Format snaps – convert data from document format to binary format.
- Write snaps – write or send data to various end points.
- Snap Packs
- A logical grouping of snaps based on functionality or endpoints.
- Each snap within a snap pack provides a subset of the overall functionality.
- There are 3 types of snap packs: core snap packs, premium snap packs, and custom snap packs.
- Core snap packs – enable you to perform data operation and data transformation connected to commonly used end points. These are available by default.
- Premium snap packs – enable you to interact with various data sources and applications. There is various sub categorized based on the type of end point they are connected to. These various subs categorized include analytics, data, enterprise and social snap packs. We need to purchase premium snap packs based on our requirements.
- Custom snap packs – There might be cases where we might want to connect to end points which are beyond the scope of existing snap packs or perform custom data transformation. Custom snap packs built using SnapLogic SDK and SnapLogic Maven Archetype.
- Pipelines
- A pipeline is an integration workflow which is created by connecting a set of snaps.
- A pipeline describes the workstation of the data flow between the end points.
- Patterns
- It is a reusable template that is used to create pipelines
- Tasks
- Tasks enable you to execute pipelines.
- There are 3 types of tasks: triggered, scheduled and ultra tasks.
- Triggered Tasks – generate URLs that can be used to execute a pipeline.
- Scheduled Tasks – execute a pipeline based on the specific schedule.
- Ultra Tasks – generate URL that can be used to execute an ultra-pipeline and provides low latency processing.
- Difference between triggered and ultra tasks: Ultra task is a pipeline execution mode which provides low latency processing where multiple instances of the pipeline are always running.
- Note that pipelines must meet certain criteria before they can be executed using ultra tasks.
- Pipelines which meet all the required criteria are called ultra pipelines.
- Snaplexes
- These are data processing engines where pipelines are executed.
- Snaplexes process data and stream data between applications, databases, files, social and big data sources.
- Snaplexes receive pipeline meta data from the SnapLogic control plan and connect with various end point to read and write data
- There are 2 types of snaplexes: cloudplexes and groundplexes.
- Cloudplexes – runs on the SnapLogic integration cloud,.Cloudplexes administrated and operated by SnapLogic.
- Groundplexes – runs on your server behind the firewall. Groundplexes must be administrated and operated by our organization administrators.
- Each snap plex is made of one or more nodes.
- Nodes
- A node is a host or server, running windows or Linux, with the SnapLogic application installed.
- The SnapLogic application runs as a JVM.
- If you are using cloudplexes you can reach your SnapLogic administrator to increase the processing capacity in case if you are using groundplexes you can scale out by adding more nodes or machines.
- Organizations(orgs)
- SnapLogic uses each tenant as an organization.
- Tenants used for different stages of development.
- Most customers use 3 organizations: development, testing and production.
- Subscription Features
- Organization level functions that enabled for your SnapLogic enterprise integration cloud instance.
- Features include ultra tasks and enhanced account encryption.
- Ultra tasks – provides low-latency processing.
- Enhanced account encryption –
- Provides additional security for account information.
- Encrypt account information stored on SnapLogic enterprise integration cloud using public key.
- Decrypts the account information on groundplexes using private key.
- Supports key rotation.
EXPRESSION LANGUAGES:
- Enables you to write expressions to transform and manipulate data.
- Commonly used in snaps such as filter, router and mapper.
- Based on JavaScript with some exceptions.
- Uses parameters and fields to interact with data.
PARAMETERS:
- Enables you to dynamically pass inputs when invoking pipelines.
- Variables whose values remain unchanged during the pipeline’s execution.
- Commonly used scenarios – parent-child pipeline construct and triggered pipelines.
- A parameter is a key-value pair. Key is the unique identifier of some data, Value.
- Keys must contain only alpha-numeric characters.
To learn more:
TASKS
- Tasks are how pipelines become operational.
- Tasks enable you to execute pipelines.
- As a general example, consider the Pipeline designer as a developer, typically with the background of an IT specialist, who builds Pipelines that fit the data processing needs of an organization. After you design a Pipeline, you then need to run it on a schedule or after an event occurs. Hence, Tasks provide an easy way to accomplish the productization of Pipelines.
- Tasks enable you to execute pipelines by scheduling or triggering a URL.
- Choose this option if you need to execute pipelines by triggering the pipeline by using http call.
- A Triggered Task also allows passing data into and retrieving data from a Pipeline.
- Scheduled Task:
- Choose this option if you want to execute pipeline for a certain time or schedule.
- Ultra Task:
- Generates a URL for executing the ultra-pipelines and provide low latency processing.
- Low latency means the time that it takes to travel from source to destination.
- Choose this option for either specialized, low-latency jobs that need to process documents continuously or for Pipelines based on the always-on-design.
ULTRA PIPELINES:
- Ultra Pipelines provide the speed and scalability to run the most important integrations that require high availability, high throughput, and persistent execution.
- This feature ensures that data arrives at its destination in an instant, regardless of the volume of data being processed, the variety of endpoints involved, or the complexity of the integration.
- Ultra Pipelines are executed as Ultra Tasks, which enable the Pipelines to consume documents continuously from external sources (Pipeline executions that are always-on), including sources that are not compatible with Triggered Tasks or those that require low-latency processing.
- An Ultra Task can manage and load-balance multiple Ultra Pipeline instances in multiple Snaplex nodes to process multiple documents simultaneously.
- The Ultra Task monitor ensures that the required number of instances are running and restarts the Task when an instance fails, or a node restart is required.
- The Ultra Task pulls requests from the Feed Master’s queue and assigns them to its associated Ultra Pipeline instances.
DIFFERENCE BETWEEN TRIGGERED TASKS AND ULTRA TASK:
- Ultra task is a constantly running pipeline listening for the new documents streaming in.
- By the time a document is sent to an Ultra Task, the underlying Pipeline is already prepared and can start processing the document instantly.
- Depending on a variety of criteria (Pipeline size, accounts, and types of Snaps used in the Pipeline), the preparation stage can take time, which makes the Ultra Task usage beneficial when the expected response time is a matter of sub-seconds.
- Since Ultra Pipelines are always running, they can be used to process documents continually from external sources like message queues. Also, data passed into an Ultra pipeline is more reliably processed.
- In terms of Pipeline design, the Ultra Task is more restricting when compared to Triggered Tasks because of the number of unsupported Snaps and restrictions around the input and output Snaps and Pipeline parameters
- In addition, the Snaplex on which the Ultra Task runs must have a Feed Master.
——————————————————————————————————————–
SUMMARY:
1. Integration connects different systems using a source and target system.
2. A source system provides data, while a target system receives data.
3. An integration tool, such as SnapLogic, is needed to connect the source and target systems.
4. SnapLogic is an iPaaS (Integration Platform as a Service) tool.
5. SnapLogic combines platform as a service and integration.
6. As a cloud-based tool, SnapLogic can be used to connect different systems.
7. SnapLogic is defined by the Three A’s: anything, anywhere, and anytime.
8. SnapLogic provides a platform for developers to use snaps and integrate systems.
SnapLogic is an Integration Platform as a Service (iPaaS) tool that allows for the connection of various source and target systems using business rules. A source system is where data is read from, while a target system is where data is sent to. SnapLogic provides a platform for developers to use the snaps provided by SnapLogic to integrate different systems. SnapLogic is a cloud-based tool that can be used to connect to any system, anywhere, and anytime. SnapLogic is defined by the Three A’s – Anything, Anywhere, and Anytime. The tool can integrate any system, including files, APIs, and applications. SnapLogic is an iPaaS tool that combines the features of a platform as a service (PaaS) and integration (I) to provide a platform for developers to integrate different systems seamlessly. The tool’s main objective is to provide a platform that simplifies the integration process, allowing developers to focus on developing custom solutions for their clients. The platform is easy to use and provides developers with the ability to integrate various systems without the need for extensive coding. SnapLogic is a tool that simplifies the integration process and provides developers with a platform to develop custom solutions for their clients.