Visual Flow

SPARKING ETL

A Low-Code Option for Building and Scheduling Spark Jobs.

Until now, you’ve relied on a single-node ETL system. Now, your data is growing faster than your current engine can manage it. Distributed ETL engines like Spark offer tremendous upside. But when faced with coding jobs and pipelines manually, you run into a dilemma:

	You don’t have time to learn coding languages
	You can’t afford to hire external programmers
	You don’t want to master an ETL tool, a build tool, and an orchestration tool
	Your data continues to multiply out of control

The difficulty, expensiveness, and inefficiency of these options make Spark ETL seem impossible.

What You Will Get

By using Visual Flow, you will gain access to Spark ETL, transforming your data reserves into actionable wisdom.

Simplify ETL with a Drag-and-Drop Interface

Forget about Python, Scala, and Java, and embrace low- to no-code ETL on an easy-to-use interface.

Under-the-Hood Build and Orchestration

Visual Flow incorporates Argo Workflows, Kubernetes and the Spark processing engine. By mastering Visual Flow, you reap the benefits of all three software products.

Scale at the Speed of Data

Chain functions into jobs, then chain jobs into pipelines to transform raw data into Business Intelligence.

Technical Features

Visual Flow is ETL, simplified. It allows users to leverage the scaling and efficiency capabilities of Spark ETL through an intuitive drag-and-drop interface.

Visual Flow includes:

• Stage palette featuring nine discrete data transformation functions
• Slack integration for user notifications
• Support for data sources, including

● Relational DBs: IBM DB2, PostgreSQL, Oracle, MySQL, MSSQL, Amazon Redshift, ClickHouse

● NoSQL DBs: Elasticsearch, Cassandra, Redis, Mongo

● Cloud Object Storages: Amazon S3, IBM Cloud Object Storage

● Others: local files, dataframe

• Ability to externally invoke Visual Flow pipelines using POST requests
• GitHub and GitLab user directory support

Our Guiding Principles

Go Open Source

Visual Flow is an open source product that is based on other open source products.

Scale Optimally

Virtually unlimited horizontal scalability is provided by Spark.

Design Once — Run Anywhere

Once designed, deploy your ETL on any Kubernetes cluster anywhere.

Design Visually

Think of your data flows instead of the programming language syntax.

HOW TO USE VISUAL FLOW

Visual Flow is a low-cost, low-code, open-source product. Its portability, flexibility, and multi-cloud capability make it a powerful tool for a range of data projects.

• Load Data from transactional sources into the data lake
• Load application logs and structured files from a cloud object storage to the data lake
• Cleanse and transform information inside the data lake
• Load data from the data lake into the data warehouse