Use the Hadoop connector to process and store big data.
Integrating your application with Hadoop lets you use these operations in an automation. It exposes 8 actions, listed below. Add the connector from the Connections area, authenticate, then select its operations in the automation builder. Capabilities: actions.
Authentication
Before you begin, make sure you have the following information:
Connection Name: Choose a meaningful name for your connection. This name helps you identify the connection within your application or integration settings.
Authentication: The connection uses basic authentication with a Hadoop user name. Enter the following details:
Host (required): the NameNode host address.
Port (required): the port on which the HDFS NameNode accepts connections with the hdfs:// scheme.
Username (required): the user name to connect to the HDFS NameNode. This is usually set as the HADOOP_USER_NAME environment variable on your HDFS server.
WebHDFS port (optional): the port on which the HDFS NameNode accepts HTTP connections.
YARN Resource Manager (optional): the hostname of the YARN Resource Manager, and its address, scheduler address, resource tracker address and admin address. Fill these in if you run jobs through YARN.
Use hostname for DataNode (optional, off by default): turn this on to connect to DataNodes by hostname instead of IP address.
Actions
Action Name | Description |
|---|---|
Append to file | Appends to a file in Hadoop |
Create directory | Creates a directory in Hadoop |
Create file | Creates and writes to a file in Hadoop |
Delete file or directory | Deletes a file or a directory in Hadoop |
Export via Sqoop | Export data from HDFS to a relational database using Apache Sqoop |
List directories | Lists directories in Hadoop |
Open file | Opens or reads a file in Hadoop |
Read parquet metadata | Read parquet file metadata in Hadoop |