Prerequisite

• A Braincube IoT Server
• The application ADLS install (see appsmanager to install new application)
• An Azure DataLake Storage instance

The nodes settings

Once the application ADLS installed you will find new nodes in your node palette:

• ADLS list
• ADLS read
• ADLS connection (configuration node, not in left pane)

ADLS Connection

As any Braincube connector you have to create a connection that is reusable in any node of this type. You can create a new connection by selecting "Add new adls" and click on the edit button next to it:

For ADLS here is what is needed:

• Node name: give a name to your connection so you can find it easily
• Primary endpoint (mandatory): This is the URL of your Azure Blob FileSystem (ABFS) provide by Azure.
• Authentication method: Chose between three authentication method:
  • Default azure credentials: only usable if IoT Server and ADLS are in the same tenant. Client ID should be given by the environment variable but it can also be overwritten in the node settings
  • Secret: Tenant and client ID must be specified to connect to your ADLS and a secret generated from the platform should be specify
  • Shared keys: An access key generated in your Azure Storage account

Once you set up this you can click on Add, Done, Deploy to save the settings and then come back to this node configuration.

ADLS List

The ADLS-list node is designed to help your explore your Azure storage and its tree map.
You can easily drag and drop this node from the palette to your dataflow and start working with it. The first thing to do is to set up the node. (see above to create a new connection if none exists)

To list in ADLS node you will chose:

• List Type: Do you want to list directory, files or a filesystem.
• Filesystem name: Target the filesystem for your research
• Subpath research: Specifiy the folder you want to explore
• Filter regex: What regex you wan to apply to filter information.
• Output format: In which format would you like to receive the answer?
  • One message per listed element: can be usefull to feed a realtime data flow
  • All listed elements as an array: can be used to extract a dataset that should be sent somewhere else for storage.

ADLS Read

The ADLS-read node is designed to help you read your Azure Storage file content.
you can easily drag and drop this node from the palette to your dataflow and start working with it. As usual, once the connection chosen from the list you created previously (see above to create a new one), you can set up the extraction you want to perform.

To read a file you will chose:

• Filesystem name: Path of the filesystem storing the file you are looking for
• Filename: Full filename of the files you want to read
• Destination: filesystem will write the file on the IoT Server Storage Manager, under the folder BraincubeAppsData/@Braincube-adls/{YOUR_UUID_NODE}/{YOUR_FILENAME} or Buffer to get the content directly into your dataflow.

Example

You can find an example of how we implement a list & read logic below:

This flow is composed of :

Inject: this node trigger the extraction, we chose to set up this node in "interval mode" to trigger the extraction automatically every XX seconds

ADLS - List node: this node is the one we worked on during this article, it connects to an ADLS instance and list the files in the selected path

ADLS - Read node: this node extract the content selected after the list node

CSV: transform the CSV content of the files we read into an object that can be transferred to a Braincube Cloud platform.

braincubeTransfert: this node upload data to your Braincube Environment

Features available

ADLS Braincube connector covers below features:

Your ADLS node performance will depend on your network connection between your IoT Server and your Azure Datalake. It can vary from a few Ko/s to multiple Mo/s depending on your internet speed.