FTP component
The FTP component is able to connect to FTP servers (FTP, SFTP, FTPS) to write or read files. It can act either as an outbound (writing files to FTP) or inbound component (reading files from FTP).
When it is not the first component in a flow it acts as an outbound component (writing files to FTP) by default.
Basic configuration
The FTP component has the following basic configuration options:
Force inbound
This option is only available when the FTP component is not the first component in a flow. It overrules the default outbound mode (writing files) and forces the component into inbound mode (reading files).
Every time the FTP component polls the server it will create a list of all the available files. It will pick one to process, adhering to the Order by setting. This will happen as fast as the files can be downloaded until the list has been completed. Each file is treated as a seperate message that continues to follow your flow.
The body of the message before the FTP component will be overridden with the content of the consumed file. Headers with information about the file will be set, for instance filename and filesize. Refer to the Camel File - Message headers documentation for an extensive explanation.
The FTP component will wait the specified Delay time before polling the server again after reading (multiple) files. Files that are added while finishing the previous list won't be processed until this next poll of the server.
Protocol
Specify the protocol of the server.
Options
SFTP(default)FTPSFTP
- Authentication to SFTP servers can be provided by specifying a password and/or an SSH private key file.
- FTPS supports FTP over TLS up to version 1.2.
Host
Specify the host (URL or IP address) of the FTP server, e.g. ftp.example.org or 149.210.209.32.
Port
Specify the FTP port if the server uses non-default ports. When left empty the defaults are FTP: 21, SFTP: 22.
Username
The username to log in on the FTP server.
Password
The password to log in on the FTP server.
Private key file
Either specify the location of an SSH private key file (e.g. uploaded with the File component) or directly upload a file using the upload button on the right.
- Only available when the Protocol is set to
SFTP. - The private key file needs to be of the format
OpenSSH. - The uploaded key is identical for the Test and Production environment.
Path
The path to the target directory. In outbound mode this is the directory where the file(s) are written to. In inbound mode this is the directory where files are being read from.
Don't include a / at the end of the path, the component is not able to write files to the server when you do.
File name
Only pick up files with this filename in inbound mode or set the file name when outbound. You can't use wildcards in this field.
File type
Specify the file(s) type.
Options
Text(default)Binary
If you set the File Type to binary when working with large text files Dovetail doesn't have to read all the file's content into memory to validate the charset. It increases performance and can prevent errors parsing the file later in the flow.
Charset
Specify the encoding of the file.
When writing large text files such as XML or JSON, you may choose to leave this option empty. The file will be processed as binary and its content does not have to be read into memory to validate the charset. This greatly improves performance and can prevent errors when parsing the file later.
File exist
Specify what to do if a file with the same name already exists.
Options
Override(default)AppendIgnoreFail
Only available when the component is in outbound mode.
Delay
The amount of time (in milliseconds) to wait before the next poll of the file/directory.
- Default:
60000(60 seconds) - Minimum:
1000(1 second)
Only available when the component is set to inbound mode.
Advanced configuration
The FTP component has the following advanced configuration options:
Passive mode
Specify whether FTP should operate in active or passive mode.
Options
Yes(default)No
Disconnect
Whether or not to disconnect from the FTP server right after use. It will reconnect when the component is used again.
Options
Yes(default)No
Only available when the component is set to inbound mode.
Auto create
Automatically create missing directories in the file path.
Options
Yes(default)No
Only available when the component is set to outbound mode.
Order by
Specify the order in which files are read from the server.
Options
Default(no guaranteed order)File name (ascending)(A-Z)File name (descending(Z-A)Modified date (ascending)(old-new)Modified date (descending)(new-old)
Only available when the component is set to inbound mode.
Maximum batch size
Maximum number of files processed on each poll. When left empty (default) all files to be read will be processed at once, as quickly as possible.
- If the batch functionality is not working as expected try setting Passive mode to
Yes. - Only available when the component is set to inbound mode.
Recursive
Specifies if only files in the specified path should be read (No) or if files in all subdirectories should be included (Yes).
Options
YesNo(default)
If this option is set to Yes and you want to move and/or move failed files into a subdirectory after processing, make sure that these folders are hidden (prepended with a .). Hidden folders are excluded from polling and you don't end up with an infinite loop.
Read more about to how set up Recursion and Excluding directories
Only available when the component is set to inbound mode.
Delete file
Specifies wether files are deleted after processing.
Options
YesNo(default)
- When set to
Yes, the Move to option is ignored and hidden. - Only available when the component is set to inbound mode.
Move to
The relative path of the directory where files are moved to after processing. The default is set to .dovetail.
Use ../[folder_name] or ../../[folder_name] to move up folders.
Specify a path that starts without a . (dot) if your FTP server doesn't allow the creation of 'hidden' directories or files.
- Only used and shown when Delete file is set to
No. - Only available when the component is set to inbound mode.
Because the FTP component polls the Move to directory before moving a file, having a large number of files in this directory may create a performance bottleneck in file processing. Consider removing files from this directory to improve performance.
Move failed to
The relative path to the directory where files are moved to when they fail processing. The default is set to .error.
Use ../[folder_name] or ../../[folder_name] to move up folders.
Specify a path that starts without a . (dot) if your FTP server doesn't allow the creation of 'hidden' directories or files.
Only available when the component is set to inbound mode.
Because the FTP component polls the Move failed to directory before moving a file, having a large number of files in this directory may create a performance bottleneck in file processing. Consider removing files from this directory to improve performance.
Read lock
If set to Changed, the component only consumes the files if it has exclusive read-lock on the file (i.e. the file is not in-progress or being written). It will wait until the file lock is granted.
Options
None(default)Changed
- Requires write permissions on the FTP server.
- Uses file length or file modification timestamp to detect whether the file is currently being changed or not. It will at least use 1 second to check this and there will also be some time between each check, so reading files will be slower.
- Requires a lock on the latest file until a new file has been written to the FTP server. It will not be processed until a new file has been dropped on the FTP server.
Only available when the component is set to inbound mode.
Include
Include files if the filename matches the specified regular expression (regex) pattern.
- Can't be used in combination with File name.
- Only available when the component is set to inbound mode.
- Only works for filenames, not folder structures.
Exclude
Exclude files if the filename matches the specified regular expressions (regex) pattern.
- Can't be used in combination with File name.
- Only available when the component is set to inbound mode.
- Only works for filenames, not folder structures.
A great resource to create and evaluate regex is Regex 101.
Make sure to set Flavour to Java 8 in the left column when creating regex for Dovetail.
Initial delay
The amount of time (in milliseconds) to wait before the 'polling' of files/directories starts.
Only available when the component is set to inbound mode.
Recursion and excluding directories
Imagine you have a directory structure as shown below, where 'mainfolder' is in the root of the folder structure and there's a file in both folders.
mainfolder
subfolder
We want to:
- Read files from both the 'mainfolder' and 'testfolder'.
- Have a .dovetail folder for processed files.
- Have a .error folder for failed files.
Make sure the FTP component is in inbound mode and:
- Path is set to
mainfolder. - Recursive is set to
Yes. - Move to is set to
.dovetail. - Move failed to is set to
.error.
After polling and processing the files the folder structure would look like this:
mainfolder
.dovetail
.error
subfolder
.dovetail
.error
Notice:
- Both files got processed and placed in seperate .dovetail folders, relative to the folder the files were read from.
- If no files are added the next trigger to poll the FTP server results in no action because the hidden .dovetail and .error folders are ignored.
- If there are files that fail to process they would be placed in their respective .error folder.
This only works if you move (failed) files into hidden folders. Hidden folders are automatically excluded when polling the FTP server.
Reading from folders with a lot of files
When the FTP component is in inbound mode it can't pick up and process multiple files at the same time. Every time the FTP component pols the server it will create a list of all the available files and pick one to process, adhering to the Order by setting. It does this as quickly as possible, but consuming a folder with large amounts of files can still take more time than expected.
You can improve performance by dividing files over multiple subfolders and use different FTP components to 'poll' each subfolder seperately.
Order by and Maximum batch size can be combined to achieve specific behaviours such as FIFO. For example when you set Order by to Modified date (descending) and Maximum batch size to 1.