SFTP always lists directory

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

SFTP always lists directory

zurkin
Hi Colleagues,

We are using pollEnrich component to download files using SFTP protocol.
The problem is that on FTP server there is about 40 thousands of files.
Everytime we try to download a file it lists entire root folder and it
takes about 40 seconds.
I know there are parameters like useList (which is not relevant for SFTP
protocol) or stepwise (which controls traversing directories)  however from
my experience and also from the SftpConsumer class it looks like we cannot
avoid listing files using SFTP protocol.

Part of the SftpConsumer class:

protected boolean doPollDirectory(...)  {
...

if (isStepwise())  {
       files = operations.listFiles();
} else {
       files = operations.listFiles(dir);
}
...
}

It looks like it always lists which is not the case in FtpConsumer class
implementation.

Is there a possibility to avoid listing using SFTP protocol and only
download concrete file?
Why it behaves differently than in case of FTP and FTPS protocol?

Thank you
Regards
Kris
--

Pozdrawiam

Krzysztof Hołdanowicz
Reply | Threaded
Open this post in threaded view
|

Re: SFTP always lists directory

Claus Ibsen-2
The useList option is not implemented/used for SFTP.

Also JCraft library does not allow doing what we useList=false
requires which is to create a ChannelSftp.LsEntry entry manually.
That class is not public, and hence we cannot do it manually.



On Tue, Oct 31, 2017 at 11:24 AM, Krzysztof Hołdanowicz
<[hidden email]> wrote:

> Hi Colleagues,
>
> We are using pollEnrich component to download files using SFTP protocol.
> The problem is that on FTP server there is about 40 thousands of files.
> Everytime we try to download a file it lists entire root folder and it
> takes about 40 seconds.
> I know there are parameters like useList (which is not relevant for SFTP
> protocol) or stepwise (which controls traversing directories)  however from
> my experience and also from the SftpConsumer class it looks like we cannot
> avoid listing files using SFTP protocol.
>
> Part of the SftpConsumer class:
>
> protected boolean doPollDirectory(...)  {
> ...
>
> if (isStepwise())  {
>        files = operations.listFiles();
> } else {
>        files = operations.listFiles(dir);
> }
> ...
> }
>
> It looks like it always lists which is not the case in FtpConsumer class
> implementation.
>
> Is there a possibility to avoid listing using SFTP protocol and only
> download concrete file?
> Why it behaves differently than in case of FTP and FTPS protocol?
>
> Thank you
> Regards
> Kris
> --
>
> Pozdrawiam
>
> Krzysztof Hołdanowicz



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: SFTP always lists directory

Claus Ibsen-2
Hi

I logged a ticket to see if we can add support for useList on SFTP
consumer as well
https://issues.apache.org/jira/browse/CAMEL-11971

Its more tricky so we may or may not be able to do this.

On Tue, Oct 31, 2017 at 11:35 AM, Claus Ibsen <[hidden email]> wrote:

> The useList option is not implemented/used for SFTP.
>
> Also JCraft library does not allow doing what we useList=false
> requires which is to create a ChannelSftp.LsEntry entry manually.
> That class is not public, and hence we cannot do it manually.
>
>
>
> On Tue, Oct 31, 2017 at 11:24 AM, Krzysztof Hołdanowicz
> <[hidden email]> wrote:
>> Hi Colleagues,
>>
>> We are using pollEnrich component to download files using SFTP protocol.
>> The problem is that on FTP server there is about 40 thousands of files.
>> Everytime we try to download a file it lists entire root folder and it
>> takes about 40 seconds.
>> I know there are parameters like useList (which is not relevant for SFTP
>> protocol) or stepwise (which controls traversing directories)  however from
>> my experience and also from the SftpConsumer class it looks like we cannot
>> avoid listing files using SFTP protocol.
>>
>> Part of the SftpConsumer class:
>>
>> protected boolean doPollDirectory(...)  {
>> ...
>>
>> if (isStepwise())  {
>>        files = operations.listFiles();
>> } else {
>>        files = operations.listFiles(dir);
>> }
>> ...
>> }
>>
>> It looks like it always lists which is not the case in FtpConsumer class
>> implementation.
>>
>> Is there a possibility to avoid listing using SFTP protocol and only
>> download concrete file?
>> Why it behaves differently than in case of FTP and FTPS protocol?
>>
>> Thank you
>> Regards
>> Kris
>> --
>>
>> Pozdrawiam
>>
>> Krzysztof Hołdanowicz
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2