Package org.apache.nutch.protocol.ftp

Class Client

java.lang.Object
org.apache.commons.net.SocketClient
org.apache.commons.net.ftp.FTP
org.apache.nutch.protocol.ftp.Client
public class Client extends org.apache.commons.net.ftp.FTP
Client.java encapsulates functionalities necessary for nutch to get dir list and retrieve file from an FTP server. This class takes care of all low level details of interacting with an FTP server and provides a convenient higher level interface. Modified from FtpClient.java in apache commons-net. Notes by John Xing: ftp server implementations are hardly uniform and none seems to follow RFCs whole-heartedly. We have no choice, but assume common denominator as following: (1) Use stream mode for data transfer. Block mode will be better for multiple file downloading and partial file downloading. However not every ftpd has block mode support. (2) Use passive mode for data connection. So Nutch will work if we run behind firewall. (3) Data connection is opened/closed per ftp command for the reasons listed in (1). There are ftp servers out there, when partial downloading is enforced by closing data channel socket on our client side, the server side immediately closes control channel (socket). Our codes deal with such a bad behavior. (4) LIST is used to obtain remote file attributes if possible. MDTM and SIZE would be nice, but not as ubiquitously implemented as LIST. (5) Avoid using ABOR in single thread? Do not use it at all. About exceptions: Some specific exceptions are re-thrown as one of FtpException*.java In fact, each function throws FtpException*.java or pass IOException.

  • Field Summary

    Fields inherited from class org.apache.commons.net.ftp.FTP

    _commandSupport_, _controlEncoding, _controlInput_, _controlOutput_, _newReplyString, _replyCode, _replyLines, _replyString, ASCII_FILE_TYPE, BINARY_FILE_TYPE, BLOCK_TRANSFER_MODE, CARRIAGE_CONTROL_TEXT_FORMAT, COMPRESSED_TRANSFER_MODE, DEFAULT_CONTROL_ENCODING, DEFAULT_DATA_PORT, DEFAULT_PORT, EBCDIC_FILE_TYPE, FILE_STRUCTURE, LOCAL_FILE_TYPE, NON_PRINT_TEXT_FORMAT, PAGE_STRUCTURE, RECORD_STRUCTURE, REPLY_CODE_LEN, STREAM_TRANSFER_MODE, strictMultilineParsing, TELNET_TEXT_FORMAT

    Fields inherited from class org.apache.commons.net.SocketClient

    _defaultPort_, _hostname_, _input_, _output_, _serverSocketFactory_, _socket_, _socketFactory_, _timeout_, connectTimeout, NETASCII_EOL

  • Constructor Summary

    Constructors
    Constructor
    Description
    Client()
    Public default constructor

  • Method Summary

    Modifier and Type
    Method
    Description
    protected Socket
    __openPassiveDataConnection(int command, String arg)
    Open a passive data connection socket
    void
    disconnect()
    Closes the connection to the FTP server and restores connection parameters to the default values.
    String
    getSystemName()
    Fetches the system type name from the server and returns the string.
    boolean
    isRemoteVerificationEnabled()
    Return whether or not verification of the remote host participating in data connections is enabled.
    boolean
    login(String username, String password)
    Login to the FTP server using the provided username and password.
    boolean
    logout()
    Logout of the FTP server by sending the QUIT command.
    void
    retrieveFile(String path, OutputStream os, int limit)
    retrieve file for path
    void
    retrieveList(String path, List<org.apache.commons.net.ftp.FTPFile> entries, int limit, org.apache.commons.net.ftp.FTPFileEntryParser parser)
    Retrieve list reply for path
    boolean
    sendNoOp()
    Sends a NOOP command to the FTP server.
    void
    setDataTimeout(int timeout)
    Sets the timeout in milliseconds to use for data connection.
    boolean
    setFileType(int fileType)
    Sets the file type to be transferred.
    void
    setRemoteVerificationEnabled(boolean enable)
    Enable or disable verification that the remote host taking part of a data connection is the same as the host to which the control connection is attached.

    Methods inherited from class org.apache.commons.net.ftp.FTP

    __getReplyNoReport, __noop, _connectAction_, _connectAction_, abor, acct, allo, allo, allo, allo, appe, cdup, cwd, dele, eprt, epsv, feat, getCommandSupport, getControlEncoding, getReply, getReplyCode, getReplyString, getReplyStrings, help, help, isStrictMultilineParsing, isStrictReplyParsing, list, list, mdtm, mfmt, mkd, mlsd, mlsd, mlst, mlst, mode, nlst, nlst, noop, pass, pasv, port, pwd, quit, rein, rest, retr, rmd, rnfr, rnto, sendCommand, sendCommand, sendCommand, sendCommand, sendCommand, sendCommand, setControlEncoding, setStrictMultilineParsing, setStrictReplyParsing, site, size, smnt, stat, stat, stor, stou, stou, stru, syst, type, type, user

    Methods inherited from class org.apache.commons.net.SocketClient

    addProtocolCommandListener, applySocketAttributes, connect, connect, connect, connect, connect, connect, createCommandSupport, fireCommandSent, fireReplyReceived, getCharset, getCharsetName, getConnectTimeout, getDefaultPort, getDefaultTimeout, getKeepAlive, getLocalAddress, getLocalPort, getProxy, getReceiveBufferSize, getRemoteAddress, getRemotePort, getSendBufferSize, getServerSocketFactory, getSoLinger, getSoTimeout, getTcpNoDelay, isAvailable, isConnected, removeProtocolCommandListener, setCharset, setConnectTimeout, setDefaultPort, setDefaultTimeout, setKeepAlive, setProxy, setReceiveBufferSize, setSendBufferSize, setServerSocketFactory, setSocketFactory, setSoLinger, setSoTimeout, setTcpNoDelay, verifyRemote

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • Client

      public Client()
      Public default constructor

  • Method Details

    • __openPassiveDataConnection

      protected Socket __openPassiveDataConnection(int command, String arg) throws IOException, FtpExceptionCanNotHaveDataConnection
      Open a passive data connection socket
      Parameters:
      command - the FTP command to be sent to the FTP server
      arg - the argument associated with the command
      Returns:
      a passive Socket connections
      Throws:
      IOException - if there is an error entering passive mode
      FtpExceptionCanNotHaveDataConnection - can occur if there is a malformed server reply

    • setDataTimeout

      public void setDataTimeout(int timeout)
      Sets the timeout in milliseconds to use for data connection. set immediately after opening the data connection.
      Parameters:
      timeout - maximum timeout in milliseconds

    • disconnect

      public void disconnect() throws IOException
      Closes the connection to the FTP server and restores connection parameters to the default values.

      Overrides:
      disconnect in class org.apache.commons.net.ftp.FTP
      Throws:
      IOException - If an error occurs while disconnecting.

    • setRemoteVerificationEnabled

      public void setRemoteVerificationEnabled(boolean enable)
      Enable or disable verification that the remote host taking part of a data connection is the same as the host to which the control connection is attached. The default is for verification to be enabled. You may set this value at any time, whether the FTPClient is currently connected or not.

      Parameters:
      enable - True to enable verification, false to disable verification.

    • isRemoteVerificationEnabled

      public boolean isRemoteVerificationEnabled()
      Return whether or not verification of the remote host participating in data connections is enabled. The default behavior is for verification to be enabled.

      Returns:
      True if verification is enabled, false if not.

    • login

      public boolean login(String username, String password) throws IOException
      Login to the FTP server using the provided username and password.

      Parameters:
      username - The username to login under.
      password - The password to use.
      Returns:
      True if successfully completed, false if not.
      Throws:
      org.apache.commons.net.ftp.FTPConnectionClosedException - If the FTP server prematurely closes the connection as a result of the client being idle or some other reason causing the server to send FTP reply code 421. This exception may be caught either as an IOException or independently as itself.
      IOException - If an I/O error occurs while either sending a command to the server or receiving a reply from the server.

    • logout

      public boolean logout() throws IOException
      Logout of the FTP server by sending the QUIT command.

      Returns:
      True if successfully completed, false if not.
      Throws:
      org.apache.commons.net.ftp.FTPConnectionClosedException - If the FTP server prematurely closes the connection as a result of the client being idle or some other reason causing the server to send FTP reply code 421. This exception may be caught either as an IOException or independently as itself.
      IOException - If an I/O error occurs while either sending a command to the server or receiving a reply from the server.

    • retrieveList

      public void retrieveList(String path, List<org.apache.commons.net.ftp.FTPFile> entries, int limit, org.apache.commons.net.ftp.FTPFileEntryParser parser) throws IOException, FtpExceptionCanNotHaveDataConnection, FtpExceptionUnknownForcedDataClose, FtpExceptionControlClosedByForcedDataClose
      Retrieve list reply for path
      Parameters:
      path - a path on the FTP server
      entries - a initialized List of FTPFile's to populate with entries found at the path
      limit - optionally impose a download limit if this value is >= 0, otherwise no limit
      parser - a configured FTPFileEntryParser
      Throws:
      IOException - if there is a fatal I/O error, could be related to opening a passive data connection or retrieving data from the specified path
      FtpExceptionCanNotHaveDataConnection - if an error occurs whilst opening a passive data connection
      FtpExceptionUnknownForcedDataClose - if there is a bad reply from the FTP server
      FtpExceptionControlClosedByForcedDataClose - some ftp servers will close control channel if data channel socket is closed by our end before all data has been read out

    • retrieveFile

      public void retrieveFile(String path, OutputStream os, int limit) throws IOException, FtpExceptionCanNotHaveDataConnection, FtpExceptionUnknownForcedDataClose, FtpExceptionControlClosedByForcedDataClose
      retrieve file for path
      Parameters:
      path - a path on the FTP server
      os - an OutputStream to write data to
      limit - optionally impose a download limit if this value is >= 0, otherwise no limit
      Throws:
      IOException - if there is a fatal I/O error, could be related to opening a passive data connection or retrieving data from the specified path
      FtpExceptionCanNotHaveDataConnection - if an error occurs whilst opening a passive data connection
      FtpExceptionUnknownForcedDataClose - if there is a bad reply from the FTP server
      FtpExceptionControlClosedByForcedDataClose - some ftp servers will close control channel if data channel socket is closed by our end before all data has been read out

    • setFileType

      public boolean setFileType(int fileType) throws IOException
      Sets the file type to be transferred. This should be one of FTP.ASCII_FILE_TYPE , FTP.IMAGE_FILE_TYPE , etc. The file type only needs to be set when you want to change the type. After changing it, the new type stays in effect until you change it again. The default file type is FTP.ASCII_FILE_TYPE if this method is never called.

      Parameters:
      fileType - The _FILE_TYPE constant indcating the type of file.
      Returns:
      True if successfully completed, false if not.
      Throws:
      org.apache.commons.net.ftp.FTPConnectionClosedException - If the FTP server prematurely closes the connection as a result of the client being idle or some other reason causing the server to send FTP reply code 421. This exception may be caught either as an IOException or independently as itself.
      IOException - If an I/O error occurs while either sending a command to the server or receiving a reply from the server.

    • getSystemName

      public String getSystemName() throws IOException, FtpExceptionBadSystResponse
      Fetches the system type name from the server and returns the string. This value is cached for the duration of the connection after the first call to this method. In other words, only the first time that you invoke this method will it issue a SYST command to the FTP server. FTPClient will remember the value and return the cached value until a call to disconnect.

      Returns:
      The system type name obtained from the server. null if the information could not be obtained.
      Throws:
      FtpExceptionBadSystResponse - indicating bad reply of SYST command
      IOException - If an I/O error occurs while either sending a command to the server or receiving a reply from the server.

    • sendNoOp

      public boolean sendNoOp() throws IOException
      Sends a NOOP command to the FTP server. This is useful for preventing server timeouts.

      Returns:
      True if successfully completed, false if not.
      Throws:
      org.apache.commons.net.ftp.FTPConnectionClosedException - If the FTP server prematurely closes the connection as a result of the client being idle or some other reason causing the server to send FTP reply code 421. This exception may be caught either as an IOException or independently as itself.
      IOException - If an I/O error occurs while either sending a command to the server or receiving a reply from the server.