Client Module

Client interfaces for making requests to resources.

Client

Bases: ABC

Abstract base class defining the interface for all clients.

This defines the contract that any client implementation must follow, whether it's an HTTP client, file client, or other protocol. The Client abstraction enables dependency injection throughout the system and allows for easy swapping of implementations for different protocols or testing.

All client implementations must provide at least the get() method to fetch resources. Specific implementations may add additional methods or parameters as needed.

get(resource) abstractmethod

Fetch a resource.

Retrieves the content associated with the provided resource.

Parameters:

Name	Type	Description	Default
resource	Resource	The Resource to fetch	required

Returns:

Type	Description
Response	A Response object containing the result

NoneClient

Bases: Client

Null object implementation of Client that returns empty responses.

This implementation serves as a placeholder when no client is needed or available. It follows the Null Object pattern to avoid null checks throughout the codebase. The NoneClient always returns empty responses without actually making any network requests.

This class is useful for: - Providing a default when no client is specified - Testing components that need a Client but shouldn't make real requests - Stubbing functionality during development

get(resource)

Return an empty response without doing any work.

Parameters:

Name	Type	Description	Default
resource	Resource	The resource that would be requested (unused)	required

Returns:

Type	Description
Response	An empty Response object with the same URL

Request dataclass

Bases: Resource

Request representation of an operation to be performed on a resource.

Request extends Resource to represent an operation that can be performed. This maintains the URL identity of the resource while providing a foundation for additional request-specific properties like headers, parameters, or operation types.

This class serves as a base class for protocol-specific request implementations, such as HttpRequest.

Attributes:

Name	Type	Description
url	Url	URL of the resource (inherited from Resource)

Response dataclass

Bases: Resource

Response representation of a resource request operation.

Response extends Resource to represent the result of a client request operation. This ensures that responses maintain the URL identity of their source resource while adding request tracking and content storage capabilities.

By inheriting from Resource, Response objects can be used anywhere a Resource is expected, which enables chaining of operations. This design provides a consistent pattern where the output of one operation can serve as the input to another.

Attributes:

Name	Type	Description
url	Url	URL of the response (inherited from Resource)
request	Request	The Request object that generated this response
content	bytes	Binary content returned by the operation (empty bytes by default)

Example

from ethicrawl.client import Response, Request from ethicrawl.core import Resource, Url req = Request(Url("https://example.com")) resp = Response(req.url, req, b"Example") resp.url == req.url # Maintains resource identity True len(resp.content) 22

__post_init__()

Validate response attributes.

Performs type checking and value validation: - Ensures content is bytes or None - Ensures request is a Request instance

Raises:

Type	Description
TypeError	If any attribute has an invalid type

Transport

Bases: ABC

Abstract base class for HTTP transport implementations.

Transport defines the interface for making HTTP requests through various backends. Concrete implementations handle the actual request logic using different libraries or mechanisms (e.g., requests, selenium, etc.).

This abstraction allows swapping transport mechanisms without changing the client interface, supporting different use cases such as basic HTTP requests or full browser automation.

user_agent property writable

Get the User-Agent string used by this transport.

Returns:

Type	Description
str	The current User-Agent string

get(request) abstractmethod

Make a GET request using the provided request object.

Parameters:

Name	Type	Description	Default
request		The request to perform	required

Returns:

Type	Description
Response	Response object containing the result

Raises:

Type	Description
NotImplementedError	If not implemented by subclass

head(request)

Make a HEAD request.

Default implementation raises NotImplementedError. See GitHub issue #18 for planned implementation of HEAD and other HTTP verbs.

Parameters:

Name	Type	Description	Default
request		The request to perform	required

Returns:

Type	Description
Response	Response object containing headers and status

Raises:

Type	Description
NotImplementedError	Default implementation raises this exception