Payloads

PayloadSerde

Interface to implement for creating a new compression scheme for the payload.

serialize_payload(payload_serde::PayloadSerde, weights::Vector{Float32})

Serialize weights by applying the transformation defined by the payload_serde.

deserialize_payload(payload_serde::PayloadSerde, data::Vector{UInt8}, from::String)

Deserialize data by applying the inverse transformation of the serialization process. from is the URL from which the data are coming, which may be an empty string if unused by the payload_serde.

VanillaPayloadSerde() <: PayloadSerde

Basic serialization where no compression is applied. This PayloadSerde is only a wrapper around the data serialization in order to match the interface to implement.

serialize_payload(::VanillaPayloadSerde, weights)

Serialize weights with the VanillaPayloadSerde where only basic serialization is applied.

deserialize_payload(::VanillaPayloadSerde, data, from)

Deserialize data with the VanillaPayloadSerde where only basic deserialization is applied.

QuantizedPayloadSerde{T <: Unsigned}() <: PayloadSerde

Apply quantization to the weights before serialization in order to reduce the payload size. This process is lossy, thus a loss of information is to be expected after the deserialization.

QPayload{T <: Unsigned}(data, minval, maxval)

Payload generated by the quantization compression scheme. minval and maxval are metadata fields required to efficiently quantize and dequantize.

serialize_payload(payload_serde::PayloadSerde, weights::Vector{Float32})

Serialize weights by applying the transformation defined by the payload_serde.

serialize_payload(::QuantizedPayloadSerde, weights)

Serialize weights with the QuantizedPayloadSerde where quantization is applied before serialization.

deserialize_payload(payload_serde::PayloadSerde, data::Vector{UInt8}, from::String)

Deserialize data by applying the inverse transformation of the serialization process. from is the URL from which the data are coming, which may be an empty string if unused by the payload_serde.

deserialize_payload(::QuantizedPayloadSerde, data, from)

Deserializes data with the QuantizedPayloadSerde where dequantization is applied after deserialization.

QDiffPayloadSerde{T <: Unsigned}(chunksize::Int, is_client::Bool) <: PayloadSerde

Apply both quantization and diff-deduplication to the weights before serialization in order to compress the payload. Quantization is a lossy process, thus a loss of information is to be expected after deserialization.

QDiffPayload{T <: Unsigned}(data, minval, maxval)

Payload generated by the quantization compression scheme. Data are structured in a different way than QPayload in order to diff-deduplicate the client-side payloads by chunks.

serialize_payload(payload_serde::PayloadSerde, weights::Vector{Float32})

Serialize weights by applying the transformation defined by the payload_serde.

serialize_payload(::QDiffPayloadSerde, weights)

Serialize weights with the QuantizedDedupPayloadSerde where quantization and diff-deduplication of values are applied before serialization.

deserialize_payload(payload_serde::PayloadSerde, data::Vector{UInt8}, from::String)

Deserialize data by applying the inverse transformation of the serialization process. from is the URL from which the data are coming, which may be an empty string if unused by the payload_serde.

deserialize_payload(::QuantizedDedupPayloadSerde, data, from)

Deserialize data with the QuantizedDedupPayloadSerde where regeneration of deduplicated values and dequantizatoin are applied after deserialization.

GDPayloadSerde(chunksize, fingerprint, msbsize, store_host, store_port, is_client)

Apply both quantization and GD to the weights before serialization in order to compress the payload. Quantization is a lossy process, thus a loss of information is to be expected after deserialization. Depending on the chosen fingerprint, collisions can occur, also leading to a loss of information.

GDPayload(data, minval, maxval)

Payload generated by the GD compression. Since the GD compression scheme is built on top of quantization, the same metadata are presnts, while the structure of the data is under the form of a small GDFile.

serialize_payload(payload_serde::PayloadSerde, weights::Vector{Float32})

Serialize weights by applying the transformation defined by the payload_serde.

serialize_payload(::GDPayloadSerde, weights)

Serialize weights with the GDPayloadSerde where quantization and generalized deduplication are applied before serialization.

deserialize_payload(payload_serde::PayloadSerde, data::Vector{UInt8}, from::String)

Deserialize data by applying the inverse transformation of the serialization process. from is the URL from which the data are coming, which may be an empty string if unused by the payload_serde.

deserialize_payload(GDPayloadSerde, data, from)

Deserialize data with the GDPayloadSerde where generalized deduplication and dequantization are applied before deserialization.