hops.distribute package

Submodules

hops.distribute.allreduce module

Utility functions to retrieve information about available services and setting up security for the Hops platform.

These utils facilitates development by hiding complexity for programs interacting with Hops services.

hops.distribute.allreduce_reservation module

class hops.distribute.allreduce_reservation.Client(server_addr)

Bases: hops.distribute.allreduce_reservation.MessageSocket

Client to register and await node reservations.

__init__(server_addr)
Args:
server_addr:a tuple of (host, port) pointing to the Server.
await_reservations()

Poll until all reservations completed, then return cluster_info.

close()

Close the client socket.

get_reservations()

Get current list of reservations.

register(reservation)

Register reservation with server.

Args:
reservation:

Returns:

request_stop()

Request server stop.

server_addr = None

address of server

sock = None

socket to server TCP connection

class hops.distribute.allreduce_reservation.MessageSocket

Bases: object

Abstract class w/ length-prefixed socket send/receive functions.

receive(sock)

Receive a message on sock Args:

sock:

Returns:

send(sock, msg)

Send msg to destination sock Args:

sock: msg:

Returns:

class hops.distribute.allreduce_reservation.Reservations(required)

Thread-safe store for node reservations.

__init__(required)
add(meta)

Add a reservation.

Args:
meta:a dictonary of metadata about a node
done()

Returns True if the required number of reservations have been fulfilled.

get()

Get the list of current reservations.

remaining()

Get a count of remaining/unfulfilled reservations.

class hops.distribute.allreduce_reservation.Server(count)

Bases: hops.distribute.allreduce_reservation.MessageSocket

Simple socket server with length prefixed pickle messages

__init__(count)

x.__init__(…) initializes x; see help(type(x)) for signature

await_reservations(sc, status={}, timeout=600)

Block until all reservations are received.

Args:
sc: status: timeout:

Returns:

done = False
reservations = None
start()

Start listener in a background thread

Returns:
address of the Server as a tuple of (host, port)
stop()

Stop the Server’s socket listener.

hops.distribute.allreduce_reservation_client module

hops.distribute.parameter_server module

Utility functions to retrieve information about available services and setting up security for the Hops platform.

These utils facilitates development by hiding complexity for programs interacting with Hops services.

hops.distribute.parameter_server_client module

hops.distribute.parameter_server_reservation module

class hops.distribute.parameter_server_reservation.Client(server_addr)

Bases: hops.distribute.parameter_server_reservation.MessageSocket

Client to register and await node reservations.

Args:
server_addr:a tuple of (host, port) pointing to the Server.
__init__(server_addr)

x.__init__(…) initializes x; see help(type(x)) for signature

await_all_workers_finished()

Poll until all reservations completed, then return cluster_info.

Returns:

await_reservations()

Poll until all reservations completed, then return cluster_info.

close()

Close the client socket.

get_reservations()

Get current list of reservations.

Returns:

register(reservation)

Register reservation with server.

Args:
reservation:

Returns:

register_worker_finished()

Register worker as finished with server.

Returns:

request_stop()

Request server stop.

server_addr = None

address of server

sock = None

socket to server TCP connection

class hops.distribute.parameter_server_reservation.MessageSocket

Bases: object

Abstract class w/ length-prefixed socket send/receive functions.

receive(sock)

Receive a message on sock

Args:
sock:

Returns:

send(sock, msg)

Send msg to destination sock.

Args:
sock: msg:

Returns:

class hops.distribute.parameter_server_reservation.Reservations(required)

Thread-safe store for node reservations.

__init__(required)
Args:
required:
add(meta)

Add a reservation.

Args:
meta:a dictonary of metadata about a node
done()

Returns True if the required number of reservations have been fulfilled.

get()

Get the list of current reservations.

remaining()

Get a count of remaining/unfulfilled reservations.

class hops.distribute.parameter_server_reservation.Server(count)

Bases: hops.distribute.parameter_server_reservation.MessageSocket

Simple socket server with length prefixed pickle messages

__init__(count)
Args:
count:
await_reservations(sc, status={}, timeout=600)

Block until all reservations are received.

Args:
sc: status: timeout:

Returns:

done = False
reservations = None
start()

Start listener in a background thread

Returns:
address of the Server as a tuple of (host, port)
stop()

Stop the Server’s socket listener.

class hops.distribute.parameter_server_reservation.WorkerFinished(required)

Thread-safe store for node reservations.

__init__(required)
Args:
required:expected number of nodes in the cluster.
add()

Add a reservation.

Args:
meta:a dictonary of metadata about a node
done()

Returns True if the required number of reservations have been fulfilled.

remaining()

Get a count of remaining/unfulfilled reservations.

Module contents