Federated Learning with AWS

Create AWS Backed Federation

URL

POST /symetry/rest/{cid}/fedml/create?pid={new-project-id} [body=Encrypted Map]

This REST endpoint allows the user to create a new federation. The user who creates the federation also becomes its administrator. The body is a map that is encoded to a JSON string and then encrypted using the user secret key. This encryption is necessary in order to protect sensitive information like aws access key and aws secret key that are required to be passed.

Federation Key Map Value

Mandatory key	Required / Optional	Value Description
fed_name	Required	The name of the federation.
fed_local_type	Optional	The type of project to use. Supported values: cpu and gpu. The default value is cpu.
aws_access_key	Required	Your AWS access key.
aws_secret_key	Required	Your AWS secret key.
aws_region	Required	The region to use to allocate resource needed by the federation (SNS topic, SQS queue and subscription).
s3_bucket	Required	AWS S3 bucket to use to store message to other nodes in the federation.
aws_id	optional	AWS user id
sync_sched	Required	The sync schedule, consult FedML Terminology for more details.
fed_encrypt	Optional	Specify if sync message content should be encrypted. Default is false

Query Parameters

Parameter	Required / Optional	Description
pid	Required	Name of the new SymetryML project.
persist	Optional	whether to persist or not the project

HTTP Responses

HTTP Status Code	HTTP Status Message	Description
201	CREATED	Success. {"statusCode":"CREATED","statusString":" + SYMETRYML Created with id:r1","values":{}}
409	CONFLICT	SymetryML project already exists. {"statusCode":"CONFLICT","statusString":" + Customer [c1] already have SYMETRYML with id[r1], ","values":{}}

HTTP Response Entity

None

Sample Request/Response Fed Create

POST url="http://charm:8080/symetry/rest/c1/fedml/create?pid=nh_test_admin

Request Body:

KjxfT4/2TpJsfwd+gAG0zAnH7r+fs7IyKrlMXNoqh5CNQjoLioBW6HjXH5hRVqEYASueMI++fmMDAEOBIQD/e/3HDeA/SVojcmzywmxcMNotH9iTpboU9KIJZVq1JW3bxAv+e2COEuI+82qrQVJ4QifuAy5TxR+B5H0GjC0x4qiDK5h+TpEDnFB1Cjdv3xQyiDW8EetZNlCcriEdkTSvxI8duhdJIhFy4UYL1XzN7SavwmE/rddP+ETfeMmLJehseBOJr19xPiMR71maM2DybESQvCN1bWUj2XEvujasrcMMUK4+NEeC0pDe+gWoJhRgAZlRnOWgdwCKeBqcmqOc9Q==

Response:

{"statusCode":"CREATED","statusString":"Project Created with id:nh_test_admin","values":{}}

Federated Learning Get Encrypted Federation Info

URL

POST /symetry/rest/{cid}/fedml/{pid}/getEncrypted

Get Encrypted Body Key Map Value.

TBD new ResponseEntityKey.

Mandatory key	Required / Optional	Value Description
passwd	Required	Password to use to encrypt the federation information
fed_rest_host	Required	Hostname where admin can be reached at.

HTTP Responses

HTTP Status Code	HTTP Status Message	Description
200	OK	Success.
400	BAD REQUEST	SymetryML project does not exist. {"statusCode":"BAD_REQUEST","statusString":"Cannot Find SYMETRYML id[r5] for Customer id [c1]","values":{}

HTTP Response Entity

The response will be stored in the fedInfoEncrypted entity which is a string. That string is in turn a json encoded string encrypted with the provided password. Once decrypted you get a <string, string> map that contains the following key: (Consult Federation Key Map for details). The response token fed_req_token can only be used one time.

• sync_sched

• fed_name

• fed_secret_key

• aws_topic_arn

• aws_topic_region

• fed_req_token

Sample Request/Response

Request:
POST url="http://charm:8080/symetry/rest/c1/fedml/nh_test_admin/getEncrypted"

Request Body Before Encryption:
{"passwd":"passwd","fed_rest_host":"http://charm:8080"}

Request Body:
qvh4P1y937flTE/R6GgtrN8Weu28/jCx14zBIG5zk4CEqC4+rLL/HtVvKqUthPsSVZN8BgWP39O+MKEyOsUo8w==

Response:
{"statusCode":"OK","statusString":"OK","values":{"fedInfoEncrypted":"ZwsCbdhP2GxfOgDBwJHoy61uRMZLNg7D581Cs9zWAgjwWpK2wScvgpTexpsVyilbvGThFVsWXz57n9BXY+DA1EmI+L29H+K60b1lI4v/WLr45HHCdMEDCcBAL+F3/3NcRBl5RcfgBlC4RTV6+h/75rr21lgMGxnVXlYBoUh//JpBxfJhEZyxa1gv0WVdCXDDT/M3Aq819r+6rS89jj38SYj/Hzj7pJaEHvbxe+OniH5Z90xEortLZeVdG8HE5zzqLegKpChFYFE4l60RXtdyE/fa7uY3gM7Q/yU+8SQ3Yjkvl8B5Gs7coBdgkcKNmcgiBqG4YrQSg8SCuX6RsKs4B4qV61w+AZkmt9+ipp049Hpxwu0myZ2agomcAMKdZHDv"}}

Decrypted Message Example:

Once decrypted the Federation info will contain information like in the following example. This is basically a map with a few key/ value pair. These key value pairs must be used when joining a federation. See section for details.

Decrypted JSON:
{
    "sync_sched":"m1",
    "fed_rest_url":"http://charm:8080/symetry/rest/c1/fedml/nh_test_admin/createPolicies",
    "fed_name":"nh_test_doc",
    "fed_secret_key":"t6Zat/ZAJyYBtmVXATUDCQ\u003d\u003d",
    "aws_topic_arn":"arn:aws:sns:us-east-1:428117700962:nh_test_doc",
    "aws_topic_region":"US_EAST_1",
    "fed_req_token":"FEDTOKEN38f45fe16545e80c",
}

Join an AWS Backed Federation

URL

POST /symetry/rest/{cid}/fedml/createJoin?pid={new-project-id} [body=Encrypted Map]

This REST endpoint allows the user to join an existing federation. The body is a map that is encoded to a JSON string and then encrypted using the user secret key. This encryption is necessary in order to protect sensitive information like aws access key and aws secret key that are required to be passed as part of the body.

Federation Join Request Body

The request body is composed of 2 types of key/value pairs:

• Those that come from the Admin in an encrypted form. See Decrypted Message example.

  • Mandatory Keys

• Those that relates to the new node to be created.

  • New Node Mandatory Keys

Mandatory Key / Value Pairs from Federation Admin

The following key value will be received as part of the password encrypted json message from the federation admin. They need to be added to the query body without modification.

Mandatory key	Value Description
sync_sched	The sync schedule, consult FedML Terminology for more details.
fed_secret_key	The secret key used to encrypt info exchanged between nodes in a federation.
aws_topic_arn	The Amazon Resource Name of the assigned SNS topic.
fed_name	The federation id.
topic_region	The AWS region of the assigned SNS topic.
fed_rest_url	The URL of the REST service that will enable peers to join the federation.
fed_req_token	The token used to verify that a join request has not expired.

Request Body Key / Value Pairs

Additional key / value pairs are needed in the request body

Mandatory key	Required / Optional	Value Description
fed_local_type	Optional	The type of project to use. Supported values: cpu and gpu. The default value is cpu.
aws_access_key	Required
aws_secret_key	Required
aws_region	Required	AWS Region to use.
aws_id	Required	AWS ID. This is needed so that the federation admin can give a new node publish and subscribe rights.
s3_bucket	Required	AWS S3 bucket. This is used to store messages for other nodes in the federation.

Query Parameters

Parameter	Required / Optional	Description
persist	Optional	Whether or not to persist the project. Defaults to true.
pid	Optional	Project name

HTTP Responses

HTTP Status Code	HTTP Status Message	Description
201	CREATED	Success. {"statusCode":"CREATED","statusString":" + Fed Project created with join","values":{}}
409	CONFLICT	SymetryML project already exists. {"statusCode":"CONFLICT","statusString":" + Customer [c1] already have SYMETRYML with id[r1], ","values":{}}

HTTP Response Entity

None

Sample Request/Response Fed Create Join

POST url="http://charm:8080/symetry/rest/c1/fedml/createJoin?pid=node1"

Request Body:
KjxfT4/2TpJsfwd+gAG0zAnH7r+fs7IyKrlMXNoqh5CNQjoLioBW6HjXH5hRVqEYASueMI++fmMDAEOBIQD/e/3HDeA/SVojcmzywmxcMNotH9iTpboU9KIJZVq1JW3bxAv+e2COEuI+82qrQVJ4QifuAy5TxR+B5H0GjC0x4qiDK5h+TpEDnFB1Cjdv3xQycXSud7eRwy72gtY+wi8Rey3Ku1BerIRbhRVTBaHicb2duiZTZivdx1VJKA7BbH2s...


Response:
{"statusCode":"CREATED","statusString":"Fed Project created with join","values":{}}

Get AWS Info

This rest endpoint is provided so that one can inspect the AWS resources utilized by the federation.

URL

POST /symetry/rest/{cid}/fedml/{pid}/awsinfo [body=Encrypted Map]

Request Body Map Keys

Key	Description
aws_access_key	Required
aws_secret_key	Required
aws_region	Optional, if not passed, result will be returned for all aws region

HTTP Responses

HTTP Status Code	HTTP Status Message	Description
200	OK	Success.

HTTP Response Entity

HTTP Response Entity	Example
AwsInfo	Please consult the AwsInfo for an example. The response will be an encrypted JSON string. The entity key will be awsInfoEncrypted.

Sample Request/Response

Request
POST url="http://charm:8080/symetry/rest/c1/fedml/awsinfo"

Response Header:
Location: http://charm:8080/symetry/rest/c1/jobs/4

Response:
{"statusCode":"ACCEPTED","statusString":"Job Created","values":{}}

Job Request:
GET url="http://charm:8080/symetry/rest/c1/jobs/4