S3control client create job. No json format for Manifest only CSV

[damolaobaleke]

Hi,

When passing in the parameters for the create_job(). In the Request syntax. The format attribute in the manifest block or object is only capable of taking csv and not the actual manifest.json why ?

So 'Format': 'S3BatchOperations_CSV_20180820'|'S3InventoryReport_CSV_20161130', but no json

 Manifest={
            'Spec': {
                'Format': 'S3BatchOperations_CSV_20180820',
                'Fields': ['Bucket','Key']
            },
            'Location': {
                'ObjectArn':f"arn:aws:s3:::{manifest_path}",
                'ETag': str(random_string)
         }

But clearly in the GUI you have 3 different options of which the manifest.json is one of them. A manifest.csv is never actually created asides if you want to explicitly extract from the data dir in the inventory source bucket and rename or create the file yourself but that's not the best way why not just generate a manifest.csv as well along side the manifest.json and checksum file ?

Because in the image below there is really no point of having the csv option unless you create it your self and add it to another bucket your self

[tim-finnigan]

Both the AWS CLI create-job and boto3 create_job commands correspond to the S3 Control CreateJob API. We can reach out to the S3 Control team for input on their API designs but we don't have direct input or insight into that.

Do you get an error when trying to upload a JSON file? In the JobManifest documentation it says:

If the manifest is in CSV format, also describes the columns contained within the manifest.

This suggests that a CSV file may not be the mandatory format.

[damolaobaleke]

Yes, it cant read it because its not an allowed specified format. The only format allowed is CSV according to the docs showing it here https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_JobManifestSpec.html

The error gotten explicitly is this:

Reasons for failure
Reading the manifest is forbidden: 403 Forbidden

Which makes a lot of sense due to the only allowed readable format being csv. so their wording wasn't the best there.

I eventually had to do an extensive work around.

1. Download the s3 csv object from the data directory
2. Unzipping the downloaded csv getting the info
3. Writing to a blank manifest.csv locally with the Bucket and Key field as columns
4. Storing/Putting that in a new s3 bucket.

After then making a request to get the manifest.csv from the bucket and assigning the

 'Location': {
   'ObjectArn':"arn:aws:s3:::{manifest_csv_path}",
     'ETag': e_tag.replace('"', '').strip()
}

Please reach out to the S3 Control team for input on their API designs, it would be brilliant to have json as an allowed format.

[tim-finnigan]

Ok I created a tracking issue here and reached out to the S3 team: aws/aws-sdk#547. Will update that issue when we hear back.

[tim-finnigan]

Following up as we heard back from the S3 team who provided this answer: aws/aws-sdk#547 (comment). Please let us know if you have any follow up questions.