From: "Robert Gibbon (JIRA)" <jira(a)apache.org>
To: tstclair(a)redhat.com
Sent: Monday, July 1, 2013 12:22:20 PM
Subject: [jira] [Commented] (HADOOP-9680) Extend S3FS and S3NativeFS to work with AWS IAM
Temporary Security
Credentials
[
https://issues.apache.org/jira/browse/HADOOP-9680?page=com.atlassian.jira...
]
Robert Gibbon commented on HADOOP-9680:
---------------------------------------
I took a look at your patch in HADOOP-9623. Some comments:
* Bucket keyspace listings running over a s3-native fs will be broken by your
patch, they make use of the method
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(String
key) when querying S3 for the given URI; if the URI does not correspond to a
single key in a bucket (ie a single object) an exception is thrown. In the
above mentioned method, the exception's Message property was being parsed
for the string "ResponseCode=404" to interpret that the URI is not a single
key. If the condition is met, it returns null. Its a horrible piece of code
and a very poorly defined contract with the calling party. It is also broken
by jets3t 0.9.0, which doesn't pass back that message anymore in that
situation. I adapted it to look at the ResponseCode property for the integer
404 instead, but someone who knows that code better than me would do a good
deed to fix it more sustainably.
* I needed to upgrade jets3t to 0.9.0 because I need support for AWS IAM
federated access tokens (temporary, time limited access credentials, tied to
a session ticket). I don't see any support for that in the patch in
HADOOP-9623, for me its of no value unless it supports temp security tokens.
I think we're aligned on the need for an uprev of the jets3t implementation
in hadoop.
HTH
> Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security
> Credentials
> ------------------------------------------------------------------------------
>
> Key: HADOOP-9680
> URL:
https://issues.apache.org/jira/browse/HADOOP-9680
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Robert Gibbon
> Priority: Minor
> Attachments: s3fs-temp-iam-creds.diff.patch
>
>
> Here is a patch in unified diff format to enable Amazon Web Services IAM
> Temporary Security Credentials secured interactions with S3 from Hadoop.
> It bumps the JetS3t release version up to 0.9.0.
> To use a temporary security credential set, you need to provide the
> following properties, depending on the implementation (s3 or s3native):
> fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id
> issued by AWS IAM
> fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary
> secret access key issued by AWS IAM
> fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued
> by AWS IAM along with the temporary key
> fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:
http://www.atlassian.com/software/jira