Hadoop S3A

{"value":"Hadoop S3A\u4e3ahadoop\u5b98\u65b9\u63d0\u4f9b\u7684\u5728hadoop\u7cfb\u7edf\u4e2d\u4f7f\u7528S3\u7684\u5de5\u5177\u5305\uff0c\u901a\u8fc7S3A\u53ef\u4ee5\u50cf\u64cd\u4f5chdfs\u4e00\u6837\u64cd\u4f5cS3\u5b58\u50a8\u3002\u76ee\u524d\u79c1\u6709\u4e91\u5b58\u50a8\u4e5f\u53ef\u4ee5\u652f\u6301\u5927\u90e8\u5206\u5e38\u7528\u7684S3A\u7684\u529f\u80fd\u3002\u5173\u4e8eS3A\u7684\u66f4\u591a\u4ecb\u7ecd\u53c2\u89c1\uff1a[Hadoop-AWS module: Integration with Amazon Web Services](https:\/\/hadoop.apache.org\/docs\/current\/hadoop-aws\/tools\/hadoop-aws\/index.html)\n\n1 \u6dfb\u52a0jar\u5305 [hadoop-aws.jar](https:\/\/mvnrepository.com\/artifact\/org.apache.hadoop\/hadoop-aws)\uff0c\u76f8\u5173\u5305\u4f9d\u8d56\u53ef\u53c2\u8003\u5b98\u7f51\u6587\u6863\n\n2 \u914d\u7f6e{hadoop}\/core-site.xml\n\n```\n<property>\n <name>fs.s3a.access.key<\/name>\n <value>{your-qiniu-access-key-id}<\/value>\n<\/property>\n<property>\n <name>fs.s3a.secret.key<\/name>\n <value>{your-qiniu-secret-key-id}<\/value>\n<\/property>\n<property>\n <name>fs.s3a.endpoint<\/name>\n <value>{qiniu-s3-endpoint}<\/value>\n<\/property>\n```\n\n3 \u4f7f\u7528S3A\u5728hadoop\u73af\u5883\u4e0b\u9a8c\u8bc1\u4f7f\u7528\u79c1\u6709\u4e91\u5b58\u50a8\n\n```\nhadoop fs -ls s3a:\/\/bucketname\/\n```\n\n# \u529f\u80fd\u5217\u8868\u652f\u6301\u8bf4\u660e\n\n\u76ee\u524d\u79c1\u6709\u4e91\u5b58\u50a8\u5df2\u652f\u6301\u7684Hadoop shell command\u5982\u4e0b\uff1a\n\n| \u529f\u80fd | \u517c\u5bb9\u8bf4\u660e |\n| ----------------- | ------------------------------------------------------------ |\n| mkdir | \u5728\u5b58\u50a8\u7cfb\u7edf\u521b\u5efa\u65b0\u76ee\u5f55 |\n| put\/copyFromLocal | \u4e0a\u4f20\u672c\u5730\u6587\u4ef6\u5230\u5b58\u50a8\u7cfb\u7edf |\n| cat | \u8f93\u51fa\u5b58\u50a8\u7cfb\u7edf\u5355\u4e2a\u6216\u591a\u4e2a\u6587\u4ef6\u5185\u5bb9\u5230\u63a7\u5236\u53f0 |\n| tail | \u8f93\u51fa\u5b58\u50a8\u7cfb\u7edf\u6307\u5b9a\u6587\u4ef6\u5c3e\u90e81k\u5b57\u8282\u5185\u5bb9\u5230\u63a7\u5236\u53f0 |\n| head | \u8f93\u51fa\u5b58\u50a8\u7cfb\u7edf\u6307\u5b9a\u6587\u4ef6\u5934\u90e81k\u5b57\u8282\u5185\u5bb9\u5230\u63a7\u5236\u53f0 |\n| ls | \u5217\u51fa\u5b58\u50a8\u7cfb\u7edf\u6307\u5b9a\u8def\u5f84\u4e0b\u6240\u6709\u6587\u4ef6 |\n| lsr | \u7c7b\u4f3c ls -R |\n| get\/copyToLocal | \u62c9\u53d6\u5b58\u50a8\u7cfb\u7edf\u6587\u4ef6\u5230\u672c\u5730 |\n| getmerge | \u62c9\u53d6\u5b58\u50a8\u7cfb\u7edf\u6587\u4ef6\u5230\u672c\u5730\u5e76\u5408\u5e76\u6210\u4e00\u4e2a\u6587\u4ef6 |\n| du\/dus | \u5c55\u793a\u76ee\u5f55\u6216\u8005\u6587\u4ef6\u5927\u5c0f |\n| cp | \u62f7\u8d1d\u5b58\u50a8\u7cfb\u7edf\u8def\u5f84\u4e0b\u6587\u4ef6\u5230\u53e6\u4e00\u4e2a\u8def\u5f84\uff08\u53ef\u8de8\u5b58\u50a8\u7cfb\u7edf\u62f7\u8d1dhdfs=>kodo \u6216kodo=>hdfs\uff09 |\n| mv | \u79fb\u52a8\u5b58\u50a8\u7cfb\u7edf\u8def\u5f84\u6587\u4ef6\u5230\u53e6\u4e00\u4e2a\u8def\u5f84\uff08\u4e0d\u652f\u6301\u8de8\u5b58\u50a8\u7cfb\u7edf\u79fb\u52a8\uff09 |\n| rm\/rmr | \u5220\u9664\u5b58\u50a8\u7cfb\u7edf\u4e2d\u7684\u6307\u5b9a\u76ee\u5f55\u6216\u6587\u4ef6 |\n| text | \u4ee5\u6587\u672c\u8f93\u51fa\u5b58\u50a8\u7cfb\u7edf\u538b\u7f29\u6587\u4ef6 |\n| touchz | \u521b\u5efa\u4e00\u4e2a\u5b58\u50a8\u7cfb\u7edf\u7a7a\u6587\u4ef6 |\n| count | \u7edf\u8ba1\u5b58\u50a8\u7cfb\u7edf\u8def\u5f84\u4e0b\u7684\u76ee\u5f55\u6570\u3001\u6587\u4ef6\u6570\u548c\u603b\u5b57\u8282\u6570 |"}