文章目录
🐷 背景
由于AWS整体成本略低于阿里云，公司决定将文件存储模块的业务迁移至AWS，迁移过程中发现，关于视频播放，转码之前阿里云已自动集成相关功能，但Aws并没有原生支持，为此踩了很多坑！
🦥 简述
本次主要是完成视频上传Aws并完成转码播放功能，基础功能已实现，但速度方面略低于阿里云，后续还需要优化改进。
Aws指南
🐥 Aws服务
🦜 AWS CloudFormation
CloudFormation是一个模板，该模板包含一套解决方案所需的所有资源，将一个通用解决方案配置成一个模板，需要再次部署相同解决方案则使用模板即可快速实现复制。启动堆栈可对模板进行部署实施。
Aws点播使用的模板
整体转码流程
官方服务说明
堆栈部署产生的aws服务说明
mediaconvert作业模板
Lambda
Lambda很重要！Lambda很重要！Lambda很重要！
IAM Role
IAM很重要！IAM很重要！IAM很重要！
SQS
SQS消息队列，转码完成后会向消息队列写入消息，业务侧可读取消息实现转码完成事件通知。
🐞 问题
使用Workflow trigger选项VideoFile的话，部署的方案会默认转码成m3u8文件；会默认使用3个模板进行转码（MediaConvert_Template_2160p、MediaConvert_Template_1080p、MediaConvert_Template_720p）于现有阿里云转码成单个mp4文件不一致，现有业务对多种品质输出没有要求，需要能够定制转码输出格式。
备注：如要实现类似点播清晰度选择的话可考虑多个质量转码模板。
解决方案在github上
🐉 落地方案
调整堆栈参数
Workflow trigger选择MetadataFile ，工作流触发参数由VideoFile调整为Metadata，堆栈会进行自动调整。
调整后s3桶事件通知中由之前的视频文件触发工作流，变更为由metadata触发，metadata就是.json的一个文件，具体内容参考github
Metadata文件创建
堆栈调整后，上传视频文件不再触发工作流，此时需要对上传的视频文件生成对应的metadata文件（.json文件），可以由业务侧进行两次上传，先上传视频文件，再上传metadata文件，进而触发工作流。考虑存在失败的可能以及业务侧需要实现额外的逻辑，因此使用了Lambda创建Metadata文件，文件上传完成触发Lambda，由Lambda实现生成Metadata文件。
const aws = require('aws-sdk');
const s3 = new aws.S3({ apiVersion: '2006-03-01' });
exports.handler = async (event, context) => {
// Get the object from the event and show its content type
const bucket = event.Records[0].s3.bucket.name;
const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
if(key.endsWith('.json')){
console.log('json file');
return;
}
const params = {
Bucket: bucket,
Key: key,
};
console.log(params);
const metadata={
"srcVideo": key,
"ArchiveSource": true,
"FrameCapture":true,
"JobTemplate":"jiaoyubaoCompanyVideo_trans_template"
};
const metadataKey=key.substring(0,key.lastIndexOf('.'))+'.json';
console.log('metadatakey:',metadataKey);
try {
//const { ContentType } = await s3.headObject(params).promise();
//console.log('CONTENT TYPE:', ContentType);
//upload metadata
const uploadParams = {Bucket: bucket, Key: metadataKey, Body: JSON.stringify(metadata)};
console.log(uploadParams);
// call S3 to retrieve upload file to specified bucket
const res=await s3.upload (uploadParams).promise();
return res;
} catch (err) {
console.log(err);
throw err;
}
};
自定义转码模板
默认模板转码为m3u8格式文件，需要实现mp4格式转码，自定义转码模板jiaoyubaoCompanyVideo_trans_template
质量优化级别建议使用单一传递 HQ，太低转码文件质量较低，但文件大小会降低明显，考虑效果，采用中间质量。
最大比特率(位/秒)设置为了2000000，该值会影响转码视频大小，实际测试2000000转码视频较源视频大小略有压缩，固采用该值，如需高质量可参考该值设置更大的值，如需更多的压缩，可降低该值。
业务侧上传视频
客户端直连S3实现分片续传思路与实践
https://aws.amazon.com/cn/blogs/china/s3-multipul-upload-practice/
授权方案
方案一：STS临时授权
策略：ClientUploadVideoS3Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject"
],
"Resource": "arn:aws-cn:s3:::videoondemand-company-source234242b7/*"
}
]
}
角色：ClientVideoUploadAssumeRole
AssumeRole
private Amazon.SecurityToken.Model.AssumeRoleResponse GetOriginalAwsAssumeRole(int durationSeconds=3600)
{
//Amazon Resource Name
var roleArnToAssume = "arn:aws-cn:iam::77480923422862:role/ClientVideoUploadAssumeRole";
var client = new Amazon.SecurityToken.AmazonSecurityTokenServiceClient(AWSCredentialsConfig.AwsAccessKeyId,AWSCredentialsConfig.AwsSecretAccessKey,Amazon.RegionEndpoint.CNNorthWest1);
// Create the request to use with the AssumeRoleAsync call.
var assumeRoleReq = new Amazon.SecurityToken.Model.AssumeRoleRequest()
{
DurationSeconds = durationSeconds,
RoleSessionName = "AwsUpload",
RoleArn = roleArnToAssume
};
var assumeRoleRes = client.AssumeRole(assumeRoleReq);
return assumeRoleRes;
}
{
"AssumedRoleUser": {
"Arn": "arn:aws-cn:sts::77421422862:assumed-role/ClientVideoUploadAssumeRole/AwsUpload",
"AssumedRoleId": "AROA3IZSCNY432ZRBYTQU5:AwsUpload"
},
"Credentials": {
"AccessKeyId": "ASIA3234fsdfwe42Q3UK6",
"Expiration": "2022-12-30T15:24:37+08:00",
"SecretAccessKey": "khoROTW8vyDSD7tcaDmvMZg7S0Ujte6TiRP+ql2b",
"SessionToken": "IQoJb3JpZ2luX2VjEGEaDmNuLW5vcnRod2VzdC0xIkYwRAIgUZ7b3+OGVn+agrp3IJ+JRQDjTTgKcGW+F92eOi0CsMICICqooZD6FJoCnOdBNaAPfrvzB5iG4fxzSQL234234uEKp8CCI///wEQABoMNzc0ODA5NzQyODYyIgw+fkUE58kq72ukAWkq8wEbHoeO5inZTKxAr/kafYBQ5RFOyYx8EJjrYd/uCReS6Te/Ix64D9/aqSM/BimyACmnjM3BmeDuys12zYDdHLfOAhuOUR4dyjKEXPWTwV6usxVb+5EYw4T9Z47dX9gV8EoI0n342342jzKnY8iQIikpidsYFjo6g27Q/dddCUXGqhIUB0QtzAhoMRAM9ndVl9w2VH7CBUsFPFXPhUaxhhIol59p8L2342342BKkFFpY/Zf8wSMLBYLLO3p9/uCBA4D06chibtYVoiIPLoMvPDlOEU6iAk+LtKTOtdmZ5/J8+WTGeZB8Y6R7II14wpe++rAY6ngGPSJw/q0JoPm8WMMH84c3Lv47V9n9sDTSX8WHlASPvgySRiuP+DvSd3RqMDootuk3awg1x3hvzU438xJ+BCo5qAWFIJE2niwyslWh3zrxiGADBAqqg/XT2P4yQg8wiscQh38KxSOlj+LxgvTcdVfe4/Yy4uWtudvWR4EkcVvpWWIw+sxDkX0LlYvfHz3yhAq209Gt8E5zcX4NWQ8mnug=="
},
"PackedPolicySize": 0,
"SourceIdentity": null,
"ResponseMetadata": {
"RequestId": "15c423624-923f4-452b-902e-1e8561175523",
"Metadata": {
},
"ChecksumAlgorithm": 0,
"ChecksumValidationStatus": 0
},
"ContentLength": 1463,
"HttpStatusCode": 200
}
方案二：使用预签名 URL（未验证）
https://docs.amazonaws.cn/AmazonS3/latest/userguide/using-presigned-url.html
jssdk直传
<script src="/js/aws-sdk-2.45.0.min.js"></script>
<script>
var accessKeyId = "{$:token.Credentials.AccessKeyId}";
var accessKeySecret = "{$:token.Credentials.SecretAccessKey}";
var sessionToken = "{$:token.Credentials.SessionToken}";
var endpoint = `${location.protocol}//s3.cn-northwest-1.amazonaws.com.cn`;
var s3 = new AWS.S3({ accessKeyId: accessKeyId, secretAccessKey: accessKeySecret, sessionToken: sessionToken, region: 'cn-northwest-1', endpoint: endpoint, signatureVersion: 'v4' });
var filePicker = document.getElementById("videoPicker");
var progress = document.getElementById("progress");
filePicker.addEventListener('change', function (event) {
console.log(event.target.files[0]);
if (event.target.files[0].size / 1024 / 1024 > 50) {
alert("文件超出大小限制，不得超过50M");
document.getElementById("videoPicker").value = null;
return;
}
});
function start() {
if (filePicker.files.length <= 0) {
alert("请先选择视频");
return;
}
var file = filePicker.files[0];
var ext = file.name.substring(file.name.lastIndexOf('.'));
$("#progressInfo").css({ width: '100%' });
filePicker.setAttribute("disabled", "disabled");
var params = {
Bucket: 'videoondemand-company-source-gbwnqmc885b7',
Key: `{$:companyInfo.comid}_${Date.now()}${ext}`,
Body:file
};
var upload = s3.upload(params, {
queueSize: 1,//分片上传并发队列，代表了能同时上传的分片数量
connectTimeout: 60 * 1000 * 10,
httpOptions: {
timeout: 60 * 1000 * 10
}
}).on('httpUploadProgress', function (e) {
var percent = parseInt(e.loaded, 10) / parseInt(e.total, 10);
percent = percent.toFixed(2) * 100;
setTimeout(function () {
var p = percent + "%";
$(".progress span").css({ width: p });
});
});
upload.send(function (err, data) {
console.log(err);
console.log(data);
if (err) {
alert("上传失败，请稍后重试");
$("#progressInfo").css({ width: '0' });
}
//上传成功
$.ajax({
url: '/CompanyHandler.ashx',
type: 'post',
dataType: 'json',
data: { action: 'saveVideo', object: data.Key, rand: Math.random() },
success: function (result) {
if (result.result == 1) {
//关闭弹窗，刷新列表页
parent.location.reload();
}
else {
alert(result.description);
}
},
error: function () {
alert("发生异常，请稍后重试或联系客服反馈");
}
});
filePicker.removeAttribute("disabled");
});
}
</script>
S3 CORS
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"https://*.cn"
],
"ExposeHeaders": [
"ETag"
]
},
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"http://*.jiaoyubao.cn"
],
"ExposeHeaders": [
"ETag"
]
},
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"http://localhost:53762"
],
"ExposeHeaders": [
"ETag"
]
}
]
SQS异步通知机制
VideoonDemand-Company-Notify
const http = require('http')
exports.handler = async (event) => {
const data = JSON.stringify(event);
console.log(data)
//其他业务复用当前部署堆栈的话，异步通知需要差异化的话，可以考虑通过视频前缀约束在此处进行差异化处理。
const options = {
hostname: 'vip3.shhddg.cn',
port: 80,
path: '/awsnotifications.aspx',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': data.length
}
}
await new Promise(function (resolve, reject) {
const req = http.request(options, res => {
console.log(`状态码: ${res.statusCode}`)
res.on('data', d => {
})
});
req.on('error', error => {
console.error(error)
});
req.write(data);
req.end();
});
return event;
};
域名解析到S3
🦉 Aws vs Aliyun
机构后台使用STS临时授权方案上传S3，对比阿里云接口调用和方式，Aws和阿里云从风格到接口基本一致。
阿里云
private AssumeRoleResponse GetStsToken()
{
//获取sts凭证
const string REGIONID = "cn-hangzhou";
const string ENDPOINT = "sts.cn-yuenan.aliyuncs.com";
// 构建一个 Aliyun Client, 用于发起请求
// 构建Aliyun Client时需要设置AccessKeyId和AccessKeySevcret
DefaultProfile.AddEndpoint(REGIONID, REGIONID, "Sts", ENDPOINT);
IClientProfile profile = DefaultProfile.GetProfile(REGIONID, AppCode.ConstSetting.AccessKeyId, AppCode.ConstSetting.AccessKeySecret);
DefaultAcsClient client = new DefaultAcsClient(profile);
// 构造AssumeRole请求
AssumeRoleRequest request = new AssumeRoleRequest();
request.AcceptFormat = FormatType.JSON;
// 指定角色Arn
request.RoleArn = "acs:ram::14898694342448426:role/jiaoyubaovideorole";
request.RoleSessionName = "upload";
// 可以设置Token有效期，可选参数，默认3600秒；
// request.DurationSeconds = 3600;
// 可以设置Token的附加Policy，可以在获取Token时，通过额外设置一个Policy进一步减小Token的权限；
// request.Policy="<policy-content>"
try
{
AssumeRoleResponse response = client.GetAcsResponse(request);
//Token过期时间；服务器返回UTC时间，这里转换成北京时间显示；
return response;
}
catch (Exception ex)
{
//Console.Write(ex.ToString());
return null;
}
}
Aws
private Amazon.SecurityToken.Model.AssumeRoleResponse GetOriginalAwsAssumeRole(int durationSeconds=3600)
{
var roleArnToAssume = "arn:aws-cn:iam::7743242742862:role/ClientVideoUploadAssumeRole";
var client = new Amazon.SecurityToken.AmazonSecurityTokenServiceClient(AWSCredentialsConfig.AwsAccessKeyId,AWSCredentialsConfig.AwsSecretAccessKey,Amazon.RegionEndpoint.CNNorthWest1);
// Create the request to use with the AssumeRoleAsync call.
var assumeRoleReq = new Amazon.SecurityToken.Model.AssumeRoleRequest()
{
DurationSeconds = durationSeconds,
RoleSessionName = "AwsUpload",
RoleArn = roleArnToAssume
};
var assumeRoleRes = client.AssumeRole(assumeRoleReq);
return assumeRoleRes;
}
🍄 避坑指南
坑1：文档版本不一致，不好找
服务商给到的视频点播文档中的方案链接https://www.amazonaws.cn/solutions/video-on-demand/?nc1=h_ls
搜索引擎或官方给到的视频点播文档链接https://aws.amazon.com/cn/solutions/implementations/video-on-demand-on-aws/
坑2：SQS Lambda 触发器需要额外的权限
The provided execution role does not have permissions to call ReceiveMessage
Lambda函数 配置-权限-执行角色 的权限策略需要添加AWSLambdaSQSExecutionRole 策略
坑3：S3 CORS
分片上传ETag问题，使用分片上传的话ExposeHeaders务必设置ETag，否则会存在跨域问题，可参考以下文档
https://www.cnblogs.com/duhuo/p/14828021.html
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"https://*.hfdhsa.cn"
],
"ExposeHeaders": [
"ETag" //坑
]
},
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"http://*.geertfd.cn"
],
"ExposeHeaders": [
"ETag"
]
},
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT",
"POST",
"DELETE"
],
"AllowedOrigins": [
"http://localhost:53762" //坑，本地测试，多端口尚未解决
],
"ExposeHeaders": [
"ETag"
]
}
]
坑4：jssdk
var accessKeyId = "{$:token.Credentials.AccessKeyId}";
var accessKeySecret = "{$:token.Credentials.SecretAccessKey}";
var sessionToken = "{$:token.Credentials.SessionToken}";//坑，要传
//坑，要传，特别是协议cors
var endpoint = `${location.protocol}//s3.cn-northwest-1.amazonaws.com.cn`;
var s3 = new AWS.S3({ accessKeyId: accessKeyId
, secretAccessKey: accessKeySecret
, sessionToken: sessionToken
, region: 'cn-northwest-1'
, endpoint: endpoint
, signatureVersion: 'v4' });//坑 signatureVersion不指定报错
坑5：Lambda
await new Promise(function (resolve, reject) {
const req = http.request(options, res => {
console.log(`状态码: ${res.statusCode}`)
res.on('data', d => {
})
});
req.on('error', error => {
console.error(error)
});
req.write(data);
req.end();
});
