Using ECS Fargate, you'll have to take care of docker image and setup ECS infrastructure i.e. Either of them can be triggered by Cloudwatch Event triggered on an S3 event.Ī. Your problem is simply time out, so you may use either ECS Fargate, or Glue Python Shell Job. I wouldn't recommend data pipeline as it might be an overhead (It will start an EC2 instance to run your commands).But 15 minutes is still a lot of time, so you can implement alternative solution meanwhile. With bigger datasets, as you already know, Lambda may time out.There are a number of frameworks that can set up the Lambdas and Step Function as one unit. The Step function is an action sequencer and the Lambdas are the actions. Once the status checker lambda says the COPY is complete the step function launches the additional actions Lambda.has a wait loop that calls the "status checker" Lambda every 30 sec (or whatever interval you want) and keeps looping until the checker says that the COPY completed successfully.launches the first Lambda (initiates the COPY).All these Lambdas and their interactions are orchestrated by a Step Function that: Once COPY completes you can start another Lambda to run the additional actions. If you need to take additional actions after the COPY then you need a polling Lambda that checks to see when the COPY completes. The COPY command will run to completion and if this is all you need to do then your done. With the Redshift Data API you can launch a SQL command (COPY) and close your Lambda function.
A server-less approach I've recommended clients move to in this case is Redshift Data API (and Step Functions if needed).
0 kommentar(er)
