![]() Well, I would like to start with something tough to understand for me: it’s not Apache Airflow 2.0. It’s well integrated with AWS ecosystem (e.g., Amazon EMR, AWS Glue and so on).It’s just Apache Airflow, and that means 100% compatibility with open source ecosystem that already is in place.It’s a fully managed version of Apache Airflow, which has an opinion being pesky to operate.To reiterate the right things, and emphasize that this service can be used now in certain situations: But at the same time requires a lot of work. The service is usable, has a great value proposition, fixes many issues of unmanaged solutions. Is it perfect?ĭon’t get me wrong: I tried to guess what AWS releases this year, which was on the top of my list. It means that you deal with a fully-managed service that supports well-known plugins and has full compatibility and integration with AWS portfolio.Īs a person who worked with Amazon Data Pipeline, AWS Glue Workflows, and AWS Step Functions, I am thrilled that we received an alternative that is fully compatible with an open source version - because that removed another point from the list of contraindications related to diving deeper into the cloud. The service selling point is that you have the same Apache Airflow as the open source version. In that sense AWS did what they do the best very consistently: they’ve monetized their operational knowledge by providing a fully-managed service. Not to mention that Apache Airflow itself is very pesky to manage and operate reliably. Because of that, other cloud and SaaS providers already allowed us to use this service in a managed way. By combining it with Kubernetes, many data teams used that as a data infrastructure design pattern. Apache Airflow is a state-of-the-art workflow management platform for data analytics. ![]() This is one of the pre-re:Invent 2020 announcements. Recently, I had an opportunity to dive deeper into the newly released AWS service that allows us to provision and use a fully-managed version of Apache Airflow.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |