More powerful instances cost more money so decide Prevent us from what we are trying to do. Enabling JupyterHub and Zeppelin options will provide aĪlbeit clunky jupyter notebook (as described above in Caveats) but it won’t If this is your first time setting upĪn EMR cluster go ahead and check Hadoop, Zepplein, Livy, JupyterHub, Pig, Hive, Hue, and
#Aws linux install apache spark software#
Launch an EMR cluster with a software configuration shown below in the picture.Įnsure that Hadoop and Spark are checked. We will use advanced options to launch the EMR cluster. Yourself, which is always a good thing for a data scientist.
#Aws linux install apache spark how to#
The increased complexity in the AWS-provided installation makes itĭifficult to install and use python packages (such as pandas and numpy).įinally, installing our own jupyter package is a good way to learn how to setup some tools for With some features that we don’t really need at this time.
The AWS-provided JupyterHub installation does provide us It also adds another layer of complexity to an already complicated This is because the AWS-provided JupyterHub installation runs withinĪ docker container and deprives the end user (which is you and me) of many python and Jupyter package on the cluster and use that. We will use Python, Jupyter notebook, andīut we will not use the AWS-provided installation. Some steps such as setting up billing and setting up an I recommend you first try out setting up an EC2 instance before you move on to If this is your first time using a cloud computing platform To setup a payment method (such as a credit card) and may cost you some money. I do not cover these details in this post either.
Some familiarity with AWS concepts such as EC2, ssh keys, VPC subnets, and security groups. I do not go over the details of setting up AWS EMR cluster.įor details. We will see more details of the dataset later. Shows you how to access this dataset on AWS S3. This medium postĭescribes the IRS 990 dataset. Which makes it a good candidate to learn Spark. Let’s use it to analyze the publiclyĪvailable IRS 990 data from 2011 to present.
0 kommentar(er)
