Task 1: Higher-Order Functions (Racket Only) 30 pts A minimal arithmetic expression language is defined by the following grammar: arith-expr =| (Neg ) | (Plus ) | (Times ) In this task, you are to define and use some higher-order functions for expressions of this minimal language. (a) map-expr 10 pts Implement map-expr for expressions, which takes the following as input: • a function f that takes a number as input and returns a number, and • an expression expr. Function map-expr applies f to each number appearing in the expression and returns the resulting expression, which should have the same structure as the original one. (b) fold-expr 20 pts You are given an implementation of function fold-expr, which takes the following as input: • a function f-num that folds a number, • a function f-neg that folds (Neg c), • a function f-plus that folds (Plus l r), 1 • a function f-times that folds (Times l r), and • an expression expr. Function fold-expr folds the expression as described by the input functions and returns the result. Notice that the result type of fold-expr depends on what those input higherorder functions return. Read the implementation of fold-expr in the starter code to understand how this is done. Implement two functions expr-to-number and expr-to-list that respectively fold expressions to numbers or their list representations using fold-expr: • Function expr-to-number takes an expression as input and evaluates the arithmetic operations, then returns the resulting number. • Function expr-to-list takes an expression as input and converts it to its list representation (or a number if it is just a number). See the examples in the starter code for the exact notation. Note that these implementations should be very short and only call fold-expr. You are only allowed to fill in the appropriate inputs of f-num, f-neg, f-plus, and f-times in the calls to fold-expr to implement the two functions required. We will check this in your implementation. 2 Task 2: Currying (Haskell Only) 40 pts (a) curry, uncurry, curry3, and uncurry3 20 pts We can think of a function f with two inputs in two ways: • Uncurried: f takes a tuple as input and returns a value. For example: ucPlus :: (Int, Int) -> Int ucPlus = (x, y) -> x + y –- same as: ucPlus (x, y) = x + y • Curried: f is a higher-order function which takes a value as input (the “first” input) and returns a function which itself takes a value as input (the “second” input) and returns a value. For example: cPlus :: Int -> (Int -> Int) cPlus = x -> (y -> x + y) This is how Haskell functions are defined by default, so we can write the example above more simply as: cPlus :: Int -> Int -> Int cPlus x y = x + y Implement four functions curry, uncurry, curry3, and uncurry3: • curry takes as input an uncurried function and returns an equivalent curried function. For instance, curry ucPlus is functionally equivalent to cPlus. (curry ucPlus) 2 3 == 5. • uncurry takes as input a curried function and returns an equivalent uncurried function. For instance, uncurry cPlus is functionally equivalent to ucPlus. (uncurry cPlus) (2, 3) == 5. • curry3 and uncurry3 work similarly to curry and uncurry respectively, but for functions with three inputs. (b) zip, unzip, and zipWith 20 pts Implement two functions zip and unzip which convert a pair of lists to a list of pairs and vice versa. Additionally, implement zipWith which combines two lists into one list using a given function. • zip takes two lists xs and ys, and returns a list of pairs zs. If x and y are the k-th elements of xs and ys respectively, the k-th element of zs is a pair (x, y). If one input list is longer than the other, then the remaining elements in the longer list are ignored. For example: zip [1, 2, 3] [‘a’, ‘b’, ‘c’] = [(1, ‘a’), (2, ‘b’), (3, ‘c’)] zip [1, 2, 3] [‘a’, ‘b’] = [(1, ‘a’), (2, ‘b’)] 3 • unzip takes a list of pairs zs and returns a pair of lists (xs, ys). If (x, y) is the k-th element of zs, then x and y are the k-th elements of xs and ys respectively. Moreover, xs and ys have the same length as the input list. For example: unzip [(1, ‘a’), (2, ‘b’), (3, ‘c’)] == ([1, 2, 3], [‘a’, ‘b’, ‘c’]) • zipWith takes two lists xs and ys, and a function f which can combine elements of xs and ys, and returns a list zs. If x and y are the k-th elements of xs and ys respectively, the k-th element of zs is the result of calling f x y. If one input list is longer than the other, then the remaining elements in the longer list are ignored. For example: zipWith (+) [1, 2, 3] [2, 4, 6] == [3, 6, 9] 4 Task 3: Evaluation (Haskell Only) 30 pts Recall the language from Assignment 1 defined by the following syntax: expr = (’λ () ) | ( ) | (’+ ) | | In class, we learned about evaluation models based on substitution and closures and their implementations in Racket. In this task, you are to write functions that evaluate expressions of this language in Haskell based on two evaluation models. See the starter file for detailed semantics of each function for each subtask. (a) evalEagerSubst 15 pts Implement evalEagerSubst in Haskell. For this subtask, we have provided you the data type definitions of expressions of this language and an implementation of the substitution function subst in Haskell. (We do not require capture-avoiding substitution.) (b) evalEagerEnv 15 pts Implement evalEagerEnv in Haskell. For this subtask, we have provided you the data type definitions of values of this language and an implementation of a simple map in Haskell. 5 Submission and Instructions Submit the files a2.rkt and A2.hs to MarkUs. Make sure to complete all sections labeled “Complete me” in a2.rkt and “undefined” in A2.hs. For all assignments: • You are responsible for making sure that your code has no syntax errors or compile errors. If your file cannot be imported in another file, you may receive a grade of zero. • If you’re not intending to complete one of the functions, do not remove the function signature. If you are including a partial solution, make sure it doesn’t cause a compile error. If your partial solution causes compiles errors, it’s better to comment out your solution (but not the signature). • In Racket, you may not use any iterative or mutating functionality unless explicitly allowed. Iterative functions in Racket are functions like loop and for. Mutating functions in racket have a ! in their names, for example set!. If you use the materials discussed in class and do not go out of your way to find these functions, your code should be okay. • Do not modify the (provide …) (in Racket) and module (…) where (in Haskell) lines of your code. These lines are crucial for your code to pass the tests.
Task 1: Intro to Racket & Haskell 20 pts (a) celsius-to-farenheit (Racket and Haskell) 10 pts The function celsius-to-farenheit (in Haskell: celsiusToFarenheit) takes a temperature in degrees Celsius and converts it to farenheit, rounded to the nearest integer. (b) remove-second (Racket and Haskell) 10 pts The function remove-second (in Haskell: removeSecond) takes a list, removes the second element of the list, and returns the resulting list. If the input list has less than two elements, it returns the list unmodified. 1 Task 2: Recursion 45 pts (a) collatz (Racket and Haskell) 10 pts Consider the following operation on a positive integer n: – If n is even, divide it by 2. Specifically, return n/2. – If n is odd, triple it and add 1. Specifically, return 3n + 1. The collatz conjecture is a well-known unsolved problem which claims that if we start with any positive integer n and apply the above operation repeatedly, we will eventually reach 1. For example, starting with 12, we get the sequence 12, 6, 3, 10, 5, 16, 8, 4, 2, 1. This conjecture has been verified for all integers up to 1020, but not proven in general. You will implement a function collatz which takes a positive integer n and returns the sequence described above, starting at n and ending at 1. The sequence starting at the input n is guaranteed to lead to a 1. (b) better-fibonacci (Racket and Haskell) 20 pts The Fibonacci sequence is defined as follows: fn = ( 1 if n = 0 or n = 1 fn−1 + fn−2 otherwise The function fibonacci takes a non-negative integer n and returns the n-th element of the fibonacci sequence. You are given a simple implementation of fibonacci in the starter code. 1. Asnwer the short-answer question in the Racket file labelled “QUESTION 1”: QUESTION 1: When calling (fibonacci 5), how many times is ‘fibonacci’ called (including the initial call and all recursive calls)? Assign your answer to the variable fibonacci-saq as a number. See the Racket starter file for an example and a hint. 2. Implement fibonacci more efficiently, as better-fibonacci (betterFibonacci in Haskell). The key idea is to use a helper fibonacci-helper. The function fibonacci-helper (fibonacciHelper in Haskell) takes a non-negative integer n and returns a pair: the (n-1)-th and n-th elements of the fibonacci sequence. In other words, given input n, the helper returns (fn−1, fn). Note that please return (0, 1) for n = 0 and (1, 1) for n = 1 directly. 3. Answer the short-answer question in the Racket file labelled “QUESTION 2”: QUESTION 2: When calling (better-fibonacci 5), how many times is ‘fibonacci-helper’ called? Assign your answer to the variable better-fibonacci-saq as a number. See the Racket starter file for a hint. 2 Note that the short answer questions only need to be answered in the Racket file. (c) factorial-tail (Racket and Haskell) 15 pts The factorial of a positive integer n is defined as the product of all integers from 1 to n (or equivalently, from n to 1). You are given a simple implementation of factorial in the starter code. This implementation is not a tail recursion. Your goal is to implement factorial-tail which computes the factorial of a given positive integer n. As indicated by the name, your implementation of factorial-tail must use tail recursion. See the starter code for a hint. Task 3: Pattern Matching 35 pts (a) area-or-volume (Racket and Haskell) 20 pts We will define some syntax to describe a few basic “shapes”. We will use different notations in the Racket and Haskell exercises. Notation for Racket: Notation for Haskell: shape = (list ’circle ) Shape = Circle | (list ’triangle ) | Triangle | (list ’square ) | Square | (list ’rectangle ) | Rectangle | (list ’sphere ) | Sphere | (list ’cube ) | Cube | (list ’prism ) | Prism The function area-or-volume will take a shape as input. If the shape is 2D (circle, triangle, square, or rectangle), it will return its area, and if it is 3D (sphere, cube or prism), it will return its volume. See below for area and volume formulas. Assume π = 3. Circle with radius r: A = πr2 ≈ 3r 2 Triangle with base b and height h: A = 1 2 bh Square with size a: A = a 2 Rectangle with sides w and h: A = wh Sphere with radius r: V = 4 3 πr3 ≈ 4r 3 Cube with side a: V = a 3 Prism with height h and a base with area A: V = hA Note that the base of a prism can be any 2D shape (in our case: circle, triangle, square, or rectangle). The volume of a prism equals its height multiplied by the area of its base. 3 The Haskell exercise makes use of algebraic data types which will be covered in more detail later. For now, you only need to understand how to pattern match on these kinds of data types. See the function shapeToText in the Haskell starter code for an example. (b) subst (Racket Only) 15 pts Consider the language described by the following syntax: expr = (’λ () ) | ( ) | (’+
Part A: Build a classification model using text data For Part A, you will be solving a text classification task. The training data is stored in the Homework 4 Data folder. The data consists of headlines that have beenlabeled for whether they are clickbait. 1. Import the data. The headlines will become your vectorized X matrix, and the labels indicate a binary classification (clickbait or not). 2. Convert the headline data into an X feature matrix using a simple bag of words approach. 3. Run logistic regression to predict clickbait headlines. Remember to train_test_split your data and use GridSearchCV to find the best value of C. You should evaluate your data with F1 scoring. 4. Run 2 more logistic regression models by changing the vectorization approach (e.g. using n-grams, stop_words, and other techniques we discussed). In both cases, keep your logistic regression step the same. Only change how you’re generating the X matrix from the text data. 5. Which of your 3 models performed best? What are the most significant coefficients in each, and how do they compare? Part B: Build a Predictive Neural Network Using Keras In Part B, you will run a multilayer perceptron on the iris dataset to predict flower type. 1. Load the data. Data can be imported directly using pd.read_csv() and the link http://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv. 2. Using the Sequential interface in Keras, build a model with 2 hidden layers with 16 neurons in each. Compile and fit the model. Assess its performance usingaccuracy on data that has been train_test_split. 3. Run 2 additional models using different numbers of hidden layers and/or hidden neurons. 4. How does the performance compare between your 3 models? Part A Part B Send
Part A: 1. Describe the importance of training and test data. Why do we separate data into these subsets? 2. What is k-fold cross validation and what do we use it for? 3. How is k-fold cross validation different from stratified k-fold cross validation? 4. Name the 4 types of supervised learning models that we have learned thus far that are used to predict categorical dependent variables like whether an emailis labeled “spam” or “not spam.” 5. Name the 3 types of supervised learning models that we have learned thus far that are used to predict continuous dependent variables like test scores. Part B: 1. Import the spam dataset and print the first six rows. 2. Read through the documentation of the original dataset here: http://archive.ics.uci.edu/ml/machine-learning-databases/spambase/spambase.names (http://archive.ics.uci.edu/ml/machine-learning-databases/spambase/spambase.names) . The dependent variable is “spam” where one indicates that an email isspam and zero otherwise. Which three variables in the dataset do you think will be important predictors in a model of spam? Why? 3. Visualize the univariate distribution of each of the variables in the previous question. 4. Choose one model from Part A Question 4. Split the data into training and test subsets. Build a model with the three variables in the dataset that you think will be good predictors of “spam”. Run the model and evaluate prediction error using k-fold cross-validation. Describe why you chose any particular parameters for your model (e.g., if you used KNN how did you decide to choose a specific value for k). 5. Repeat the previous question but with a different model from Part A Question 4. 6. Repeat the previous question but with a different model from Part A Question 4. 7. Repeat the previous question but with a different model from Part A Question 4. 8. Now rerun all 4 models with 3 additional variables that you think will help the prediction accuracy. Did this cause the performance to improve over your previous models? 9. What is a variable that isn’t available in this dataset but you think could increase your final model’s predictive power if you had it? Why do you think it would improve your model? Part A Part B
Part A: Regression on California test scores 1. Find the url for the California Test Score Data Set from the following website: https://vincentarelbundock.github.io/Rdatasets/datasets.html (https://vincentarelbundock.github.io/Rdatasets/datasets.html) . Read through the “DOC” file to understand the variables in the dataset, then use the following url to import the data: https://vincentarelbundock.github.io/Rdatasets/csv/Ecdat/Caschool.csv (https://vincentarelbundock.github.io/Rdatasets/csv/Ecdat/Caschool.csv) . The target data (i.e. the dependent variable) is named “testscr”. You can use all variables in the data except for “readscr” and “mathscr” in the following analysis (those two variables were used to generate the dependent variable). 2. Visualize the univariate distribution of the target feature and each of the three continuous explanatory variables that you think are likely to have a relationshipwith the target feature. 3. Visualize the dependency of the target on each feature you just plotted. 4. Split the data into training and test sets. Build models that evaluate the relationship between all available quantitative X variables in the California test dataset and the target variable. Evaluate KNN (for regression), Linear Regression (OLS), Ridge, and Lasso using cross-validation with the default parameters. How different are the results? 5. Try running your models from the previous question with and without StandardScaler. Does using StandardScaler help? 6. Tune the parameters of the models where possible using GridSearchCV. Do the results improve? 7. Compare the coefficients of your two best linear models (not KNN). Do they agree on which features are important? 8. Discuss which final model you would choose to predict new data. Part B: Classification on red and white wine characteristics 1. First, import the red and the white wine csv files into separate pandas dataframes from the following website. Note that you’ll need to adjust the argument forread_csv() from sep=’,’ to sep=’;’ https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv (https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv) https://archive.ics.uci.edu/ml/machine-learningdatabases/wine-quality/winequality-red.csv (https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv) 2. Add a new column to each data frame called “winetype”. For the white wine dataset label the values in this column with a 0, indicating white wine. For the red wine dataset, label values with a 1, indicating red wine. Combine both datasets into a single dataframe. The target data (i.e. the dependent variable) is “winetype”. 3. Visualize the univariate distribution of the target feature and each of the three explanatory variables that you think are likely to have a relationship with the target feature. 4. Split data into training and test sets. Build models that evaluate the relationship between all available quantitative X variables in the dataset and the target variable. Evaluate Logistic Regression, Penalized Logistic Regression, and KNN (for classification) using cross-validation. How different are the results? 5. Try running your models from the previous question with and without StandardScaler. Does using StandardScaler help? 6. Tune the parameters of the models where possible using GridSearchCV. Do the results improve? 7. Compare the coefficients for Logistic Regression and Penalized Logistic Regression. Do they agree on which features are important? 8. Discuss which final model you would choose to predict new data. Part A Part B Course Chat Send
For this homework, we will be using the same connection as in lecture but rely on a different database called `witchcraft`. Use the information below to connect to the database.– MySQL database – user: `student` – password: `gues` – dbname: `witchcraft` – host: `columbia-mds-mysql.csbmzoea3lu9.us-east-1.rds.amazonaws.com` – port: `3306`The data comes from a project on “Scottish Witchcraft” and contains all people known to have been accused of witchcraft in early modern Scotland. There is information on where and when they were accused, how they were tried, what their fate was etc.>Julian Goodare, Lauren Martin, Joyce Miller and Louise Yeoman, ‘The Survey of Scottish Witchcraft’, http://www.shca.ed.ac.uk/Research/witches/)#### 1. Getting to know the dataa) Show all tables in the database. b) Show the first three entries in the table `accused`. c) How many people are included in the accused table? d) Looks like the `age` is missing for some observations. Count the number of non-missing values for `age` in the data. e) Show a list of unique `occupation`s. f) What proportion of accused are female?#### 2. Seeing the DevilLet’s look at some appearances of the devil in the `devilappearance` table.a) List the unique `devil_type`s in the data. b) There is also a little description of the type of the devil sighting in the `devil_text` column. How many of the sightings mention the word “black” in the description? c) What proportion of the devils (in `devil_type`) are male?#### 3. The trialLet’s take a look at the information on the `trial`.a) What are the average and maximum numbers of male and female accusers? b) Count the number of `sentence`s by sentence type. List them in a table (in descending order), excluding missing values. Rename the column headings to something sensible. c) Do the number of accusers matter for the `verdict`? Compare the average number of accusers by the type of verdict. Again make sure the table is sorted and the headings make sense.#### 4. Tortured Trutha) Left join the `trial` and `confession` tables. For what share of trials does the database record confessions? Create a results table with the number of all trials, the number of confessions, and the share of trials with confessions recorded.b) Only a small number of trials have records of torture. Is there a higher share of confessions among trials with records of torture than trials without such record? _Hint:_ You will need to merge on the `confession` table.### SubmissionPlease follow the [instructions](/Exercises/homework_submission_instructions.md) to submit your homework. The homework is due on Wednesday, November 20 at 5pm.
This exercise focuses on extracting data for S&P 500 companies from individual company pages on Wikipedia, with an emphasis on data consistency and error handling.Wikipedia does have an API. For this exercise, we will pretend there is no API and simply read-in the HTML.### 1. Create a Function to Scrape Company InfoboxesWe’ll start by building a function to gather information from each company’s individual Wikipedia page. Choose a single company to begin, such as “[Apple Inc.](https://en.wikipedia.org/wiki/Apple_Inc.).” This page contains an infobox with information like CEO, revenue, and number of employees, which we want to extract.a) Write a function called `get_company_info` which: – Takes a company’s Wikipedia URL as input. – Extracts the following fields from the infobox: – **Company Name** – **Industry** – **Revenue** (normalized to billions of USD if available) – **Net Income** (normalized to billions of USD if available) – **Number of Employees** – **Market Cap** (normalized to billions of USD if available) – Handles variability in field names if necessary. – Make sure to handle exceptions gracefully. – Returns a dictionary with these fields as keys and the extracted data as values. If a field is missing, the function should return `None` for that entry.Test `get_company_info` with the Apple Inc. page to verify it works and manages exceptions gracefully.b) Expand the function to gather information on key people in the company. Specifically, add the following fields: – **CEO** – **Founder(s)** – **Founded** (year)Make sure you parse the HTML for the `Founder(s)` field to get the actual names as a comma separated string. For the CEO, parse the HTML for the key people section and identify the CEO. Add these fields to the dictionary returned by the function.Test the function with the Apple Inc. page to verify it works and manages exceptions gracefully.### 2. Retrieve S&P 500 TableNavigate to the Wikipedia page with the full list of S&P 500 companies: [List of S&P 500 companies](https://en.wikipedia.org/wiki/List_of_S%26P_500_companies). This page contains a table with general information about each company.a) Write code to scrape this table and create a `pandas` DataFrame with the following columns: – **Company Name** – **Link to Company Wikipedia page** – **GICS Sector** – **GICS Sub-Industry** – **Headquarters Location**Display the first five rows of the DataFrame.### 3. Add Detailed Information to the S&P 500 DataFrameUsing the `get_company_info` function you created, retrieve additional data for each company in the list:a) Loop through each company in the DataFrame from the previous step, accessing its individual Wikipedia page to get the detailed infobox data, and add it as new columns to your main DataFrame. Once complete, export your updated DataFrame with the additional columns to a CSV.### Submission ———-Submit your completed assignment according to the submission instructions provided [here](/Exercises/homework_submission_instructions.md) by Wednesday, November 13 at 5pm. Ensure your submission includes:– A jupyter notebook with code and suggested outputs. – The CSV file with the collected data.
In this assignment, you will interact with web APIs using Python’s `requests` package. You’ll choose an API, fetch data from it, and then write a simple API client function to make this task easier.Writing a simple API client ============================### Choose your APITo accommodate your different interests and data needs, this exercise asks you to choose a data API yourself. It is perfectly fine if the data collection from such an API is useful for another class, project, or your thesis.Here are a couple of suggestions for API choices:**Select one API of your choice**:– A huge list of ~40k APIs – you should have no trouble finding one you have some interest in! https://rapidapi.com/hubHere are some fun ones:– **NASA API**:– **Description**: Offers access to a wide range of NASA data, including asteroid information, and Mars Rover photos. – **API Documentation**: [NASA API](https://api.nasa.gov/) – **Example Data**: Asteroid information, Mars Rover photos.– **IP Geolocation API**:– **Description**: Provides information about an IP address, including country, city, timezone, latitude, and longitude. – **API Documentation**: [IP Geolocation API](https://ipgeolocation.io/) – **Example Data**: Location information based on IP address.– **PokéAPI**:– **Description**: Provides data on Pokémon, including information on species, abilities, and moves. – **API Documentation**: [PokéAPI](https://pokeapi.co/) – **Example Data**: Pokémon species, abilities, moves, types, etc.– **Open-Meteo API**:– **Description**: Offers global weather forecast API for non-commercial use. – **API Documentation**: [Open-Meteo](https://open-meteo.com/) – **Example Data**: Temperature, wind speed, precipitation, etc.– **Rick and Morty API**:– **Description**: Provides information about the characters, episodes, and locations from the “Rick and Morty” TV show. – **API Documentation**: [Rick and Morty API](https://rickandmortyapi.com/) – **Example Data**: Characters, episodes, and locations details.– **JokeAPI**:– **Description**: Serves programming jokes and general jokes through a simple endpoint. – **API Documentation**: [JokeAPI](https://v2.jokeapi.dev/) – **Example Data**: Text-based jokes categorized by type.– **REST Countries API**:– **Description**: Provides information about countries, such as name, population, area, and languages spoken. – **API Documentation**: [REST Countries](https://restcountries.com/) – **Example Data**: Country details including population, area, languages, etc.– **ExchangeRate-API**:– **Description**: Provides exchange rate information between different currencies. – **API Documentation**: [ExchangeRate-API](https://www.exchangerate-api.com/) – **Example Data**: Currency conversion rates.… or any other data API you can find on the internet.### TaskThe task is to use the `requests` package (do not use possibly existing specific python packages for the API) to:– interact with the API – create a dataset with multiple records by requesting data from the API using the `requests` package – write a simple API client, i.e. a function that would make downloading/interacting with the API simple for a Python user who does not know `requests`#### 1. Choose an APIa) For this assignment, select a web Application Programming Interface (API). You can choose from the suggestions above or find one that interests you. Ensure that the API you choose is not one we’ve already covered in class (e.g., NYTimes, Github, etc.).#### 2. Authenticationa) Briefly explain how the API authenticates the user. b) If required, apply for an API key and explain how others can do the same (with relevant URL). Do **not** include your API key in your submission.#### 3. Send a Simple GET requesta) Use the `requests` package to send a GET request and fetch a small amount of data from your API. Describe and use a few query parameters in your request. If you have a choice of the output the API returns (e.g. XML or JSON), I suggest to choose JSON because it easier to work with. Your output here should include the code for the `GET` request, including the query parameters, as well as a snippet of the output.b) Check and display the status of your request.c) Identify and display the type of the response (e.g., JSON, XML, CSV).#### 4. Parse the response and create a dataseta) Convert the API response into a usable Python object (e.g., list, vector, pandas data frame). Show the code how this is done.b) Use the API to create a dataset with multiple records (sample size > 100). Include some interesting features.c) Provide summary statistics of your dataset. Include the data frame in a .csv file named `data.csv` with your submission.#### 5. Write an API Client Functiona) Wrap your code from the previous sections into a simple API client function. This function should:– Allow users to specify query parameters. – Run a `GET` request with these parameters. – Check the request’s status and inform users of any errors. – Parse the response and return a Python object (list or data frame). – Include docstrings explaining the parameters, output, and a usage example. Run the function with default values and display the output.Note: There is no need to make this into an Python package here. A simple function is sufficient.In the notebook, include your full function to access the API functionality. Set some sensible default values for the query parameters.Run the function with default values and display the output.For this part of the question, I am not expecting a full-fledged API client. Rather, I want you to wrap some of the code from the previous questions into a function and generalize a bit.### SubmissionPlease follow the [instructions](/Exercises/homework_submission_instructions.md) to submit your homework. The homework is due on Wednesday, November 6 at 5pm.
In this exercise, you will use `pytest` to create and test functions that manage the state of a Tic-Tac-Toe game. You will implement functions to initialize the game board, make moves, check for a win or a draw, and reset the game.Below are the definitions of the `initialize_board`, `make_move`, `check_winner`, and `reset_game` functions you’ll be working with:“`python def initialize_board(): “””Creates a 3×3 Tic-Tac-Toe board initialized with empty spaces.””” return [[‘ ‘ for _ in range(3)] for _ in range(3)]def make_move(board, row, col, player): “”” Places the player’s symbol (‘X’ or ‘O’) on the board at the specified position.Args: board (list): The current game board. row (int): The row index (0-based). col (int): The column index (0-based). player (str): The player’s symbol (‘X’ or ‘O’).Returns: bool: True if the move was successful, False otherwise. “”” if board[row][col] == ‘ ‘: board[row][col] = player return True return Falsedef check_winner(board): “”” Checks the current board for a winner.Args: board (list): The current game board.Returns: str: ‘X’ or ‘O’ if there is a winner, ‘Draw’ if it’s a draw, or None if the game is ongoing. “”” # Check rows, columns, and diagonals lines = board + list(zip(*board)) # Rows and columns lines.append([board[i][i] for i in range(3)]) # Main diagonal lines.append([board[i][2-i] for i in range(3)]) # Anti-diagonalfor line in lines: if all(cell == ‘X’ for cell in line): return ‘X’ if all(cell == ‘O’ for cell in line): return ‘O’# Check for draw if all(cell != ‘ ‘ for row in board for cell in row): return ‘Draw’return Nonedef reset_game(): “””Resets the game by reinitializing the board.””” return initialize_board() “`### 1. Test the `initialize_board`, `make_move`, `check_winner`, and `reset_game` Functions#### a) Write a test function `test_initialize_board` that verifies the `initialize_board` function creates an empty 3×3 board.#### b) Write a test function `test_make_move_valid` that checks whether make_move successfully places a player’s symbol on an empty cell. Test this for both players ‘X’ and ‘O’._Hint:_ start with a pre-specified board configuration and then add a valid move (using the `make_move()` function), assuring that is done correctly.#### c) Write a test function `test_make_move_invalid` that ensures `make_move` does not overwrite an occupied cell and returns `False` when attempting to do so.#### d) Write an integration test `test_game_integration` that performs a series of operations: initializing the board, making multiple moves, checking for a winner, and resetting the game. After each operation, verify the state of the board and the game status (i.e. if there is a winner).#### e) Add the tests from 1a, 1b, 1c, and 1d to a file called `test_tictactoe.py` in your folder. Run all the tests from your command line using `pytest` (with verbosity setting `-vv`) and include the output in your homework solution.### 2. Advanced Testing#### a) Utilize the `@pytest.mark.parametrize` decorator to create a parameterized test function `test_make_move` that tests multiple scenarios of making moves, including edge cases such as invalid row or column indices, and placing a marker in an already occupied spot._Hint_: In this test, you’re verifying the behavior of the `make_move` function under different scenarios. To do this, you’ll need to create specific board configurations (i.e., `initial_board`) to test whether the move works as expected.Think of each test case as setting up a board in a known state and then trying to make a move on that board:1. Empty Board: A completely empty board where any valid move should succeed. 2. Occupied Cell: A board where one cell is already taken. Try to make a move on that cell and check if it returns `False`. 3. Valid Move: A board where you attempt to place a move on a valid, empty cell. 4. Out-of-Bounds Moves: Consider moves where row or col is outside the 3×3 grid (e.g., 3, 3 or -1, 0). These should raise an IndexError.For each test case, start with a specific `initial_board`, then use the `make_move` function with a given `row`, `col`, and `player`. Based on the scenario, check if the move is successful (or not) and if the board is updated correctly.Here is an example of an `initial_board` configuration: “` [[‘X’, ‘ ‘, ‘ ‘], [‘ ‘, ‘ ‘, ‘ ‘], [‘ ‘, ‘ ‘, ‘ ‘]] “`#### b) Use the pytest `fixture` functionality to create a common setup for a game board that will be used in multiple test functions. Ensure that this fixture initializes a fresh board before each test to maintain test independence.## Submit Your Homework– Ensure that your initialize_board, make_move, check_winner, and reset_game functions are correctly implemented and handle all edge cases.– All test functions should be commented, explaining the logic where necessary.– Ensure that the `pytest` output is properly captured and included in your submission.– Your submission should include the `tictactoe.py` and `test_tictactoe.py` files, along with a Jupyter notebook containing the `pytest` output.Please follow the [instructions](/Exercises/homework_submission_instructions.md) to submit your homework on Github. The homework is due on Wednesday, October 23 at 5pm.
We’re exploring a dataset containing narratives from ~6k aviation incident reports from 2021 to 2024 from the National Transportation Safety Board (NTSB) Aviation Incident Database. Your goal is to extract structured information from unstructured text data using regular expressions and perform basic analysis of aviation incidents.Dataset OverviewThe dataset `ntsb_narratives_2021_2024.csv` contains textual descriptions of aviation incidents. The narratives may contain details like aircraft models, pilot experience, airport codes, and engine failures. For example:“`text On September 11, 2024, about 0940 mountain daylight time, a Cessna 172S airplane, N20747, was substantially damaged when it was involved in an accident near Meridian, Colorado. The pilots sustained minor injuries. The flight operated under the provisions of Title 14 Code of Federal Regulations Part 91 as an instructional flight.The airplane was performing touch and go landings to runway 17R at the Centennial Airport, Englewood, Colorado. When on climb out for the traffic pattern, the airplane was about 200 ft above ground level when the pilots noticed a partial loss of engine power. The instructor reported the engine RPMs to read 2,100 and then 1,600. He performed a forced landing to a golf course and during the landing roll, the airplane’s nose landing gear collapsed, and airplane came to rest inverted.The airplane was retained for further examination. “`## Exercise### 1. Engine Incidents and Failures#### a) Extracting Engine IncidentsFind narratives mentioning the word “engine” (or “engines”). Create a boolean column engine (called `engine`) that marks incidents as engine-related or not. Show the count of engine-related narratives._Note_: To simplify, there is no need to consider alternative terms like “Powerplant, Motor, Turbine, Propeller, Jet.” We’ll just focus on `engine` but do account for plural/singular and capitalization.#### b) Use regular expressions to specifically extract narratives that describe engine failures, differentiating them from general engine incidents.Create regex patterns to capture terms like `engine failure`, `engine stopped`, or `failed engine`. Consider the two (2) words before or after the term “engine” (or its variations) to identify narratives describing failure._Note_: No need to go overboard to be fully comprehensive but do show your abilities to employ the following regular expression techniques: – finding alternatives – handling optional words – identify word boundaries – account for singular/plural and capitalizationCreate a boolean column called `engine_failure` that marks engine related incidents as engine failures or not.Provide summary statistics of what proportion of the narratives in the dataset contain a mention of the engine, and what proportion of these mentions you identified as engine failures.#### c) Randomly select 10 entries from the dataset. Display the portion of the narratives with the 2 words before and after the mention of engine, alongside your two new variables `engine` and `engine_failure`. Briefly discuss how this identification process could be improved. Mention potential issues with false positives or missed entries.### 2. Extracting the Time of Day#### a) Use regular expressions to extract the time of day from the narratives. Look for patterns such as: – “0940” – “14:30” – “2:15 PM”– Your regex should be able to identify various time formats including: – **24-hour format** (e.g., 0940, 14:30) – **12-hour format** with AM/PM (e.g., 2:15 PM, 11:00 AM)– Extract the time and create a new column `time_of_day` that stores the time in a uniform format (e.g., 24-hour format). If no time is mentioned in the narrative, leave the field blank._Hints_: – Use capturing groups in your regex to extract the hours and minutes. – For 12-hour formats, ensure that you correctly convert the time to 24-hour format (e.g., 2:15 PM should become 14:15).Show the first 5 entries where the time_of_day is successfully extracted. Provide a summary statistic on how many of the narratives contain a time mention.#### b) Analyze the relationship between time of day and engine failures. Create a bar plot where the x-axis represents the hour of day and the y-axis represents the proportion of engine failures (as a share of all mentions of the aircraft model). Comment briefly on what you found.### 3. Identifying Aircraft Models for a Common Manufacturer##### a) Use regular expressions to search for mentions of the following 10 manufacturers in _all_ incidents: `Cessna, Piper, Beechcraft, Boeing, Airbus, Cirrus, Robinson, Embraer, Bombardier, Mooney`. Count how many times each manufacturer is mentioned and show the information in a table, alongside the share of engine failures._Hint_: To simplify, there is no need to account for misspellings or abbreviations of manufacturer names.#### b) For the single most common aircraft manufacturer identified in (3a) only, create a regular expression to extract aircraft models related to that manufacturer. For example, for Cessna, you might expect models like “Cessna 172”, “Cessna 182”, “Cessna 150” (but do account for variations like “Cessna 172”, “Cessna-172”, “172 Cessna”), for Piper the models may be of the format “Piper PA-28, Piper PA-32”.Count how many times each model is mentioned in the dataset for the manufacturer and create a table showing the model number, number of mentions, share of engine related incidents, and share of engine failures.Note: Throughout the assignment, ensure that you handle variations in wording, capitalization, and abbreviations. Use regular expressions efficiently to capture as many relevant cases as possible. Comment your code to explain your logic and any assumptions made.## Submit your homeworkPlease follow the [instructions](/Exercises/homework_submission_instructions.md) to submit your homework. The homework is due on Wednesday, October 16 at 5pm.
You’re collaborating with a local animal shelter to modernize their animal management and adoption tracking system. They need a foundational system for managing their animals and facilitating adoptions effectively.## 1. Animal Class### a) Create `Animal` classCreate a class called `Animal`. Initialize it with:– `name` (string) – `species` (string) – `age` (integer: the animal’s age in years)Demonstrate the initialization of a single `Animal` object and print the object’s attributes.### b) Add `__str__` method to the `Animal` classAdd a method to `Animal` that returns a string with the following format:“`plaintext ” the is years old.” “`Create an `Animal` object and print its string representation using the `__str__` method.### c) Create `Animal` objectsCreate 6 `Animal` objects using the following data and print their string representations:“`json animals_data = [ {“name”: “Buddy”, “species”: “Dog”, “age”: 3}, {“name”: “Whiskers”, “species”: “Cat”, “age”: 2}, {“name”: “Chirpy”, “species”: “Bird”, “age”: 1}, {“name”: “Nibbles”, “species”: “Rabbit”, “age”: 4}, {“name”: “Goldie”, “species”: “Fish”, “age”: 1}, {“name”: “Spike”, “species”: “Lizard”, “age”: 5} ] “`### d) Add Adoption StatusAdd a method to `Animal` called `adopt` that marks the animal as adopted. Ensure that an animal cannot be adopted more than once.“`python def adopt(self): # Mark the animal as adopted if it isn’t already “`Demonstrate this method by adopting “Whiskers” and printing the updated string representation.## 2. Shelter Class### a) Create `Shelter` classCreate a class called `Shelter` that takes a list of `Animal` objects as an argument during initialization.Demonstrate the initialization by creating a `Shelter` object using the animals created in 1c) and print the object’s attributes.### b) Check Animal AvailabilityAdd a method to `Shelter` called `check_availability` that takes an animal species as an argument and returns a boolean indicating whether there is an available animal of that species.Demonstrate this method by checking the availability for “Dog” and print the result.### c) List Available Animals by SpeciesAdd a method to `Shelter` called `list_by_species` that takes a species name as an argument and returns a list of all available (i.e., not yet adopted) animals of that species.Demonstrate this method by listing all available “Cat” species.## Bonus Question (for additional points): Implementing a Weight Monitoring System– Modify the `Animal` class to include a `weight` attribute (float) and an `update_weight` method to change the animal’s weight. Ensure the weight cannot be negative.– Add a method `average_weight_by_species` to the `Shelter` class that takes a species name and returns the average weight of all available animals of that species.Demonstrate this system by calculating and printing the average weight for “Dog” and “Fish” species.## Submit your homeworkPlease follow the [instructions](/Exercises/homework_submission_instructions.md) to submit your homework. The homework is due on Wednesday, October 9 at 5pm.
Find a complete social network, preferably one with at least some attributes about the nodes with it. (If you simply have a social network, but no real attributes, you will need to pick an additional network to compare that first one to.) 1. Describe the social network(s) to me, in terms of how it was collected, what it represents and so forth. Also give me basic topography of the network: the nature of the ties; direction of ties; overall density; and if attributes are with the network, the distribution of the categories and variables of those attributes. 2. Calculate degree centrality (in- and out-degree, too, if you have such data); closeness centrality; betweenness centrality; and eigenvector centrality. Correlate those measures of centrality. Highlight which nodes are most central and least central, along different dimensions. Now, do 1 of the following, but not both: 3a. If you have a network with attribute data, then state some hypothesis about how an attribute may be related to some (or all of the) measures of centrality. Explains why you think these two variables should be related. 3b. If you don’t have a network with attribute data, then pick another network to compare your first network against. Calculate all of the same measures as above for Network #2. Consider if normalization is appropriate for any of these measures. Then state some hypothesis about why some (or all of the) measures of centrality in one network will be the same or different from the second network. Explain why you think these two networks should be similar or different. 4. In either case, when you are done above, then considers alternate specifications of your variables and codings and decisions and models. What would you want to consider changing and why. If you can, report on what are the consequences of those changes? 5. Lastly, give your best conclusion as to what you learned from your analysis. Did it make sense, given your initial expectations? Why? Why not?
QMSS- Networks Lab Report #3 Find a complete social network, preferably one with at least some attributes about the nodes with it. 1. Describe the social network(s) to me, in terms of how it was collected, what it represents and so forth. Also give me basic topography of the network: the nature of the ties; direction of ties; overall density; and if attributes are with the network, the distribution of the categories and variables of those attributes. 2. Run the Girvan-Newman community detection algorithm. Then run the random walk community detection algorithm. 3. Tell me how many groups each algorithm finds. Analyze how similar the two partitioning algorithms are in terms of putting nodes into groups with each other. 4. Visualize the network (either in R or Gephi), coloring the nodes by either Girvan-Newman grouping or the random walk grouping. 5. Tell me anything else about whether the partitioning makes sense, based on attributes or who the nodes are, and so on.
Networks Lab #1 Assignment 1. Develop a hypothesis about how some ego-network measure (e.g., degree/size, density, diversity, average-level of alters, homophily, structural holes, or brokerage) may be related to some other variable of interest. 2. Explain why you think these two variables should be related. 3. Tell me about your variables. What is your dependent variable? What are your independent variables? How are they coded? How are they recoded? How are the calculated, if appropriate? 4. Present your initial results from your first few models. What do they indicated about your hypothesis? 5. Consider alternate specifications of your variables (i.e., recodings of various kinds). Consider interactions among your variables. 6. And give your best conclusion as to whether your initial hypothesis held up – and if not, why not.
Choose 1 of the following options: Fixed/Random Effects 1. (a) Run an OLS regression, including at least one independent variable and a time variable (as dummies). Explain how you think your independent variable relates to your dependent variable. Interpret your results. Did you find what you expected to find? (b) Then run a fixed effect model version of that OLS model. Interpret your results. Did you find what you expected to find? Why? Why not? (c) Then include an additional predictor in your fixed effects model that you think might account for the initial relationship you found between your X and your Y. What effect does that new independent variable have in your new regression? (d) Then run a random effects model equivalent to your fixed effects model in step (b). Interpret the results. (e) Run a Hausman test to compare your fixed effects and your random effects models. What do you conclude? SEM Models ~ SEM two-way cross-lagged model 1- Run usual OLS models with one variable as independent variable and one as dependent variable, and vice versa. Explain what relationships you find. 2-Run an SEM two-way cross-lagged model. Explain whether either of the cross-lags are significant. Interpret your results. 3- Look at some fit measures when multiple waves are constrained to be equivalent, etc. Survival Analysis Run a multiple variable survival analysis. You can perform the survival analysis either using discrete-time methods (i.e., event history analysis) or you can use Cox proportional hazards methods, either one is fine. (a) State what your “failure” variable is and how you expect your independent variables to affect it. (b) Explain how you determined the #risk window” (due to right truncation and left-censoring) and who is eligible for failure over the time you are studying. (c) Explain whether the results were consistent with your expectations, and do that by interpreting the coefficients from the models, model fit, and so on. FE/RE = Student runs an OLS regression, including at least one independent variable and a time variable (as dummies) // Survival = Student runs a multiple variable survival analysis // SEM = Student chooses two variables to be related to each other and explain why they might be reciprocally related RE/FE = Student explains how they think their independent variable relates to your dependent variable and interprets the results. // Survival = They explain whether they are using discrete-time methods (i.e., event history analysis) or use Cox proportional hazards methods. / SEM = Student runs usual OLS models with lagged dependent variable models for each variable. RE/FE = Student runs a fixed effect model version of that OLS model. // Survival = State what their “failure” variable is // SEM = Student interpret the OLS models RE/FE = Student interprets the fixed effects output. // Survival = Student states how they expect their independent variables to affect it. // SEM = = Student creates an SEM model on two waves of data that allows for these variables to mutually affect each other, and explain how you made it. RE/FE = Student then runs a random effects model equivalent to the fixed effects model. // Survival = They explain how you determined the “risk window” (due to right truncation and left-censoring) and who is eligible for failure over the time you are studying. // SEM = Student runs the SEM model. RE/FE = Student interprets the random effects output. // Survival = Student explains whether the results were consistent with their expectations, by interpreting the coefficients from the models // SEM = Student interprets the key statistics and coeffcicients. RE/FE = Student runs a Hausman test to compare their fixed effects and their random effects models. // Survival = Student also talks about model fit at least, if not other measures too. // SEM = Student states how the two variables look like they influence each other? Overall, the student’s work is error-free, interesting and of high quality.
1.Create a multivariate time series; perform any interpolations. 2.Graph the relationships between X and Y. Explain how you think Y should relate to your key Xs. 3.Run a simple time series regression, with one X and no trend. Interpret it. 4.Run a time series regression with one X and trend. Interpret it. Perform autocorrelation diagnostics. Explain what you found. 5.Consider running a time series regression with many Xs and trend. Interpret that. Check VIF. 6.Run a first differenced time series regression. Interpret that. 7.Check your variables for unit roots. Do some tests. Interpret them. 8.Performan Automatic ARIMA on the residuals from one of your earlier models. Tell me what it says. 9.Run an ARIMA that follows from Step 8. Interpret that, too.
Choose 1 of these assignments: 1. Conduct a trend analysis of some variable of interest. Graph it and try different functional forms. Look for subgroup variation across time, too. Extra credit if you consider other variables as a means of explaining the trend. Explain all of your results. 2. Conduct an unpooled regression comparison across time-periods. Compare at least two time periods, running regressions on each and comparing coefficients. Explain your results. 3. Conduct an age-period-cohort analysis. Make plots of the different dimensions. Develop a parsimonious model of what you think is going on. Explain your results. 4. Conduct a difference-in-difference analysis. Explain which group is treated and which is not. Provide pre-treatment trends and generate a regression summarizing your results. Explain what you found. Student picks some variables of interest. 7.14 pts Full Marks 0 pts No Marks Student explains what variables they picked. 7.14 pts Full Marks 0 pts No Marks Trend: Student graphs the variable across time // Unpooled: Student runs regression for one time period // DiD: Student explains which group is treated // APC: Student makes plots of various APC dimensions 7.14 pts Full Marks 0 pts No Marks Trend: Student runs a regression // Unpooled: Student runs regression for another time period // DiD: Student explains which group is the untreated // APC: Student makes additional plots of remaining APC dimensions 7.14 pts Full Marks 0 pts No Marks Trend: Student explains the regression // Unpooled: Student interprets at least some of the coefficients // DiD: Student provides analysis and plot of pretreatment trends // APC: Student runs a parsimonious regression of age period cohort variables 7.14 pts Full Marks 0 pts No Marks Trend: Student tries one other functional form (e.g., quadratic, dummies) // Unpooled: Student formally compares coefficients between time periods // DiD: Student generates a regression summarizing their results // APC: Student interprets the results of the regression 7.14 pts Full Marks 0 pts No Marks Trend: Student explains the result of the functional form and/or subgroup analysis // Unpooled: Student explains their results // DiD: Student interprets their results // APC: Student explains why their parsimonious model makes sense 7.14 pts Full Marks 0 pts No Marks Overall, the quality of the student’s report was well-done, without issues, followed the directions and reflects time, effort and thought. 50 to >0.0 pts Full Marks 0 pts No Marks Send
1. Run a multiple linear probability model (have at least 2 Xs in the model). Tell me how you think your independent variables will affect your dependent variable. Interpret your results. Were your expectations correct? Why or why not? 2. Run a multiple (binary) logistic model. (It can be the same as the above LPM or a new model.) If it is a new model, tell me how you think your independent variables will affect your dependent variable. Interpret your results in the logit scale. Were your expectations correct? Why or why not? 3. Get odds ratios from your logit model in Question 2 and interpret some of them. 4. Get predicted probabilities from your logit model in Question 2 for some constellations of X values and interpret the results. Run a multiple linear probability model (have at least 2 Xs in the model). Tell me how you think your independent variables will affect your dependent variable. Interpret your results. Were your expectations correct? Why or why not? 20 pts Full Marks 0 pts No Marks Run a multiple (binary) logistic model. (It can be the same as the above LPM or a new model.) If it is a new model, tell me how you think your independent variables will affect your dependent variable. Interpret your results in the logit scale. Were your expectations correct? Why or why not? 20 pts Full Marks 0 pts No Marks Get odds ratios from your logit model in Question 2 and interpret some of them. 20 pts Full Marks 0 pts No Marks Get predicted probabilities from your logit model in Question 2 for some constellations of X values and interpret the results. 20 pts Full Marks 0 pts No Marks The work is done well with a show of a lot of effort. 20 pts Full Marks 0 pts No Marks
