In this assignment, you create a NoSQL database of movies and their information. The movies data are directly extracted from assignment 2 and transferred into the NoSQL database.Implementation Platform We use Neo4J [1] in this assignment. While you may find many tutorials online, attending the tutorials sessions are strongly recommended. For any help re: programming, or questions on the platform, please see PODs.The data transfer is done by converting the data from each relation from the RDBMS into a csv (or tsv) or json data. You will then use the data and directly import it into Neo4j. • https://neo4j.com/developer/guide-import-csv/ • https://neo4j.com/labs/apoc/4.1/import/load-json/Entities / Nodes In this assignment you creating the following entities (nodes) and populate the data. • Movies (attributes: title, plot, content rating, viewer rating, release year, genres, languages, AKAs, and optional watchmode id) • Actors (first name, last name) • Countries • KeywordsData Files and Scripts [10 pts] Extract the data from your database into the data file (csv, tsv, or json1 ). [40 pts] Write scripts to create the database and populate the data in Neo4J. Note that Neo4J supports array attributes, which are normally represented using weak entities in relational model: https://neo4j.com/docs/cypher-manual/current/functions/list/.To populate the such data (i.e. genres, languages), you may use a separate csv file. 1Using JSON is not recommended, but is permitted. Naturally, a relational data may be directly represented using a tabular data format such as CSV, TSV, etc.Provide the answers to the following: A) [5 pts] Find all movies that are played by a sample actor. B) [5 pts] Find the number of movies with and without a watch-mode info. C) [5 pts] Find all movies that are released after the year 2023 and has a viewer rating of at least 5.D) [5 pts] Find all movies with two countries of your choice. Make sure your query returns more than one movie. List movies that may be associated with either of the countries (not necessarily both). E) [10 pts] Find top 2 movies with largest number of keywords.F) [10 pts] Find top 5 movies (ordered by rating) in a language of your choice. G) [5 pts] Build full text search index to query movie plots. H) [5 pts] Write a full text search query and search for some sample text of your choice. Make sure all above queries return data. Modify the data in your database, if necessary.Submit your assignment electronically on Moodle: https://moodle.concordia.ca Include your name and student ID in the submission. Make sure that you upload the assignment to the correct assignment box on Moodle. No email submissions are accepted. Assignments uploaded to the wrong system, wrong folder, or submitted via email will be discarded and no resubmission will be allowed. Make sure you can access Moodle prior to the submission deadline. The deadline will not be extended.References 1. https://neo4j.com/
In this assignment, you create a local database of movies and their information. The assignment targets writing SQL queries for data de nition, data collection, as well as search queries.While IMDB [1], TMDB [2], or any other public movie database may be used as the main source of the data, in this assignment you are required to create your own database and populate the data. The data population may be done through public APIs [3] such as IMDbOT [4]. Watchmode [5] provides a mapping between IMDB and TMDB IDs as well as streaming information. Most APIs return requested data in JSON format. Using a JSON Visualizer [6] would be extremely helpful in locating the info items that are returned by the API.Example: GET https://search.imdbot.workers.dev/?tt=tt0068646You may optionally start with some pre-populated database (see ref. [7]), however, your database must strictly follow the following design. Movies are searchable by TMDB-ids and IMDB-ids. Both may be used for alternate search keys. Using TBDM-id is required, however, IMDB-id is optional.It is recommended that you use your own primary key. For each movie, store: title, plot, content rating (certi cates), viewers rating (a oating point number, between 0 and 10, with 1 decimal digit after the decimal point), genres, actors, directors, release year, AKAs, countries, languages, keywords. Also, store the watchmode id for each movies. In theory not all movies have a watchmode id.Make sure your database only contains few movies with watch-mode id. Note that both IMDbOT and the above Sample Movies Database are given as examples.You may nd alternatives sites on the internet. You are allowed to use any data source that ts the purpose. Implementation Platform In this assignment, you use PortgreSQL [8] to implement the database tables.Provide answer to each of the following parts in separate .sql les. Part 1. [20 pts] Provide the DDL queries for creating the database tables that correspond entities and their relationships. Use internal integer primary keys (no TMDB/IMDB id). Include referential integrity / unique constraints. Include full country name and short code for countries. Create foreign tables for genres, content rating, and keywords. Use singular names for tables / relations. Make sure your database include a few records with missing IMDB-ids. You may manually delete the IMDB data and/or use fake information.For the data population of the data, demonstrate: Part 2. [15 pts] Consume a web service, either via a script or a small code that performs get operations for each movie in question. Part 3. [15 pts] Create database instance via a database client (i.e. Psycopg2 for python [9]). You may chose any programming language as you wish. Using Python is not mandatory.Part 4. [7 pts] Generate the ERD of your database using a reverse engineering tool i.s. pdadmin 10.Database DDL Part 5. [8 pts] Provide the DML queries that populates the data in the database. You may back up your database as SQL (see pddump [11] as an example). Make sure you create at least 50 movies. Remove unnecessary statements from your sql dump le. Note that the graders will run the DDL queries in the previous section to create your database. Make sure the queries are provided in correct order.Use of Views Part 6. [7 pts] Create the following view namely `movie-summary’ that displays tmdb key, imdb key, title, description / plot, content rating, runtime, number of keywords, number of countries. Provice the DDL SQL.Part 7. [28 pts] Provide the following queries: A) [4 pts] Find the total number of movies with and without IMDB id in the database. Use one query. B) [4 pts] Pick an actor. Find all movies by that actor that is released between 2000 and 2020. List TMDB-id, IMDB-id, movie title, release date, and watchmode-id.C) [4 pts] Find movies that have highest number of reviews. List top 3. D) [4 pts] Find number of movies that are in more than one language. E) [4 pts] For each language list how many movies are there in the database. Order by highest rank. F) [4 pts] Find top 2 comedies (higher ratings). G) [4 pts] Write a batch-update query that rounds up all the ratings. IMPORTANT: Do not run the query. The grader will run the query and verify the result.Submission Make sure all above queries return data. Modify the data in your database, if necessary. Submit your assignment electronically on Moodle: https://moodle.concordia.ca Include your name and student ID in the submission. Make sure that you upload the assignment to the correct assignment box on Moodle. No email submissions are accepted. Assignments uploaded to the wrong system, wrong folder, or submitted via email will be discarded and no resubmission will be allowed. Make sure you can access Moodle prior to the submission deadline. The deadline will not be extended.References 1. https://www.imdb.com/ 2. https://developer.themoviedb.org/reference/search-movie 3. https://github.com/public-apis/public-apis 4. https://github.com/SpEcHiDe/IMDbOT 5. https://rapidapi.com/meteoric-llc-meteoric-llc-default/api/watchmode/details 6. https://jsonviewer.stack.hu/ 7. https://www.databasestar.com/sample-database-movies/ 8. https://www.postgresql.org/ 9. https://www.freecodecamp.org/news/postgresql-in-python/ 10. https://www.pgadmin.org/docs/pgadmin4/development/erd_tool.html 11. https://www.dbvis.com/thetable/a-complete-guide-to-pg-dump-with-examples-tips-and-tricks/
1. [30 Points] Classify the following languages into one of the three categories a) regular, b) context-free but not regular, and c) not context-free. Prove your answer. (a) L1 = {a i b j c k | k = i × j and 0 < i < 10 < j} (b) L2 = {xyz | x, y, z ∈ {a, b} ⋆ and na(x) = nb(z)}(c) L3 = {wuwR | w, u ∈ {a, b} ⋆ and |w| = |u|} Note that in order to show that a language is context-free but not regular, you need to prove both that it is context-free and also that it is not regular.2. [10 Points] Give a Turing machine for L = {a}·{a, b} + that does not halt on rejection.3. [20 Points] Give a Turing machine for each of the following languages: (a) L1 = {a n b mc k | m ≥ n, k ≥ 1}. (b) L2 = {xy | x ∈ {a, b} +, y ∈ {c} + and na(x) = nc(y)}.4. [20 Points] Draw transition diagrams for Turing machines that compute the following functions. In each case, give a brief description in English of your solution strategy. (a) f(1n ) = 12n (b) f(1n ) = 1n 2
Q 1. [40 pts] Consider the following system description for a museum information systems: A museum has multiple departments.A museum has collections. A collection belongs to a department.Art-objects belong to a collection. An art-object has a unique acquisition number, object type, title, description, dimension, production date, and production country. Examples of object types are paintings, sculptures, etc. An art-object may be linked to an artist. Some art-objects are just without one. An art-object belongs to a period (i.e. Roman Empire).An exhibition takes place in a museum which exhibits various art objects from various collections. An exhibition has a start and end date, as well as a home department. Art objects may be borrowed temporarily from various museums for a specific exhibition.A) [20 pts] Design a data model that implements the above description. Identify the entities, relations, attributes, keys, and their cardinalities. Identify strong and weak entities and relations, if applicable. Represent the end result in the form of an ERD diagram. single arrow – no arrow notation for cardinalities and single line – double line for partial and total participations [??].B) [10 pts] Provide the DDL script for the above. C) [10 pts] Provide a database instance that demonstrate an exhibition with local (from home department) and borrowed art objects.Q 2. [40 pts] Consider the following system description to manage group or private lessons: Consider an organization that offers group or private lessons (of various types, such as yoga, swimming, etc.) to clients.The organization owns or rents space (gyms, rooms, swimming pools) in various locations in possibly different cities across the province and each location is made available over some given schedule, where a schedule is essentially a sequence of day-time slots. E.g. “The EVBuilding gym Room 7, in Montreal, is available for Judo classes on Sundays from 12 noon to 3PM, from September 1st to November 30th, 2024.” The length of a time slot is not fixed (and you are free to create schedules as you see fit). For example, a swimming lesson would normally be for half hour, but a Judo class would normally be for one hour.A given location can only accommodate one lesson at any given time slot on a given day. The same type of lesson can be offered in both modes (private, group). For example, at the same location the organization would offer private swimming lessons and group swimming lessons.The organization does not have permanent instructors, but it hires seasonal instructors of various specializations. An instructor would register with the system by entering their credentials (name and phone number) and specialization as well as to register their availability in one or possibly several cities. For example, “Grace (514 – . . . ) is a swim instructor, available to work in Montreal and Laval.”Once registered, an instructor can subsequently take on possibly several offerings that the organization makes available. All offerings are made available to po2 tential instructors, but only those that have been taken by instructors are made available to the public in order to attract clients. For example, “We offer private and group Judo classes in EV-Building on Sundays from 12PM to 3PM from 1.Sep to 30.Nov as follows: 12:00 – 13:00. Group. Instructor: . . . 13:00 – 14:00. Group. . . . 14:00 – 14:30. Private. . . . 14:30 – 15:00. Private. . . . We offer swimming classes at . . . . . . ”A) [25 pts] Design a data model that captures the above description. Identify the entities, relations, attributes, keys, and their cardinalities. Identify strong and weak entities and relations, if applicable. Represent the end result in the form of an ERD diagram, using chen notation [??].B) [6 pts] Provide the schema and explicitly identify the domains of each attribute. C) [9 pts] Provide a database instance that demonstrate offers to a sample instructor, one of which is accepted, one is rejected, and one is pending response. Demonstrate a group and a private lesson in your example.Q 3. [20 pts] Represent the ERD of either questions in crow’s foot notation using PlantUml [??], [??]. Make sure you submit both PlantUML source and the diagram image. Submit your assignment electronically on Moodle: https://moodle.concordia.ca Include your name and student ID in the submission. Make sure that you upload the assignment to the correct assignment box on Moodle. No email submissions are accepted.Assignments uploaded to the wrong system, wrong folder, or submitted via email will be discarded and no resubmission will be allowed. Make sure you can access Moodle prior to the submission deadline. The deadline will not be extended.References 1. https://jcsites.juniata.edu/faculty/rhodes/dbms/ermodel.htm 2. https://vertabelo.com/blog/chen-erd-notation/ 3. https://vertabelo.com/blog/crow-s-foot-notation/ 4. https://plantuml.com/ie-diagram
1. [10 Points] Show that following grammar G, where S is the starting variable, is ambiguous. S → AB | aaaB, A → a | Aa, B → b. 2. [10 Points] A context-free grammar G = ⟨V, T, S, P⟩ is said to be a simple grammar or s-grammar if all its productions are of the form A → ax, where A ∈ V, a ∈ T, x ∈ V ⋆ , and any pair (A, a) occurs at most once in P. Find an s-grammar for L = {a n b 2n : n ≥ 1}. 3. [10 Points] Give an NPDA with 2 states that accepts L = {a n b n+1 : n ≥ 0}. 4. [20 Points] For each of the following CFLs, give a “direct” design of an NPDA. That is, it is not acceptable to first find a CFG and then convert it into an NPDA. (a) L1 = {a n b 2n+1 : n ≥ 0} (b) L2 = {w ∈ {a, b} ⋆ : na(w) ≤ 3nb(w)} 5. [20 Points] Show that the following CFLs are deterministic. (a) L1 = {(ab) n b(ba) n : n ≥ 0} ∪ {(ab) n b : n ≥ 0} (b) L3 = {w ∈ (a, b) ⋆ : na(w) ̸= nb(w)}
1. [20 Points] For each of the following languages,give a context-free grramar (CFG). (a) (5 Points) La = {a n b m : m, n ≥ 0 and 2n ≤ m ≤ 3n} (b) (5 Points) Lb = {a n b mc k : k = 2m + n}(c) (5 Points) Lc = {a n b mc k : n = m or m ≤ k}(d) (5 Points) Ld = {w ∈ {a, b} ∗ : w ̸= xx, for any x ∈ {a, b} ∗}.2. [10 Points] Consider the language L = {a n+1b n : n ≥ 0}. (a) (5 points) Describe in English the complement L of L. Your description should specify the types of strings that are in L. That is, it is not acceptable to say L includes every string over {a, b} that is not in L, which is obviously true. (b) (5 points) Give a CFG for L.3. [15 Points] Let G be the following CFG in which S is the start variable: S → AB | aB A → aab | λ B → bbA(a) (5 Points) Using the procedure discussed in the class, convert G into an equivalent grammar in Chomsky Normal Form (CNF). (b) (5 Points) Find an equivalent grammar to G in Greibach Normal Form (GNF).(c) (5 Points) Suppose we modify the original grammar G as follows: remove the λ-production A → λ and instead add the unit producion A → A. Let us call the resulting grammar as G′ . Convert G′ into CNF, and simplify, if possible. Also describe in English the language L(G′ ).
1. [20 Points] For each of the following languages over Σ = {a, b}, write a regular grammar and then convert it into an equivalent NFA using the procedure described in class. (a) (10 Points) L(r) where r = ((a + b)(a + b))∗ b + a((a + b)(a + b))∗ ; and (b) (10 Points) {w ∈ {a, b} ∗ : w ends in a and |w| ≡ 1 (mod 3)}.2. [25 Points] Fix an alphabet Σ. For any string w with |w| ≥ 2, let skip(w) be the string obtained by removing the first two symbols of w. Define two operators on languages: f1(L) = {w ∈ Σ ∗ : skip(w) ∈ L}, and f2(L) = {skip(w) ∈ Σ ∗ : w ∈ L}(a) (5 Points) Consider L 0 = L(bba∗ ) over the alphabet Σ = {a, b}. Write a regular expression representing f1(L 0 ). Write another regular expression representing f2(L 0 ).(b) (10 Points) Claim: for every regular language L the language f1(L) is regular. Clearly state whether the claim is TRUE or FALSE, and then prove your answer. (c) (10 Points) Claim: for every regular language L the language f2(L) is regular. Clearly state whether the claim is TRUE or FALSE, and then prove your answer.3. [20 Points] For each of the following languages, use the Pumping Lemma and/or closure properties of regular languages to show that the language is not regular. (a) (10 Points) L1 = {0 k1 ` : k ≥ ` 4 ≥ 0}.(b) (10 Points) L2 = {a n : n is not a perfect cube}.
1. [20 Points] For each of the following languages over the alphabet Σ = {a, b} give an NFA (as a transition diagram) with the specified number of states. Hint: try simplifying a DFA and/or use λ transitions. (a) The language {a n : n ≥ 0} ∪ {b na : n ≥ 1} with at most 4 states.(b) The language {w : w either has no consecutive a’s or no consecutive b’s} with at most 5 states. (c) The language {w : w contains an even number of a’s or exactly two b’s} with at most 6 states. (d) The language {ab, aab, aba} ∗ with at most 4 states.2. [20 Points] Let Σ = {a, b}. Convert each NFA below to a DFA using the subset construction. Draw the transition diagram of your DFA, label the states of your DFA by subsets of states of the original NFA. (a) a a b b λ λ q0 q1 q2 q3 q4 (b) a, b a, b a b b a q0 q1 q2 q33. [20 Points] Find a regular expression for each of the following languages. (a) {ban b m : n ≥ 3, m ≥ 2}. (b) {w ∈ {a, b} ∗ : every maximal substring of w consisting entirely of symbols a is of length exactly 3}.(c) {w ∈ {a, b} ∗ : w does not contain bab as a substring}.(d) {w ∈ {a, b} ∗ : w begins with bb and nb(w) mod 3 = 0}.
Question 1 [5 marks] Write a function that replaces a specific digit in an int number with another digit and returns the result as an int. Return the same number if the target/replacement character is not a digit. Use this function header: int replaceDigits(int number, char target, char replacement) For example: replaceDigits(1, ‘1’, ‘2’) should return 2 replaceDigits(-232, ‘3’, ‘0’) should return -202 replaceDigits(123, ‘4’, ‘5’) should return 123 replaceDigits(1223, ‘!’, ‘?’) should return 1223 replaceDigits(5670, ‘5’, ‘0’) should return 670 (leading zero will not be part of the resulting number) replaceDigits(30400, ‘0’, ‘9’) should return 39499You can assume the number does not have leading zeros (e.g., we will not call replaceDigits(01, ‘1’, ‘2’)), except when the number is actually zero (i.e., we might call replaceDigits(0, ‘1’, ‘1’)). Only include the a1_question.h header file and the function definition (and your helper functions, if any) in the source file and name it as a1_question1.c. Do not use recursion in your answer.Question 2 [5 marks] Write a function that takes in 4 parameters: an int array, its size, the left index, and the right index; and checks if the elements between the left and right index (inclusive) in the int array are sorted in ascending order.If the left index is larger than the right index, swap them first. Then, if the left index is invalid (e.g., negative, larger than the size), use the leftmost valid index of the array; if the right index is invalid, use the rightmost valid index of the array. Use this function header:bool rangedCheckforSorted(int array[], unsigned int size, int leftIndex, int rightIndex) For example (suppose there is a myIntArray created as [-4, 3, -12, 0, 5, 72, 88, 128, 1, 64]): rangedCheckforSorted(myIntArray, 10, -2, 3) will check indexes between 0 and 3, and return false rangedCheckforSorted(myIntArray, 10, 7, 2) will check indexes between 2 and 7, and return true rangedCheckforSorted(myIntArray, 10, 5, 5) will check indexes between 5 and 5, and return true rangedCheckforSorted(myIntArray, 10, 0, 9) will check indexes between 0 and 9, and return false You can assume the array has one or more elements and the size is always correct. Content of the int array does not change after calling this function.Only include the a1_question2.h header file, , and the function definition (and your helper functions, if any) in the source file and name it as a1_question2.c. Do not use recursion in your answer.Question 3 [6 marks] In C arrays can be accessed using pointers. As an exercise, write a function that reverses the polarity of every element in a 2D int array (i.e., -1 becomes 1, 46 becomes -46, …etc.) using only pointers. That is, your code must not use any [ ], which is the non-pointer version of array access. Use this function header: void reversePolarity(unsigned int row, unsigned int col, int** array)For example, suppose we have a 2-by-3 array my2DIntArray with the content [[0, 1, -2], [-999, 2345, 678]]. Then the function will be called as (the array needs to be created differently, see the test file): reversePolarity(2, 3, my2DIntArray) And when the function returns, the content of my2DIntArray becomes [[0, -1, 2], [999, -2345, -678]]. You can assume that the row & col parameters always correctly indicate the number of rows & columns of the array. Do not use [ ] anywhere in your answer (if you do you get 0 for this question). Only include the a1_question3.h header file and the function definition (and your helper functions, if any) in the source file and name it as a1_question3.c.Coding Style [4 marks] Your program should be properly indented, have clear and meaningful variable names (e.g., no singleletter variable names except loop iterators) and enough white space and comments to make it easy to read. Named constants should be used where appropriate. Each line of code should not exceed 80 characters. White space should be used in a consistent manner. Remember to include your information. To help you to get into the habit of good coding style, we will read your code and marks will be deducted if your code is not styled properly.Before You Submit Before submitting your assignment, make sure you have test-run your code in our CSIL machines using the steps covered in class: open your assignment folder in VS Code, use the gcc command with options in the terminal to compile your code, and run the executable. We will use these steps when marking your submission and will deduct marks if your code does not compile/run – to maintain fairness we do not accept reasons like “but it worked on my computer” or “but it ran just fine in my IDE”.The Makefile provided in this assignment is used by a command in the CSIL machines called “make” to quickly compile your code. It is especially useful if you have multiple source files. To use it, type the following command in the prompt (make sure you are in the directory with all the files of Assignment 1): $ make test1The example above illustrates how Question 1 is compiled into an executable called “test1” when using the Makefile. Replace the “test1” with “test2”, “test3”, …etc. for other questions. You can then run the executable by typing ./test1 to test your code for Question 1. If you make changes to your code, use the make command again. You can also use make all if you want to compile all your code at once. The test files (test1.c, test2.c, …etc.) are provided in this assignment for you to test your code. Each typically contains a main function along with other tester functions and/or calls. You can modify them to further test your code, but do not submit these test files because we will be using our test files that are similar but not identical to grade your assignment. This makes sure that your code is not written to produce hard-coded output.The header files (question1.h, question2.h, …etc.) are there to make the compilation work. You can look at them but do not modify them. You also do not have to submit them.Submission Submit only the 3 source files indicated above (a1_question1.c, a1_question2.c, a1_question3.c) to CourSys. Refer to the corresponding Canvas assignment entry for details. Assignment late penalty: 10% per calendar day (each 0 to 24 hour period past due), max 2 days late.Question 1 [6 marks] Write a recursive solution that does exactly the same as what the function in Question 1 of Assignment 1 does. That is, replaces a specific digit in an int number with another digit and returns the result as an int. Return the same number if the target/replacement character is not a digit. Use this function header: int replaceDigitsRecursive(int number, char target, char replacement)For example (for more examples refer to the description in Assignment 1): replaceDigitsRecursive(1, ‘1’, ‘2’) should return 2 replaceDigitsRecursive(-232, ‘3’, ‘0’) should return -202 replaceDigitsRecursive(998, ‘9’, ‘0’) should return 8 (leading zeros will not be part of the resulting number)You can assume the number does not have leading zeros (e.g., we will not call replaceDigitsRecursive(01, ‘1’, ‘2’)), except when the number is actually zero (i.e., we might call replaceDigitsRecursive(0, ‘1’, ‘1’)). This question aims to let you practice recursion. One way to do so is to look at your answer for Assignment 1, separate the recursive case (most likely where you use a loop) from the base case (most likely the last step you do before returning the result). You should be able to see a significant simplification to your code. Do not use any global or static variables, or pass-by-reference (if you do so you get 0 for this question). Depends on how you solve this, you might have to create a recursive helper function that passes an extra parameter preserving the partial answer. IOnly include the a2_question1.h header file and the function definition (and your helper functions, if any) in the source file and name it as a2_question1.c.Question 2 [4 marks] In the header file for this question, a struct called Superhero is defined like this: typedef struct { char* name; short feetInHeight; short inchesInHeight; char* superpower; char* traits; } Superhero; First, write a function that takes in 5 parameters: a pointer to a char array representing a name, two shorts representing feet in height and inches in height respectively, a pointer to a char array representing a superpower, and a pointer to a char array representing the traits; and returns the address of a dynamically (i.e., uses malloc) created Superhero struct variable storing those parameters.Use this function header: Superhero* createSuperhero(const char* name, short feetInHeight, short inchesInHeight, const char* superpower, const char* traits) For example, given the code (name, superpower and traits are Cstrings storing the proper information): Superhero* superhero = createSuperhero(name, 6, 0, superpower, traits); printf(“%s %d’%d” %s: %s ”, superhero->name, superhero->feetInHeight, superhero->inchesInHeight, superhero->superpower, superhero->traits); will result in an output like this: Thunderstrike 6’0″ Weather Control: Harnessing storms, Thunderstrike commands thunder and lightning, using nature’s fury to protect the innocent and fight against evil. You can assume all the height measurements are valid (i.e., feet & inches will not be negative), and all the Cstrings are properly formed ( terminated).Field variables name, superpower, and traits in the struct must be created dynamically and are copies of the parameters, instead of simply pointing to the parameters’ addresses. This is called “deep-copy”. Next, write another function that takes in 1 parameter: the address of a Superhero struct variable; and releases (i.e., uses free) the memory created for the 3 field variables name, superpower, and traits. Use this function header: void clearSuperhero(Superhero* superhero) Note that the parameter can be NULL (if so the function should do nothing). Also, this function does not release the memory used for the struct variable, but only those used by the variable’s fields. To release all the memory dynamically allocated for the struct variable, you should call the free() function with the address of this struct variable right after the function returns. For details read Question 3. Only include the a2_question2.h header file and the function definitions(and your helper functions, if any) in the source file and name it as a2_question2.c. Do not use recursion in your answer.Question 3 [6 marks] In this question you are going to make use of your answer for Question 2 to create a working program (so finish it first). You are also going to write your own main function so the program runs. Create a program that reads all the superhero information from the provided superheros.txt file and prints the information to the screen. Your program must meet these requirements: • Include the header file from Question 2 and use the functions defined there (createSuperhero and clearSuperhero) to complete this question. Do not redefine those functions. • Use a dynamic array of Superhero struct pointers to store the superhero information. This is the recommended approach (instead of a static array with a fixed size) as we might change the number of entries in the provided file, and dynamic arrays can accommodate that variation.• There must be no memory leaks (e.g., your program requests some memory but does not release them all before it terminates). We will use a tool called valgrind to test the program compiled from your code which reports if there is any memory leaks (refer to Canvas for details). • Start with a fancy banner. There is no specific requirement besides it must include your name, 9- digit SFU ID, and your SFU email address. Let your creativity shine, just nothing offensive. • Print all the entries from the first to the last, along with a Superhero #. Do not reorder the entries*. *The entries are randomly generated using ChatGPT with a specific format and then lightly modified. Hence, the content might not make sense and have very similar wording patterns – no need to worry. Here is a sample output (the program terminates once the last entry is printed, note the Superhero #):And here are some hints for you to write your code: • Use a loop to go through the file and call the functions from Question 2 in each iteration to build the dynamic array. A while-loop is likely the best as the number of entries in the file can change. • To learn to read and write to files see section “C Programming Files” in https://www.programiz.com/c-programming or https://www.tutorialspoint.com/cprogramming/c_file_io.htm or any other online resources. o In particular, look at the fscanf and fgets functions from stdlib.h (you’ll use both). • Don’t forget to close the file after reading it.• To create a dynamic array with unknown size, a typical strategy is to start with a small array and double its size when it is at capacity. For example: int capacity = 16; //initial small size int used = 0; //no items in use yet int *intArray = (int *)malloc(sizeof(int)*capacity); //create an array of size 16 //… after filling up the array, keep increasing used each time you assign an item if (used == capacity) { capacity = capacity * 2; //double the capacity intArray = (int *)realloc(sizeof(int)*capacity); //request an array of size 32, keeping the items } Use this to build your dynamic array to store the superhero information (in this example each item is an int, in your code each item should be a pointer to a Superhero struct variable). You can assume the file will always have the name superheros.txt and will be available in the same directory as the program.You can also assume the format for a superhero entry is consistent (first line starts with **Name:**, second line starts with **Height:**, third line starts with **Superpower:**, fourth line are the traits, and fifth line is an empty line), with each line having at most 300 characters. However, there is no guarantee on how many superhero entries are inside the file (the provided file is just an example), though each entry will have the exact same set of information. Write the main function (and your helper functions, if any) in the source file and name it as a2_question3.c. You can include any libraries that are covered in class (i.e., stdio, stdlib, string, math, stdbool).Coding Style [4 marks] Your program should be properly indented, have clear and meaningful variable names (e.g., no singleletter variable names except loop iterators) and enough white space and comments to make it easy to read. Named constants should be used where appropriate. Each line of code should not exceed 80 characters. White space should be used in a consistent manner. Remember to include your information. Keep your code concise and efficient. If your code is unnecessarily long or inefficient (e.g., hard-code for all possible cases, extraneous function calls), we might deduct marks. To help you to get into the habit of good coding style, we will read your code and marks will be deducted if your code is not styled properly. Before You Submit Before submitting your assignment, make sure you have test-run your code in our CSIL machines using the steps covered in class: open your assignment folder in VS Code, use the gcc command with options in the terminal to compile your code, and run the executable. We will use these steps when markingyour submission and will deduct marks if your code does not compile/run – to maintain fairness we do not accept reasons like “but it worked on my computer” or “but it ran just fine in my IDE”. The Makefile provided in this assignment is used by a command in the CSIL machines called “make” to quickly compile your code. It is especially useful if you have multiple source files. To use it, type the following command in the prompt (make sure you are in the directory with all the files of Assignment 2): $ make test1The example above illustrates how Question 1 is compiled into an executable called “test1” when using the Makefile. Replace the “test1” with “test2”, “test3”, …etc. for other questions. You can then run the executable by typing ./test1 to test your code for Question 1. If you make changes to your code, use the make command again. You can also use make all if you want to compile all your code at once. The test files (test1.c, test2.c, …etc.) are provided in this assignment for you to test your code. Each typically contains a main function along with other tester functions and/or calls. You can modify them to further test your code, but do not submit these test files because we will be using our test files that are similar but not identical to grade your assignment. This makes sure that your code is not written to produce hard-coded output. The header files (a2_question1.h, a2_question2.h, …etc.) are there to make the compilation work. You can look at them but do not modify them. You also do not have to submit them. Submission Submit only the 3 source files indicated above (a2_question1.c, a2_question2.c, a2_question3.c) to CourSys.Refer to the corresponding Canvas assignment entry for details. Assignment late penalty: 10% per calendar day (each 0 to 24 hour period past due), max 2 days late.Question 1 [16 marks] In this question you are going to extend the program you made in Assignment 2 by providing more functionalities. You are again going to write your own main function so the program runs. Create a program that reads all the superhero information from a file and provide an interface for the user to query superhero entries. Your program must support the following functionalities as options:• Option 1: to load a data file – ask the user to input the name of the file containing superhero information and load the entries from that file (report # of entries). Note that the file can be nonexistent (if so the program should print the error and keep running), but if it exists you can assume the format inside that file is valid and consistent, with each line having at most 300 characters. It is possible for the user to load a different file by choosing this option again, which will replace the previously loaded entries. You can assume the filename has at most 50 characters and no space.• Option 2: to list entries sorted by height – list the superheroes from shortest to tallest. Within entries with the same height (if exist), they should be ordered by name (as determined by strcmp). • Option 3: to list entries sorted by name – list the superheroes sorted by name (as determined by strcmp). Within entries with the same name (if exist), they should be ordered by height.• Option 4: to lookup a superhero – ask the user for a superpower for at most 50 characters (space included), then report all entries with superpower having the input as a substring. There can be more than one entry. The lookup is case-sensitive (i.e., “fly” and “Fly” are not the same). o If one or more entries are found, print all the information. Any order is acceptable. o If no entry is found, print “No such superhero on record.” You can assume the user will always enter at most 50 characters. • Option 5: to terminate the program – thank the user and end the program.Your program must meet these requirements: • Include the provided header file (.h file) and use the functions defined in the corresponding implementation file (.c file) in your driver file (where main is), do not redefine those functions. • Use a dynamic array of Superhero struct pointers to store the superhero information. This is the recommended approach from A2 (instead of a static array with a fixed size) as we might change the number of entries in the provided file, and dynamic arrays can accommodate that variation.• There must be no memory leaks (e.g., your program requests some memory but does not release them all before it terminates). In CSIL you can use the Valgrind tool as described to check for that. • Start with a fancy banner. There is no specific requirement besides it must include your name, 9- digit SFU ID, and your SFU email address. Let your creativity shine, just nothing offensive.• If a functionality (e.g., list, lookup) is requested by the user but no file has been loaded, the program should print an error message telling the user that no file have been loaded and prompt the user to do so (e.g., “No superheroes file loaded. Load one first.”). • The sorting for Options 2&3 can be done using any sorting algorithms covered in class, including the built-in qsort, which requires to you to write your own compare functions. See a3_superherolib.h for details.And here are some hints for you to write your code: • The program interface is essentially a do-while loop where in each iteration it asks for an option and determines what to do, with a repeating conditional of the terminating option has not been inputted. It is a common approach for menu-based interfaces.• Since each item in the dynamic array for the superhero entries are pointers, be careful with how you are casting the pointer variables in the compare functions for qsort (they are pointers to the items) – suppose the dynamic array has the type A**, then the variables should be cast to A**. • The scanf() function works slightly differently between different formats. One difference is how it handles leading and trailing newlines (look up “dangling newline character” if you are curious). One option to get around this is put a space in front of the format string (e.g., “ %d”, “ %s”). You are encouraged to explore different ways to get around this issue.*The entries are randomly generated using ChatGPT with a specific format and then lightly modified. Hence, the content might not make sense and have very similar wording patterns – no need to worry. Include the function definitions corresponding to a3_superherolib.h (and your helper functions, if any) in a source file named as a3_superherolib.c. Remember the .h file only contains the function headers. Include your main function definition (and of your helper functions, if any) in another source file named as a3_superheroLookupSystem.c. We call this the driver file because this is where the program starts.To determine where to place a helper function, think about its purpose. If it provides functionality with the Superhero structs (e.g., create/clear Superheroes, compare Superheroes), it is a library function; if it provides functionality of the program (e.g., print a fancy banner), it is a program function. Incorrectly placing these functions will lead to mark deductions in the Coding Style category.Here are some sample input and output (you can assume user input is always valid, but could be incorrect): Figure 1. Inputting an incorrect option (9), opening a non-existent file (incorrect input) then an existing file (with 100 entries).Figure 2. Listing superheroes sorted by height (note those with the same height are sorted by name). Figure 3. Looking up for “strength” which has no result (but with “Strength” there will be one – Iron Guardian).Figure 4. Looking up for “Sound” (4 results, order doesn’t matter) – yes there are actually name duplicates in the file! Coding Style [4 marks] Your program should be properly indented, have clear and meaningful variable names (e.g., no singleletter variable names except loop iterators) and enough white space and comments to make it easy to read. Named constants should be used where appropriate. Each line of code should not exceed 80 characters. White space should be used in a consistent manner. Remember to include your information. Keep your code concise and efficient. If your code is unnecessarily long or inefficient (e.g., hard-code for all possible cases, extraneous function calls), we might deduct marks. To help you to get into the habit of good coding style, we will read your code and marks will be deducted if your code is not styled properly.Before You Submit Before submitting your assignment, make sure you have test-run your code in our CSIL machines using the steps covered in class: open your assignment folder in VS Code, use the gcc command with options in the terminal to compile your code, and run the executable. We will use these steps when marking your submission and will deduct marks if your code does not compile/run – to maintain fairness we do not accept reasons like “but it worked on my computer” or “but it ran just fine in my IDE”.The Makefile provided in this assignment is used by a command in the CSIL machines called “make” to quickly compile your code. It is especially useful if you have multiple source files. To use it, type the following command in the prompt (make sure you are in the directory with all the files of Assignment 3): $ make test1The example above illustrates how Question 1 is compiled into an executable called “test1” when using the Makefile. Replace the “test1” with “test2”, “test3”, …etc. for other questions. You can then run the executable by typing ./test1 to test your code for Question 1. If you make changes to your code, use the make command again. You can also use make all if you want to compile all your code at once.The header file a3_superherolib.h is there to make the compilation work. Take a look at it for information about how each function should work. You might have to modify it and you will have to submit it this time.Submission Submit only the 3 source files (a3_superherolib.h, a3_superherolib.c, a3_superheroLookupSystem.c) to CourSys. Refer to the corresponding Canvas assignment entry for details. Assignment late penalty: 10% per calendar day (each 0 to 24 hour period past due), max 2 days late.Question 1 [16 marks] In this question you are going to extend the program you made in Assignment 2&3 by providing more functionalities. You are also again going to write your own main function so the program runs. In addition, you are going to implement everything in C++. Create a program that reads all the superhero information from a file and provide an interface for the user to query superhero entries*.Your program must support the following functionalities as options: • Option 1: to load a superheroes file – ask the user to input the name of the file containing superhero information and load the entries from that file (report # of entries). Note that the file can be non-existent (if so the program should print the error and keep running by printing all the options again), but if it exists you can assume the format inside that file is valid and consistent, with each line having at most 300 characters. It is possible for the user to load a different file by choosing this option again, which will replace the previously loaded entries. You can assume the filename has at most 50 characters and no space.• Option 2: to list entries sorted by height – list the superheroes from shortest to tallest. Within entries with the same height (if exist), they should be ordered by name (as determined by strcmp). • Option 3: to list entries sorted by name – list the superheroes sorted by name (as determined by strcmp). Within entries with the same name (if exist), they should be ordered by height.• Option 4: to lookup a superhero – ask the user for a superpower for at most 50 characters (space included), then report all entries with superpower having the input as a substring. There can be more than one entry. The lookup is case-sensitive (i.e., “fly” and “Fly” are not the same). o If one or more entries are found, print all the information. Any order is acceptable. o If no entry is found, print “No such superhero on record.”You can assume the user will always enter at most 50 characters. • (NEW) Option 5: to add an entry – ask the user for all the information for a superhero (name, height, superpower, traits) and add it to the currently loaded entries. After adding, report and print “Entry added.”. You can assume the user will always input something in the correct format. o There is no need to check for duplicates. o Check for invalid height values: feet that are userInput).• The built-in stable_sort from C++ (defined in ) can sort items in different kinds of container classes as long as they provide access to the items using an “iterator”, which is a common design pattern in Object-Oriented Programming. Refer to our lecture examples and the reference link above for details. Also pay attention to how the compare function pointer works. • The way things are being read from the console and file is simplified with cin and fin. Besides the >> operator, there is a getline function for strings. Use it like this: getline(cin, stringVariable), or this after an int is read: getline(cin >> ws, stringVariable). You are encouraged to explore why.*The entries are randomly generated using ChatGPT with a specific format and then lightly modified. Hence, the content might not make sense and have very similar wording patterns – no need to worry. Include the function definitions corresponding to a4_superherolib.hpp (and your helper functions, if any) in a source file named as a4_superherolib.cpp. Do the same for a4_superheroList. Both the .hpp files are provided to help you get started. They are adequate to complete this assignment, but if you want you can add/remove functions as appropriate – as long as they fulfill all the requirements stated above. Include your main function (and your helper functions, if any) in another source file named as a4_superheroLookupSystem.cpp.We call this the driver file because this is where the program starts. To determine where to place a helper function, think about its purpose. If it provides functionality with the Superhero structs (e.g., create/clear Superheroes, compare Superheroes), it is a library function; if it provides functionality of the program (e.g., print a fancy banner), it is a program function; and if it provides functionality that directly accesses private members of the SuperheroList, it should be a member function of SuperheroList. Incorrectly placing these functions will lead to mark deductions in the Coding Style. Here are some sample input and output (you can assume user input is always valid, except Option 5): Figure 1. Choosing an option without opening any file then opening an existing file.Figure 2. Listing superheroes sorted by height, note how they are also sorted by name (only showing the first 2 here for space). Figure 3. Looking up for “strength” which has no result (but with “Strength” there will be one – Iron Guardian). Figure 4. Adding a superhero to the list (it’ll be Superhero #16 when choosing Option 3 next). Figure 5. Saving the entries to a file (100 entries were loaded at start, adding 1 makes it 101) and loading it back.Coding Style [4 marks] Your program should be properly indented, have clear and meaningful variable names (e.g., no singleletter variable names except loop iterators) and enough white space and comments to make it easy to read. Named constants should be used where appropriate. Each line of code should not exceed 80 characters. White space should be used in a consistent manner. Remember to include your information. Keep your code concise and efficient. If your code is unnecessarily long or inefficient (e.g., hard-code for all possible cases, extraneous function calls), we might deduct marks. To help you to get into the habit of good coding style, we will read your code and marks will be deducted if your code is not styled properly.Before You Submit Before submitting your assignment, make sure you have test-run your code in our CSIL machines using the steps covered in class: open your assignment folder in VS Code, use the gcc command with options in the terminal to compile your code, and run the executable. We will use these steps when marking your submission and will deduct marks if your code does not compile/run – to maintain fairness we do not accept reasons like “but it worked on my computer” or “but it ran just fine in my IDE”. The Makefile provided in this assignment is used by a command in the CSIL machines called “make” to quickly compile your code. It is especially useful if you have multiple source files. To use it, type the following command in the prompt (make sure you are in the directory with all the files of Assignment 3): $ make test1 The example above illustrates how Question 1 is compiled into an executable called “test1” when using the Makefile.Replace the “test1” with “test2”, “test3”, …etc. for other questions. You can then run the executable by typing ./test1 to test your code for Question 1. If you make changes to your code, use the make command again. You can also use make all if you want to compile all your code at once. The header files are there to make the compilation work. Take a look at them for information about how each function should work. You might have to modify them and you will have to submit them this time. Submission Submit only the 5 source files (a4_superherolib.hpp, a4_superherolib.cpp, a4_superheroList.hpp a4_superheroList.cpp, a4_superheroLookupSystem.cpp) to CourSys. Refer to the corresponding Canvas assignment entry for details. Assignment late penalty: 10% per calendar day (each 0 to 24 hour period past due), max 2 days late.
Question 1 [16 marks] In this question you are going to extend the program you made in Assignment 2&3 by providing more functionalities. You are also again going to write your own main function so the program runs. In addition, you are going to implement everything in C++. Create a program that reads all the superhero information from a file and provide an interface for the user to query superhero entries*.Your program must support the following functionalities as options: • Option 1: to load a superheroes file – ask the user to input the name of the file containing superhero information and load the entries from that file (report # of entries). Note that the file can be non-existent (if so the program should print the error and keep running by printing all the options again), but if it exists you can assume the format inside that file is valid and consistent, with each line having at most 300 characters. It is possible for the user to load a different file by choosing this option again, which will replace the previously loaded entries. You can assume the filename has at most 50 characters and no space.• Option 2: to list entries sorted by height – list the superheroes from shortest to tallest. Within entries with the same height (if exist), they should be ordered by name (as determined by strcmp). • Option 3: to list entries sorted by name – list the superheroes sorted by name (as determined by strcmp). Within entries with the same name (if exist), they should be ordered by height.• Option 4: to lookup a superhero – ask the user for a superpower for at most 50 characters (space included), then report all entries with superpower having the input as a substring. There can be more than one entry. The lookup is case-sensitive (i.e., “fly” and “Fly” are not the same). o If one or more entries are found, print all the information. Any order is acceptable. o If no entry is found, print “No such superhero on record.”You can assume the user will always enter at most 50 characters. • (NEW) Option 5: to add an entry – ask the user for all the information for a superhero (name, height, superpower, traits) and add it to the currently loaded entries. After adding, report and print “Entry added.”. You can assume the user will always input something in the correct format. o There is no need to check for duplicates. o Check for invalid height values: feet that are userInput).• The built-in stable_sort from C++ (defined in ) can sort items in different kinds of container classes as long as they provide access to the items using an “iterator”, which is a common design pattern in Object-Oriented Programming. Refer to our lecture examples and the reference link above for details. Also pay attention to how the compare function pointer works. • The way things are being read from the console and file is simplified with cin and fin. Besides the >> operator, there is a getline function for strings. Use it like this: getline(cin, stringVariable), or this after an int is read: getline(cin >> ws, stringVariable). You are encouraged to explore why.*The entries are randomly generated using ChatGPT with a specific format and then lightly modified. Hence, the content might not make sense and have very similar wording patterns – no need to worry. Include the function definitions corresponding to a4_superherolib.hpp (and your helper functions, if any) in a source file named as a4_superherolib.cpp. Do the same for a4_superheroList. Both the .hpp files are provided to help you get started. They are adequate to complete this assignment, but if you want you can add/remove functions as appropriate – as long as they fulfill all the requirements stated above. Include your main function (and your helper functions, if any) in another source file named as a4_superheroLookupSystem.cpp.We call this the driver file because this is where the program starts. To determine where to place a helper function, think about its purpose. If it provides functionality with the Superhero structs (e.g., create/clear Superheroes, compare Superheroes), it is a library function; if it provides functionality of the program (e.g., print a fancy banner), it is a program function; and if it provides functionality that directly accesses private members of the SuperheroList, it should be a member function of SuperheroList. Incorrectly placing these functions will lead to mark deductions in the Coding Style. Here are some sample input and output (you can assume user input is always valid, except Option 5): Figure 1. Choosing an option without opening any file then opening an existing file.Figure 2. Listing superheroes sorted by height, note how they are also sorted by name (only showing the first 2 here for space). Figure 3. Looking up for “strength” which has no result (but with “Strength” there will be one – Iron Guardian). Figure 4. Adding a superhero to the list (it’ll be Superhero #16 when choosing Option 3 next). Figure 5. Saving the entries to a file (100 entries were loaded at start, adding 1 makes it 101) and loading it back.Coding Style [4 marks] Your program should be properly indented, have clear and meaningful variable names (e.g., no singleletter variable names except loop iterators) and enough white space and comments to make it easy to read. Named constants should be used where appropriate. Each line of code should not exceed 80 characters. White space should be used in a consistent manner. Remember to include your information. Keep your code concise and efficient. If your code is unnecessarily long or inefficient (e.g., hard-code for all possible cases, extraneous function calls), we might deduct marks. To help you to get into the habit of good coding style, we will read your code and marks will be deducted if your code is not styled properly.Before You Submit Before submitting your assignment, make sure you have test-run your code in our CSIL machines using the steps covered in class: open your assignment folder in VS Code, use the gcc command with options in the terminal to compile your code, and run the executable. We will use these steps when marking your submission and will deduct marks if your code does not compile/run – to maintain fairness we do not accept reasons like “but it worked on my computer” or “but it ran just fine in my IDE”. The Makefile provided in this assignment is used by a command in the CSIL machines called “make” to quickly compile your code. It is especially useful if you have multiple source files. To use it, type the following command in the prompt (make sure you are in the directory with all the files of Assignment 3): $ make test1 The example above illustrates how Question 1 is compiled into an executable called “test1” when using the Makefile.Replace the “test1” with “test2”, “test3”, …etc. for other questions. You can then run the executable by typing ./test1 to test your code for Question 1. If you make changes to your code, use the make command again. You can also use make all if you want to compile all your code at once. The header files are there to make the compilation work. Take a look at them for information about how each function should work. You might have to modify them and you will have to submit them this time. Submission Submit only the 5 source files (a4_superherolib.hpp, a4_superherolib.cpp, a4_superheroList.hpp a4_superheroList.cpp, a4_superheroLookupSystem.cpp) to CourSys. Refer to the corresponding Canvas assignment entry for details. Assignment late penalty: 10% per calendar day (each 0 to 24 hour period past due), max 2 days late.
Question 1 [16 marks] In this question you are going to extend the program you made in Assignment 2 by providing more functionalities. You are again going to write your own main function so the program runs. Create a program that reads all the superhero information from a file and provide an interface for the user to query superhero entries. Your program must support the following functionalities as options:• Option 1: to load a data file – ask the user to input the name of the file containing superhero information and load the entries from that file (report # of entries). Note that the file can be nonexistent (if so the program should print the error and keep running), but if it exists you can assume the format inside that file is valid and consistent, with each line having at most 300 characters. It is possible for the user to load a different file by choosing this option again, which will replace the previously loaded entries. You can assume the filename has at most 50 characters and no space.• Option 2: to list entries sorted by height – list the superheroes from shortest to tallest. Within entries with the same height (if exist), they should be ordered by name (as determined by strcmp). • Option 3: to list entries sorted by name – list the superheroes sorted by name (as determined by strcmp). Within entries with the same name (if exist), they should be ordered by height.• Option 4: to lookup a superhero – ask the user for a superpower for at most 50 characters (space included), then report all entries with superpower having the input as a substring. There can be more than one entry. The lookup is case-sensitive (i.e., “fly” and “Fly” are not the same). o If one or more entries are found, print all the information. Any order is acceptable. o If no entry is found, print “No such superhero on record.” You can assume the user will always enter at most 50 characters. • Option 5: to terminate the program – thank the user and end the program.Your program must meet these requirements: • Include the provided header file (.h file) and use the functions defined in the corresponding implementation file (.c file) in your driver file (where main is), do not redefine those functions. • Use a dynamic array of Superhero struct pointers to store the superhero information. This is the recommended approach from A2 (instead of a static array with a fixed size) as we might change the number of entries in the provided file, and dynamic arrays can accommodate that variation.• There must be no memory leaks (e.g., your program requests some memory but does not release them all before it terminates). In CSIL you can use the Valgrind tool as described to check for that. • Start with a fancy banner. There is no specific requirement besides it must include your name, 9- digit SFU ID, and your SFU email address. Let your creativity shine, just nothing offensive.• If a functionality (e.g., list, lookup) is requested by the user but no file has been loaded, the program should print an error message telling the user that no file have been loaded and prompt the user to do so (e.g., “No superheroes file loaded. Load one first.”). • The sorting for Options 2&3 can be done using any sorting algorithms covered in class, including the built-in qsort, which requires to you to write your own compare functions. See a3_superherolib.h for details.And here are some hints for you to write your code: • The program interface is essentially a do-while loop where in each iteration it asks for an option and determines what to do, with a repeating conditional of the terminating option has not been inputted. It is a common approach for menu-based interfaces.• Since each item in the dynamic array for the superhero entries are pointers, be careful with how you are casting the pointer variables in the compare functions for qsort (they are pointers to the items) – suppose the dynamic array has the type A**, then the variables should be cast to A**. • The scanf() function works slightly differently between different formats. One difference is how it handles leading and trailing newlines (look up “dangling newline character” if you are curious). One option to get around this is put a space in front of the format string (e.g., “ %d”, “ %s”). You are encouraged to explore different ways to get around this issue.*The entries are randomly generated using ChatGPT with a specific format and then lightly modified. Hence, the content might not make sense and have very similar wording patterns – no need to worry. Include the function definitions corresponding to a3_superherolib.h (and your helper functions, if any) in a source file named as a3_superherolib.c. Remember the .h file only contains the function headers. Include your main function definition (and of your helper functions, if any) in another source file named as a3_superheroLookupSystem.c. We call this the driver file because this is where the program starts.To determine where to place a helper function, think about its purpose. If it provides functionality with the Superhero structs (e.g., create/clear Superheroes, compare Superheroes), it is a library function; if it provides functionality of the program (e.g., print a fancy banner), it is a program function. Incorrectly placing these functions will lead to mark deductions in the Coding Style category.Here are some sample input and output (you can assume user input is always valid, but could be incorrect): Figure 1. Inputting an incorrect option (9), opening a non-existent file (incorrect input) then an existing file (with 100 entries).Figure 2. Listing superheroes sorted by height (note those with the same height are sorted by name). Figure 3. Looking up for “strength” which has no result (but with “Strength” there will be one – Iron Guardian).Figure 4. Looking up for “Sound” (4 results, order doesn’t matter) – yes there are actually name duplicates in the file! Coding Style [4 marks] Your program should be properly indented, have clear and meaningful variable names (e.g., no singleletter variable names except loop iterators) and enough white space and comments to make it easy to read. Named constants should be used where appropriate. Each line of code should not exceed 80 characters. White space should be used in a consistent manner. Remember to include your information. Keep your code concise and efficient. If your code is unnecessarily long or inefficient (e.g., hard-code for all possible cases, extraneous function calls), we might deduct marks. To help you to get into the habit of good coding style, we will read your code and marks will be deducted if your code is not styled properly.Before You Submit Before submitting your assignment, make sure you have test-run your code in our CSIL machines using the steps covered in class: open your assignment folder in VS Code, use the gcc command with options in the terminal to compile your code, and run the executable. We will use these steps when marking your submission and will deduct marks if your code does not compile/run – to maintain fairness we do not accept reasons like “but it worked on my computer” or “but it ran just fine in my IDE”.The Makefile provided in this assignment is used by a command in the CSIL machines called “make” to quickly compile your code. It is especially useful if you have multiple source files. To use it, type the following command in the prompt (make sure you are in the directory with all the files of Assignment 3): $ make test1The example above illustrates how Question 1 is compiled into an executable called “test1” when using the Makefile. Replace the “test1” with “test2”, “test3”, …etc. for other questions. You can then run the executable by typing ./test1 to test your code for Question 1. If you make changes to your code, use the make command again. You can also use make all if you want to compile all your code at once.The header file a3_superherolib.h is there to make the compilation work. Take a look at it for information about how each function should work. You might have to modify it and you will have to submit it this time.Submission Submit only the 3 source files (a3_superherolib.h, a3_superherolib.c, a3_superheroLookupSystem.c) to CourSys. Refer to the corresponding Canvas assignment entry for details. Assignment late penalty: 10% per calendar day (each 0 to 24 hour period past due), max 2 days late.
Question 1 [6 marks] Write a recursive solution that does exactly the same as what the function in Question 1 of Assignment 1 does. That is, replaces a specific digit in an int number with another digit and returns the result as an int. Return the same number if the target/replacement character is not a digit. Use this function header: int replaceDigitsRecursive(int number, char target, char replacement)For example (for more examples refer to the description in Assignment 1): replaceDigitsRecursive(1, ‘1’, ‘2’) should return 2 replaceDigitsRecursive(-232, ‘3’, ‘0’) should return -202 replaceDigitsRecursive(998, ‘9’, ‘0’) should return 8 (leading zeros will not be part of the resulting number)You can assume the number does not have leading zeros (e.g., we will not call replaceDigitsRecursive(01, ‘1’, ‘2’)), except when the number is actually zero (i.e., we might call replaceDigitsRecursive(0, ‘1’, ‘1’)). This question aims to let you practice recursion. One way to do so is to look at your answer for Assignment 1, separate the recursive case (most likely where you use a loop) from the base case (most likely the last step you do before returning the result). You should be able to see a significant simplification to your code. Do not use any global or static variables, or pass-by-reference (if you do so you get 0 for this question). Depends on how you solve this, you might have to create a recursive helper function that passes an extra parameter preserving the partial answer. IOnly include the a2_question1.h header file and the function definition (and your helper functions, if any) in the source file and name it as a2_question1.c.Question 2 [4 marks] In the header file for this question, a struct called Superhero is defined like this: typedef struct { char* name; short feetInHeight; short inchesInHeight; char* superpower; char* traits; } Superhero; First, write a function that takes in 5 parameters: a pointer to a char array representing a name, two shorts representing feet in height and inches in height respectively, a pointer to a char array representing a superpower, and a pointer to a char array representing the traits; and returns the address of a dynamically (i.e., uses malloc) created Superhero struct variable storing those parameters.Use this function header: Superhero* createSuperhero(const char* name, short feetInHeight, short inchesInHeight, const char* superpower, const char* traits) For example, given the code (name, superpower and traits are Cstrings storing the proper information): Superhero* superhero = createSuperhero(name, 6, 0, superpower, traits); printf(“%s %d’%d” %s: %s ”, superhero->name, superhero->feetInHeight, superhero->inchesInHeight, superhero->superpower, superhero->traits); will result in an output like this: Thunderstrike 6’0″ Weather Control: Harnessing storms, Thunderstrike commands thunder and lightning, using nature’s fury to protect the innocent and fight against evil. You can assume all the height measurements are valid (i.e., feet & inches will not be negative), and all the Cstrings are properly formed ( terminated).Field variables name, superpower, and traits in the struct must be created dynamically and are copies of the parameters, instead of simply pointing to the parameters’ addresses. This is called “deep-copy”. Next, write another function that takes in 1 parameter: the address of a Superhero struct variable; and releases (i.e., uses free) the memory created for the 3 field variables name, superpower, and traits. Use this function header: void clearSuperhero(Superhero* superhero) Note that the parameter can be NULL (if so the function should do nothing). Also, this function does not release the memory used for the struct variable, but only those used by the variable’s fields. To release all the memory dynamically allocated for the struct variable, you should call the free() function with the address of this struct variable right after the function returns. For details read Question 3. Only include the a2_question2.h header file and the function definitions(and your helper functions, if any) in the source file and name it as a2_question2.c. Do not use recursion in your answer.Question 3 [6 marks] In this question you are going to make use of your answer for Question 2 to create a working program (so finish it first). You are also going to write your own main function so the program runs. Create a program that reads all the superhero information from the provided superheros.txt file and prints the information to the screen. Your program must meet these requirements: • Include the header file from Question 2 and use the functions defined there (createSuperhero and clearSuperhero) to complete this question. Do not redefine those functions. • Use a dynamic array of Superhero struct pointers to store the superhero information. This is the recommended approach (instead of a static array with a fixed size) as we might change the number of entries in the provided file, and dynamic arrays can accommodate that variation.• There must be no memory leaks (e.g., your program requests some memory but does not release them all before it terminates). We will use a tool called valgrind to test the program compiled from your code which reports if there is any memory leaks (refer to Canvas for details). • Start with a fancy banner. There is no specific requirement besides it must include your name, 9- digit SFU ID, and your SFU email address. Let your creativity shine, just nothing offensive. • Print all the entries from the first to the last, along with a Superhero #. Do not reorder the entries*. *The entries are randomly generated using ChatGPT with a specific format and then lightly modified. Hence, the content might not make sense and have very similar wording patterns – no need to worry. Here is a sample output (the program terminates once the last entry is printed, note the Superhero #):And here are some hints for you to write your code: • Use a loop to go through the file and call the functions from Question 2 in each iteration to build the dynamic array. A while-loop is likely the best as the number of entries in the file can change. • To learn to read and write to files see section “C Programming Files” in https://www.programiz.com/c-programming or https://www.tutorialspoint.com/cprogramming/c_file_io.htm or any other online resources. o In particular, look at the fscanf and fgets functions from stdlib.h (you’ll use both). • Don’t forget to close the file after reading it.• To create a dynamic array with unknown size, a typical strategy is to start with a small array and double its size when it is at capacity. For example: int capacity = 16; //initial small size int used = 0; //no items in use yet int *intArray = (int *)malloc(sizeof(int)*capacity); //create an array of size 16 //… after filling up the array, keep increasing used each time you assign an item if (used == capacity) { capacity = capacity * 2; //double the capacity intArray = (int *)realloc(sizeof(int)*capacity); //request an array of size 32, keeping the items } Use this to build your dynamic array to store the superhero information (in this example each item is an int, in your code each item should be a pointer to a Superhero struct variable). You can assume the file will always have the name superheros.txt and will be available in the same directory as the program.You can also assume the format for a superhero entry is consistent (first line starts with **Name:**, second line starts with **Height:**, third line starts with **Superpower:**, fourth line are the traits, and fifth line is an empty line), with each line having at most 300 characters. However, there is no guarantee on how many superhero entries are inside the file (the provided file is just an example), though each entry will have the exact same set of information. Write the main function (and your helper functions, if any) in the source file and name it as a2_question3.c. You can include any libraries that are covered in class (i.e., stdio, stdlib, string, math, stdbool).Coding Style [4 marks] Your program should be properly indented, have clear and meaningful variable names (e.g., no singleletter variable names except loop iterators) and enough white space and comments to make it easy to read. Named constants should be used where appropriate. Each line of code should not exceed 80 characters. White space should be used in a consistent manner. Remember to include your information. Keep your code concise and efficient. If your code is unnecessarily long or inefficient (e.g., hard-code for all possible cases, extraneous function calls), we might deduct marks. To help you to get into the habit of good coding style, we will read your code and marks will be deducted if your code is not styled properly. Before You Submit Before submitting your assignment, make sure you have test-run your code in our CSIL machines using the steps covered in class: open your assignment folder in VS Code, use the gcc command with options in the terminal to compile your code, and run the executable. We will use these steps when markingyour submission and will deduct marks if your code does not compile/run – to maintain fairness we do not accept reasons like “but it worked on my computer” or “but it ran just fine in my IDE”. The Makefile provided in this assignment is used by a command in the CSIL machines called “make” to quickly compile your code. It is especially useful if you have multiple source files. To use it, type the following command in the prompt (make sure you are in the directory with all the files of Assignment 2): $ make test1The example above illustrates how Question 1 is compiled into an executable called “test1” when using the Makefile. Replace the “test1” with “test2”, “test3”, …etc. for other questions. You can then run the executable by typing ./test1 to test your code for Question 1. If you make changes to your code, use the make command again. You can also use make all if you want to compile all your code at once. The test files (test1.c, test2.c, …etc.) are provided in this assignment for you to test your code. Each typically contains a main function along with other tester functions and/or calls. You can modify them to further test your code, but do not submit these test files because we will be using our test files that are similar but not identical to grade your assignment. This makes sure that your code is not written to produce hard-coded output. The header files (a2_question1.h, a2_question2.h, …etc.) are there to make the compilation work. You can look at them but do not modify them. You also do not have to submit them. Submission Submit only the 3 source files indicated above (a2_question1.c, a2_question2.c, a2_question3.c) to CourSys.Refer to the corresponding Canvas assignment entry for details. Assignment late penalty: 10% per calendar day (each 0 to 24 hour period past due), max 2 days late.
Question 1 [5 marks] Write a function that replaces a specific digit in an int number with another digit and returns the result as an int. Return the same number if the target/replacement character is not a digit. Use this function header: int replaceDigits(int number, char target, char replacement) For example: replaceDigits(1, ‘1’, ‘2’) should return 2 replaceDigits(-232, ‘3’, ‘0’) should return -202 replaceDigits(123, ‘4’, ‘5’) should return 123 replaceDigits(1223, ‘!’, ‘?’) should return 1223 replaceDigits(5670, ‘5’, ‘0’) should return 670 (leading zero will not be part of the resulting number) replaceDigits(30400, ‘0’, ‘9’) should return 39499You can assume the number does not have leading zeros (e.g., we will not call replaceDigits(01, ‘1’, ‘2’)), except when the number is actually zero (i.e., we might call replaceDigits(0, ‘1’, ‘1’)). Only include the a1_question.h header file and the function definition (and your helper functions, if any) in the source file and name it as a1_question1.c. Do not use recursion in your answer.Question 2 [5 marks] Write a function that takes in 4 parameters: an int array, its size, the left index, and the right index; and checks if the elements between the left and right index (inclusive) in the int array are sorted in ascending order.If the left index is larger than the right index, swap them first. Then, if the left index is invalid (e.g., negative, larger than the size), use the leftmost valid index of the array; if the right index is invalid, use the rightmost valid index of the array. Use this function header:bool rangedCheckforSorted(int array[], unsigned int size, int leftIndex, int rightIndex) For example (suppose there is a myIntArray created as [-4, 3, -12, 0, 5, 72, 88, 128, 1, 64]): rangedCheckforSorted(myIntArray, 10, -2, 3) will check indexes between 0 and 3, and return false rangedCheckforSorted(myIntArray, 10, 7, 2) will check indexes between 2 and 7, and return true rangedCheckforSorted(myIntArray, 10, 5, 5) will check indexes between 5 and 5, and return true rangedCheckforSorted(myIntArray, 10, 0, 9) will check indexes between 0 and 9, and return false You can assume the array has one or more elements and the size is always correct. Content of the int array does not change after calling this function.Only include the a1_question2.h header file, , and the function definition (and your helper functions, if any) in the source file and name it as a1_question2.c. Do not use recursion in your answer.Question 3 [6 marks] In C arrays can be accessed using pointers. As an exercise, write a function that reverses the polarity of every element in a 2D int array (i.e., -1 becomes 1, 46 becomes -46, …etc.) using only pointers. That is, your code must not use any [ ], which is the non-pointer version of array access. Use this function header: void reversePolarity(unsigned int row, unsigned int col, int** array)For example, suppose we have a 2-by-3 array my2DIntArray with the content [[0, 1, -2], [-999, 2345, 678]]. Then the function will be called as (the array needs to be created differently, see the test file): reversePolarity(2, 3, my2DIntArray) And when the function returns, the content of my2DIntArray becomes [[0, -1, 2], [999, -2345, -678]]. You can assume that the row & col parameters always correctly indicate the number of rows & columns of the array. Do not use [ ] anywhere in your answer (if you do you get 0 for this question). Only include the a1_question3.h header file and the function definition (and your helper functions, if any) in the source file and name it as a1_question3.c.Coding Style [4 marks] Your program should be properly indented, have clear and meaningful variable names (e.g., no singleletter variable names except loop iterators) and enough white space and comments to make it easy to read. Named constants should be used where appropriate. Each line of code should not exceed 80 characters. White space should be used in a consistent manner. Remember to include your information. To help you to get into the habit of good coding style, we will read your code and marks will be deducted if your code is not styled properly.Before You Submit Before submitting your assignment, make sure you have test-run your code in our CSIL machines using the steps covered in class: open your assignment folder in VS Code, use the gcc command with options in the terminal to compile your code, and run the executable. We will use these steps when marking your submission and will deduct marks if your code does not compile/run – to maintain fairness we do not accept reasons like “but it worked on my computer” or “but it ran just fine in my IDE”.The Makefile provided in this assignment is used by a command in the CSIL machines called “make” to quickly compile your code. It is especially useful if you have multiple source files. To use it, type the following command in the prompt (make sure you are in the directory with all the files of Assignment 1): $ make test1The example above illustrates how Question 1 is compiled into an executable called “test1” when using the Makefile. Replace the “test1” with “test2”, “test3”, …etc. for other questions. You can then run the executable by typing ./test1 to test your code for Question 1. If you make changes to your code, use the make command again. You can also use make all if you want to compile all your code at once. The test files (test1.c, test2.c, …etc.) are provided in this assignment for you to test your code. Each typically contains a main function along with other tester functions and/or calls. You can modify them to further test your code, but do not submit these test files because we will be using our test files that are similar but not identical to grade your assignment. This makes sure that your code is not written to produce hard-coded output.The header files (question1.h, question2.h, …etc.) are there to make the compilation work. You can look at them but do not modify them. You also do not have to submit them.Submission Submit only the 3 source files indicated above (a1_question1.c, a1_question2.c, a1_question3.c) to CourSys. Refer to the corresponding Canvas assignment entry for details. Assignment late penalty: 10% per calendar day (each 0 to 24 hour period past due), max 2 days late.
1 Introduction Rock, Paper, Scissors is a simple hand game that is played around the world, with many different names and variations. It is commonly used as a way of coming to decisions, and in some cases is even played for sport. The rules require that competing players use one hand to form one of three shapes at an agreedupon time. The person that plays the strongest “object” is the winner of the game. It’s that easy! Rock Paper Scissors is a simple game that anybody can play and win. You beat another player if your choice “beats” theirs in the following way: • Rock beats Scissors • Scissors beat Paper • Paper beats Rock Each round you score 1 for every other player you beat. The first player to score 10 or more points wins the game. In case of a tie, the tying players play against each other until there is one winner.Implement a web application that simulates a rock, paper, scissors game. For each round, the computer will guess and display a rock, paper, or scissors symbol. 1. Users are able to sign up for an account to play a game. 2. Users are able to sign in before start playing a game. 3. One of the 3 (Rock, Paper, Scissor) buttons is clicked. 4. The computer then randomly generates a rock, paper, or scissors. 5. The user’s choice and computer’s choice are checked. 6. The user’s score or the computer’s score is increased by one or no one’s score (a tie) is raised. 7. Update all the page’s results so the user is aware of what happened. 8. You should have a graphical version of what the user and computer each chose, as well as a visual cue as to who won A simple snapshot of this game is shown as below:3 Technologies • CSS3 Sprites • HTML5/6 • Bootstrap • a front-end framework (like REACT, Angulars, vue.js, pug etc) • JavaScript • jQuery and Ajax • Sass4 Submission Submit the source code of the game through Canvas. You need to give a quick presentation on your final product.
CIT has just begun construction of a multifunctional sports arena on its main campus. This sports arena will supply various state-of-the-art and high-performance facilities, like basketball courts, a health and fitness facility, and fitness studios etc. This project will focus on designing a web application to facilitate the management of the arena and to also help support the promotion of health and fitness to the public. The arena will be open to all CIT students, staff, as well as the general public. For the purposes of this project, we will make the simplifying assumption that the arena will be supervised by a single manager and that, of the several clubs operating daily in the arena, each club is operated and run by one trainer only.This web application should meet the following requirements: • All users are required to first register for membership, supplying the following information: name, gender, age, weight, height, contact details (home address, email, phone number), status (student, staff, neither), type of membership (one-year or open-ended), user name, password. • All members must access the system with a valid username and password. No more than three unsuccessful login attempts should be permitted. • All members can review and update their profile, and can reset their password. All members need to change their password every three months. The system should remind the users to update their password before its expiry date and notify the users that they have successfully changed their password after updating it. • The system should display all clubs scheduled daily. All members can search for a club by its name, operating date, or its trainer. Members can sort all clubs by the estimated number of calories that are burned off by that type of physical activity. • When a member reviews the information of a club, the system should display its associated image, its maximum size, the operating hours, the allotted facility (or studio), the personal trainer, the duration of training, and the estimated average calories burned off in that type of physical activity. • All trainers can either book or cancel facilities (including studios and places) for their training classes. They can search for the availability of facilities by their id, name, or location.• All members can either book or cancel their training in the club. They also can search through their bookings by either booking date or the name of the club. • All members can self-manage their fitness training plan with a personal fitness log. The system will keep track of all members’ BMR (Basal Metabolic Rate) values before and after their daily training. According to these values, members can arrange the type of fitness training or club activities to be taken on daily. The fitness log should also include a meal plan to calculate the total calories needing to be burned off. One of the most accurate methods of estimating your BMR is the Harris-Benedict formula: Adult male: 66 + (6.3 × body weight in lbs.) + (12.9 × height in inches) – (6.8 × age in years) = BMR Adult female: 655 + (4.3 × weight in lbs.) + (4.7 × height in inches) – (4.7 × age in years) = BMR To determine your total daily calorie needs, multiply your BMR by the appropriate activity factor, as follows: – If you are sedentary (little or no exercise): Calorie-Calculation = BMR × 1.2 – If you are lightly active (light exercise/sports 1-3 days/week): Calorie-Calculation = BMR × 1.375 – If you are moderately active (moderate exercise/sports 3-5 days/week): Calorie-Calculation = BMR × 1.55 – If you are very active (hard exercise/sports 6-7 days a week): Calorie-Calculation = BMR × 1.725 – If you are extra active (very hard exercise/sports & physical job or 2x training): CalorieCalculation = BMR × 1.9 An example for calculating the total calorie needs is shown as below: If you are sedentary, multiply your BMR (1745) by 1.2 = 2094. This is the total number of calories you need in order to maintain your current weight.• All members can make an appointment with individual trainers to seek some professional advice before joining particular clubs. All members can search and cancel their appointments. • The system should automatically generate confirmation messages for all bookings and appointments. • Your website should meet the following criteria: – secure and responsive – RESTful – cross-browser – interactive and dynamic– high accessibility and high (re)usability – maintainable and scalable • Your website should adopt a good design pattern and adhere to a good layout (like, Z/F layout etc). • Everyone can share the outcome of their fitness training through social networks. • For this project, there will be no need to use a backend database to hold all data — you can tentatively store all data within a JavaScript array or in a JSON file.• Choose a front-end framework using a shadow DOM to build this web application [40%] • Choose a front-end framework using a real DOM to build this web application [35%] • Write a maximum 4 pages reflective essay to compare the frameworks you chosen to complete in the second task. You are required to use this web application as a case study to make comprehensive comparisons between these frameworks. The justifications of your choices must be clearly elucidated. The template is available in Canvas. No marks will be rewarded if the template is not used for this task. Marks will be deducted if the whole essay is totally informal.[25%]4 Technologies • CSS3 Sprites (FlexBox, Media Query) • HTML5/6 • Bootstrap • JavaScript (IIFE) • jQuery and Ajax • Sass/LESS5 Submission Submit the source code of the sports arena through Canvas. You need to give a quick presentation on your final products built with two different frameworks. Marks will not be rewarded without presentation. The whole website must be deployed.
From the window you can see the sun shining in a lovely spring morning. Its Monday, 10am, and you are in the open plan office of a new start-up, OptimiseYourJourney, which will enter the market next year with a clear goal in mind: “leverage Big Data-based technologies for improving the user experience in transportation”. The start-up is at a very early stage, and has no clear product in mind yet. However, they have offered a short-term internship in their Data Analytics Department to help them exploring the datasets, technologies and techniques that can be applied in their future products. They do not pay very well (0€ per hour), but you see this as a good opportunity to complement your knowledge in the module Big Data & Analytics you are studying at the moment, so you have decided to give it a go.In the department meeting that has just finished your boss was particularly happy. During the weekend he was exploring public datasets over the internet and he found a dataset regarding the New York City Bike Sharing System: https://www.citibikenyc.com/ The official website (https://www.citibikenyc.com/system-data) provides publicly available datasets in a monthly basis. Each of these datasets amalgamate all the bike trips of the month. For example, the files “201905-citibike-tripdata.csv.zip”, “201906-citibike-tripdata.csv.zip” and “201907-citibike-tripdata.csv.zip” contain information for all the trips that took place in May, June and July of 2019, resp. Your boss thinks this dataset provides a great opportunity to explore the potential of MapReduce in analysing large datasets. He has already cleaned the dataset for you to perform some data analysis on it. DATASET: This dataset occupies ~80MB and contains 73 files. Each file contains all the trips registered the CitiBike system for a concrete day: • 2019_05_01.csv => All trips registered on the 1st of May of 2019. • 2019_05_02.csv => All trips registered on the 2nd of May of 2019. • … • 2019_07_12.csv => All trips registered on the 12th of July of 2019. Altogether, the files contain 444,110 rows. Each row contains the following fields: start_time , stop_time , trip_duration , start_station_id , start_station_name , start_station_latitude , start_station_longitude , stop_station_id , stop_station_name , stop_station_latitude , stop_station_longitude , bike_id , user_type , birth_year , gender , trip_id • (00) start_time ! A String representing the time the trip started at. ! Example: “2019/05/02 10:05:00” • (01) stop_time ! A String representing the time the trip finished at. ! Example: “2019/05/02 10:10:00” • (02) trip_duration ! An Integer representing the duration of the trip. ! Example: 300 • (03) start_station_id ! An Integer representing the ID of the CityBike station the trip started from. ! Example: 150 • (04) start_station_name ! A String representing the name of the CitiBike station the trip started from. ! Example: “E 2 St &; Avenue C”. • (05) start_station_latitude ! A Float representing the latitude of the CitiBike station the trip started from. ! Example: 40.7208736 • (06) start_station_longitude ! A Float representing the longitude of the CitiBike station the trip started from. ! Example: -73.98085795 • (07) stop_station_id ! An Integer representing the ID of the CityBike station the trip stopped at. ! Example: 150 • (08) stop_station_name ! A String representing the name of the CitiBike station the trip stopped at. ! Example: “E 2 St &; Avenue C”. • (09) stop_station_latitude ! A Float representing the latitude of the CitiBike station the trip stopped at. ! Example: 40.7208736 • (10) stop_station_longitude ! A Float representing the longitude of the CitiBike station the trip stopped at. ! Example: -73.98085795 • (11) bike_id ! An Integer representing the id of the bike used in the trip. ! Example: 33882. • (12) user_type ! A String representing the type of user using the bike (it can be either “Subscriber” or “Customer”). ! Example: “Subscriber”. • (13) birth_year ! An Integer representing the birth year of the user using the bike. ! Example: 1990. • (14) gender ! An Integer representing the gender of the user using the bike (it can be either 0 => Unknown; 1 => male; 2 => female). ! Example: 2. • (15) trip_id ! An Integer representing the id of the trip. ! Example: 190. TASKS / EXERCISES. The tasks / exercises to be completed as part of the assignment are described in the next pages of this PDF document. • The following exercises are placed in the folder my_code: 1. A01_Part1/A01_Part1.py 2. A01_Part2/A01_Part2.py 3. A01_Part3/A01_Part3.py 4. A01_Part4/my_mapper.py 5. A01_Part4/my_reducer.py 6. A01_Part5/my_mapper.py 7. A01_Part5/my_reducer.py 8. A01_Part6/my_mapper.py 9. A01_Part6/my_reducer.py Marks are as follows: o A01_Part1/A01_Part1.py => 16 marks o A01_Part2/A01_Part2.py => 16 marks o A01_Part3/A01_Part3.py => 18 marks o A01_Part4/my_mapper.py => 8 marks o A01_Part4/my_reducer.py => 8 marks o A01_Part5/my_mapper.py => 8 marks o A01_Part5/my_reducer.py => 8 marks o A01_Part6/my_mapper.py => 9 marks o A01_Part6/my_reducer.py => 9 marks Tasks: o A01_Part1/A01_Part1.py o A01_Part2/A01_Part2.py o A01_Part3/A01_Part3.py Complete the function my_main of the Python program. Do not modify the name of the function nor the parameters it receives. o A01_Part4/my_mapper.py o A01_Part5/my_mapper.py o A01_Part6/my_mapper.py Complete the function my_map of the Python program. Do not modify the name of the function nor the parameters it receives. o A01_Part4/my_reducer.py o A01_Part5/my_reducer.py o A01_Part6/my_reducer.py Complete the function my_reduce of the Python program. Do not modify the name of the function nor the parameters it receives. RUBRIC. Exercises 1-9. • 20% of the marks => Complete attempt of the exercise (even if it does not lead to the right solution or right format due to small differences). • 40% of the marks => Right solution and format (following the aforementioned rules) for the provided dataset. • 40% of the marks => Right solution and format (following the aforementioned rules) for any “Additional Dataset” test case we will generate. The marks will be allocated in a per test basis (i.e., if 4 extra test are tried, each of them will represent 10% of the marks). TEST YOUR SOLUTIONS. • The folder my_results contains the expected results for each exercise. o A01_Part1/result.txt o A01_Part2/result.txt o A01_Part3/result.txt o A01_Part4/1_my_map_simulation => 73 files for the output of each map process. o A01_Part4/2_my_sort_simulation/sort_1.txt => input for first reduce process. o A01_Part4/2_my_sort_simulation/sort_2.txt => input for second reduce process. o A01_Part4/3_my_reduce_simulation/reduce_sort_1.txt => output of first reduce. o A01_Part4/2_my_reduce_simulation/reduce_sort_2.txt => output of second reduce. o A01_Part5/1_my_map_simulation => 73 files for the output of each map process. o A01_Part5/2_my_sort_simulation/sort_1.txt => input for first reduce process. o A01_Part5/3_my_reduce_simulation/reduce_sort_1.txt => output of first reduce. o A01_Part6/1_my_map_simulation => 73 files for the output of each map process. o A01_Part6/2_my_sort_simulation/sort_1.txt => input for first reduce process. o A01_Part6/3_my_reduce_simulation/reduce_sort_1.txt => output of first reduce. • Moreover, the subfolder my_results/check_results allows you to see if your code is producing the expected output or not. o The file test_checker.py needs two folders and compares if their files are equal or not. o When you have completed one part (e.g., A01_Part4), copy the folder my_results/A01_Part4 into the folder my_results/check_results/Student_Attempts/A01_Part4. o Open the file test_checker.py and edit the line 112 with the value of the part you are attempting (e.g., part = 4). o Run the program test_checker.py. It will tell you whether your output is correct or not. For example, as an example let’s run the Python program test_checker.py to see if the solution attempt done by the student for A01_Part1 and A01_Part4 is correct or not. Ø python3.9 test_checker.py 1 ———————————————————- Checking : ./Assignment_Solutions/A01_Part1/result.txt ./Student_Attempts/A01_Part1/result.txt Test passed! ———————————————————- Congratulations, the code passed all the tests! ———————————————————- As we can see, the code of the student is correct, and thus it gets the marks. Ø python3.9 test_checker.py 4 ———————————————————- Checking : ./Assignment_Solutions/A01_Part4/2_my_sort_simulation/sort_1.txt ./Student_Attempts/A01_Part4/2_my_sort_simulation/sort_1.txt Test did not pass. ———————————————————- Checking : ./Assignment_Solutions/A01_Part4/2_my_sort_simulation/sort_2.txt ./Student_Attempts/A01_Part4/2_my_sort_simulation/sort_2.txt Test passed! ———————————————————- Checking : ./Assignment_Solutions/A01_Part4/3_my_reduce_simulation/reduce_sort_1.txt ./Student_Attempts/A01_Part4/3_my_reduce_simulation/reduce_sort_1.txt Test did not pass. ———————————————————- Checking : ./Assignment_Solutions/A01_Part4/3_my_reduce_simulation/reduce_sort_2.txt ./Student_Attempts/A01_Part4/3_my_reduce_simulation/reduce_sort_2.txt Test passed! ———————————————————- Sorry, the output of some files is incorrect! ———————————————————- As we can see, the code of the student is not correct, and thus it does not get the marks. The problem was that some output lines in some files were wrong. Main Message Use the program test_checker.py to ensure that all your exercises produce the expected output (and in the right format!). SUBMISSION DETAILS / SUBMISSION CODE OF CONDUCT. Submit to Canvas by the 14th of March, 11:59pm. • Submissions up to 1 week late will have 10 marks deducted. • Submissions up to 2 weeks late will have 20 marks deducted. On submitting the assignment you adhere to the following declaration of authorship. If you have any doubt regarding the plagiarism policy discussed at the beginning of the semester do not hesitate in contacting me. Declaration of Authorship I, ___YOUR NAME___, declare that the work presented in this assignment titled ‘Assignment 1: MapReduce’ is my own. I confirm that: • This work was done wholly by me as part of my BSc. in Software Development or BSc. in Web Development. • Where I have consulted the published work and source code of others, this is always clearly attributed. • Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this assignment source code and report is entirely my own work. EXERCISE 1. (16 marks) Technology: Python (without using the MapReduce simulator). Your task is to: • Compute the amount of trips starting from and finishing at each station_name. Complete the function my_main of the Python program. o Do not modify the name of the function nor the parameters it receives. o In particular, the function must read the dataset provided in input_folder and must open and write the results to output_file. o You can use the auxiliary function process_line which, given one line from a dataset file, returns a tuple with its content. o You can also program any other auxiliary functions you might need. Results: Output one text line per station_name. Lines must follow alphabetic order in the name of the station. Each line must have the following format: station_name t (total_trips_starting_from_the_station, total_trips_finishing_at_the_station) EXERCISE 2. (16 marks) Technology: Python (without using the MapReduce simulator). Your task is to: • Compute the top_n_bikes with highest total duration time for their trips. Complete the function my_main of the Python program. o Do not modify the name of the function nor the parameters it receives. o In particular, the function must read the dataset provided in input_folder and must open and write the results to output_file. The number of bikes to output is specified by the parameter top_n_bikes. For example, if top_n_bikes = 10, then you must output the 10 bikes with highest total duration time for their trips. o You can use the auxiliary function process_line which, given one line from a dataset file, returns a tuple with its content. o You can also program any other auxiliary functions you might need. Results: Output one text line per bike_id. Lines must follow decreasing order in highest total duration time for their trips. Each line must have the following format: bike_id t (total_duration_time_for_their_trips, total_number_of_trips) EXERCISE 3. (18 marks) Technology: Python (without using the MapReduce simulator). Your task is to: • Sometimes bikes are re-organised (moved) from station A to station B to balance the amount of bikes available in both stations. A truck operates this bike re-balancing service, and the trips done by-truck are not logged into the dataset. Compute all the times a given bike_id was moved by the truck re-balancing system. Complete the function my_main of the Python program. o Do not modify the name of the function nor the parameters it receives. o In particular, the function must read the dataset provided in input_folder and must open and write the results to output_file. The concrete bike to be tracked is specified by the parameter bike_id. For example, if bike_id = 35143, then you must track all the times the bike with id 35143 was moved by the truck re-balancing system. o You can use the auxiliary function process_line which, given one line from a dataset file, returns a tuple with its content. o You can also program any other auxiliary functions you might need. Results: Output one text line per moving trip. Lines must follow temporal order. Each line must have the following format: By_Truck t (time_it_was_logged_at_station2, station2_id, time_it_was_logged_at_station3, station3_id) For example, if the dataset contains the following 2 trips: o Trip1: A user used bike_id to start a trip from Station1 on 2019/05/10 09:00:00 and finished the trip at Station2 on 2019/05/10 10:00:00 o Trip2: A user used bike_id to start a trip from Station3 on 2019/05/10 11:00:00 and finished the trip at Station4 on 2019/05/10 12:00:00 And the dataset does not contain any extra trip: o Trip3: A user used bike_id to start a trip from Station2 and finish at Station3 anytime between 2019/05/10 10:00:00 and 2019/05/10 11:00:00 Then it is clear that the bike was moved from Station2 to Station3 by truck, and we output: By_Truck t (2019/05/10 10:00:00, Station2, 2019/05/10 11:00:00, Station3) EXERCISE 4 AND EXERCISE 7 (16 marks) Technology: Python – Use the MapReduce simulator. Your task is to: • Compute the amount of trips starting from and finishing at each station_name. my_mapper.py => Complete the function my_map of the Python program. o Do not modify the name of the function nor the parameters it receives. o In particular, the function must read the content of a file provided by my_input_stream and must write the results to the file provided by my_output_stream. There are no extra parameters provided via my_mapper_input_parameters. o You can use the auxiliary function process_line which, given one line from a dataset file, returns a tuple with its content. o You can also program any other auxiliary functions you might need. my_reducer.py => Complete the function my_reduce of the Python program. o Do not modify the name of the function nor the parameters it receives. o In particular, the function must read the content of a file provided by my_input_stream and must write the results to the file provided by my_output_stream. There are no extra parameters provided via my_reducer_input_parameters. o You can also program any other auxiliary functions you might need. Results: Output one text line per station_name. Lines must follow alphabetic order in the name of the station. Each line must have the following format: station_name t (total_trips_starting_from_the_station, total_trips_finishing_at_the_station) EXERCISE 5 AND EXERCISE 8 (16 marks) Technology: Python – Use the MapReduce simulator. Your task is to: • Compute the top_n_bikes with highest total duration time for their trips. my_mapper.py => Complete the function my_map of the Python program. o Do not modify the name of the function nor the parameters it receives. o In particular, the function must read the content of a file provided by my_input_stream and must write the results to the file provided by my_output_stream. There are no extra parameters provided via my_mapper_input_parameters. o You can use the auxiliary function process_line which, given one line from a dataset file, returns a tuple with its content. o You can also program any other auxiliary functions you might need. my_reducer.py => Complete the function my_reduce of the Python program. o Do not modify the name of the function nor the parameters it receives. o In particular, the function must read the content of a file provided by my_input_stream and must write the results to the file provided by my_output_stream. The extra parameter top_n_bikes is provided via my_reducer_input_parameters. o You can also program any other auxiliary functions you might need. Results: Output one text line per bike_id. Lines must follow decreasing order in highest total duration time for their trips. Each line must have the following format: bike_id t (total_duration_time_for_their_trips, total_number_of_trips) EXERCISE 6 AND EXERCISE 9 (18 marks) Technology: Python – Use the MapReduce simulator. Your task is to: • Sometimes bikes are re-organised (moved) from station A to station B to balance the amount of bikes available in both stations. A truck operates this bike re-balancing service, and the trips done by-truck are not logged into the dataset. Compute all the times a given bike_id was moved by the truck re-balancing system. my_mapper.py => Complete the function my_map of the Python program. o Do not modify the name of the function nor the parameters it receives. o In particular, the function must read the content of a file provided by my_input_stream and must write the results to the file provided by my_output_stream. The extra parameter bike_id is provided via my_mapper_input_parameters. o You can use the auxiliary function process_line which, given one line from a dataset file, returns a tuple with its content. o You can also program any other auxiliary functions you might need. my_reducer.py => Complete the function my_reduce of the Python program. o Do not modify the name of the function nor the parameters it receives. o In particular, the function must read the content of a file provided by my_input_stream and must write the results to the file provided by my_output_stream. There are no extra parameters provided via my_reducer_input_parameters. o You can also program any other auxiliary functions you might need. Results: Output one text line per moving trip. Lines must follow temporal order. Each line must have the following format: By_Truck t (time_it_was_logged_at_station2, station2_id, time_it_was_logged_at_station3, station3_id) For example, if the dataset contains the following 2 trips: o Trip1: A user used bike_id to start a trip from Station1 on 2019/05/10 09:00:00 and finished the trip at Station2 on 2019/05/10 10:00:00 o Trip2: A user used bike_id to start a trip from Station3 on 2019/05/10 11:00:00 and finished the trip at Station4 on 2019/05/10 12:00:00 And the dataset does not contain any extra trip: o Trip3: A user used bike_id to start a trip from Station2 and finish at Station3 anytime between 2019/05/10 10:00:00 and 2019/05/10 11:00:00 Then it is clear that the bike was moved from Station2 to Station3 by truck, and we output: By_Truck t (2019/05/10 10:00:00, Station2, 2019/05/10 11:00:00, Station3)
BACKGROUND. It’s been already a few weeks since you started your short-term internship in the Data Analytics Department of the start-up OptimiseYourJourney, which will enter the market next year with a clear goal in mind: “leverage Big Data technologies for improving the user experience in transportation”. Your contribution in Assignment 1 has proven the potential OptimiseYourJourney can obtain by applying MapReduce to analyse large-scale public transportation datasets as the one in the New York City Bike Sharing System: https://www.citibikenyc.com/ In the department meeting that has just finished your boss was particularly happy, again. • The very same dataset from Assignment 1 (let’s call it my_dataset_1) provides an opportunity to leverage other large-scale data analysis libraries, such as Spark SQL. • The graph structure of the dataset allows you to explore the potential of Spark GraphFrames, a small library of Spark specialised in the parallel execution of classical graph algorithms. To do so, two small graph examples (let’s call them my_dataset_2 and my_dataset_3) are provided to explore the classical algorithms of: o Dijkstra – for finding the shortest path from a source node to the remaining nodes. o PageRank – for assigning a value to each node based on its neighbourhood. DATASET 1: This dataset occupies ~80MB and contains 73 files. Each file contains all the trips registered the CitiBike system for a concrete day: • 2019_05_01.csv => All trips registered on the 1st of May of 2019. • 2019_05_02.csv => All trips registered on the 2nd of May of 2019. • … • 2019_07_12.csv => All trips registered on the 12th of July of 2019. Altogether, the files contain 444,110 rows. Each row contains the following fields: start_time , stop_time , trip_duration , start_station_id , start_station_name , start_station_latitude , start_station_longitude , stop_station_id , stop_station_name , stop_station_latitude , stop_station_longitude , bike_id , user_type , birth_year , gender , trip_id • (00) start_time ! A String representing the time the trip started at. ! Example: “2019/05/02 10:05:00” • (01) stop_time ! A String representing the time the trip finished at. ! Example: “2019/05/02 10:10:00” • (02) trip_duration ! An Integer representing the duration of the trip. ! Example: 300 • (03) start_station_id ! An Integer representing the ID of the CityBike station the trip started from. ! Example: 150 • (04) start_station_name ! A String representing the name of the CitiBike station the trip started from. ! Example: “E 2 St &; Avenue C”. • (05) start_station_latitude ! A Float representing the latitude of the CitiBike station the trip started from. ! Example: 40.7208736 • (06) start_station_longitude ! A Float representing the longitude of the CitiBike station the trip started from. ! Example: -73.98085795 • (07) stop_station_id ! An Integer representing the ID of the CityBike station the trip stopped at. ! Example: 150 • (08) stop_station_name ! A String representing the name of the CitiBike station the trip stopped at. ! Example: “E 2 St &; Avenue C”. • (09) stop_station_latitude ! A Float representing the latitude of the CitiBike station the trip stopped at. ! Example: 40.7208736 • (10) stop_station_longitude ! A Float representing the longitude of the CitiBike station the trip stopped at. ! Example: -73.98085795 • (11) bike_id ! An Integer representing the id of the bike used in the trip. ! Example: 33882. • (12) user_type ! A String representing the type of user using the bike (it can be either “Subscriber” or “Customer”). ! Example: “Subscriber”. • (13) birth_year ! An Integer representing the birth year of the user using the bike. ! Example: 1990. • (14) gender ! An Integer representing the gender of the user using the bike (it can be either 0 => Unknown; 1 => male; 2 => female). ! Example: 2. • (15) trip_id ! An Integer representing the id of the trip. ! Example: 190. DATASET 2: This dataset consists in the file tiny_graph.txt, which contains 26 edges (indeed, 13 edges, one on each direction) in a graph with 8 nodes. DATASET 3: This dataset consists in the file tiny_graph.txt, which contains 18 edges (indeed, 9 edges, one on each direction) in a graph with 6 nodes. TASKS / EXERCISES. The tasks / exercises to be completed as part of the assignment are described in the next pages: • The following exercises are placed in the folder my_code: 1. A02_Part1/A02_Part1.py 2. A02_Part2/A02_Part2.py 3. A02_Part3/A02_Part3.py 4. A02_Part4/A02_Part4.py 5. A02_Part5/A02_Part5.py Marks are as follows: 1. A02_Part1/A02_Part1.py => 18 marks 2. A02_Part2/A02_Part2.py => 18 marks 3. A02_Part3/A02_Part3.py => 18 marks 4. A02_Part4/A02_Part4.py => 18 marks 5. A02_Part5/A02_Part5.py => 28 marks Tasks: 1. A02_Part1/A02_Part1.py 2. A02_Part2/A02_Part2.py 3. A02_Part3/A02_Part3.py Complete the function my_main of the Python program. Do not modify the name of the function nor the parameters it receives. The entire work must be done within Spark SQL: § The function my_main must start with the creation operation ‘read’ above loading the dataset to Spark SQL. § The function my_main must finish with an action operation ‘collect’, gathering and printing by the screen the result of the Spark SQL job. § The function my_main must not contain any other action operation ‘collect’ other than the one appearing at the very end of the function. § The resVAL iterator returned by ‘collect’ must be printed straight away, you cannot edit it to alter its format for printing. 4. A02_Part4/A02_Part4.py Complete the function compute_page_rank of the Python program. Do not modify the name of the function nor the parameters it receives. The function must return a dictionary with (key, value) pairs, where: § Each key represents a node id. § Each value represents the pagerank value computed for this node id. 5. A02_Part5/A02_Part5.py Complete the function my_main of the Python program. Do not modify the name of the function nor the parameters it receives. The entire work must be done within Spark SQL: § The function my_main must start with the creation operation ‘read’ above loading the dataset to Spark SQL. § The function my_main must finish with an action operation ‘collect’, gathering and printing by the screen the result of the Spark SQL job. § The function my_main must not contain any other action operation ‘collect’ other than the one appearing at the very end of the function. § The resVAL iterator returned by ‘collect’ must be printed straight away, you cannot edit it to alter its format for printing. RUBRIC. Exercises 1-5. • 20% of the marks => Complete attempt of the exercise (even if it does not lead to the right solution or right format due to small differences). • 40% of the marks => Right solution and format (following the aforementioned rules) for the provided dataset. • 40% of the marks => Right solution and format (following the aforementioned rules) for any “Additional Dataset” test case we will generate. The marks will be allocated in a per test basis (i.e., if 4 extra test are tried, each of them will represent 10% of the marks). TEST YOUR SOLUTIONS. • The folder my_results contains the expected results for each exercise. o A02_Part1/result.txt o A02_Part2/result.txt o A02_Part3/result.txt o A02_Part4/result.txt o A02_Part5/result.txt • Moreover, the subfolder my_results/check_results allows you to see if your code is producing the expected output or not. o The file test_checker.py needs two folders and compares if their files are equal or not. When you have completed one part (e.g., A02_Part3), copy the folder my_results/A02_Part3 into the folder my_results/check_results/Student_Attempts/A02_Part3. o Open the file test_checker.py and edit the line 104 with the value of the part you are attempting (e.g., part = 3). Run the program test_checker.py. It will tell you whether your output is correct or not. For example, as an example let’s run the Python program test_checker.py to see if the solution attempt done by the student for A02_Part1 and A02_Part3 is correct or not. Ø python3.9 test_checker.py 1 ———————————————————- Checking : ./Assignment_Solutions/A02_Part1/result.txt ./Student_Attempts/A02_Part1/result.txt Test passed! ———————————————————- Congratulations, the code passed all the tests! ———————————————————- As we can see, the code of the student is correct, and thus it gets the marks. Ø python3.9 test_checker.py 3 ———————————————————- Checking : ./Assignment_Solutions/A02_Part3/result.txt ./Student_Attempts/A02_Part3/result.txt Test did not pass. ———————————————————- Sorry, the output of some files is incorrect! ———————————————————- As we can see, the code of the student is not correct, and thus it does not get the marks. The problem was that some output lines in some files were wrong. Main Message: Use the program test_checker.py to ensure that all your exercises produce the expected output (and in the right format!). SUBMISSION DETAILS / SUBMISSION CODE OF CONDUCT. Submit to Canvas by the 2nd of May, 11:59pm. • Submissions up to 1 week late will have 10 marks deducted. • Submissions up to 2 weeks late will have 20 marks deducted. On submitting the assignment you adhere to the following declaration of authorship. If you have any doubt regarding the plagiarism policy discussed at the beginning of the semester do not hesitate in contacting me. Declaration of Authorship I, ___ NAME___, declare that the work presented in this assignment, titled Big Data & Analytics – Assignment 2: Spark SQL & Spark GraphFrames, is my own. I confirm that: • This work was done wholly by me as part of my BSc. in Software Development or BSc. in Web Development. • Where I have consulted the published work and source code of others, this is always clearly attributed. • Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this assignment source code is entirely my own work. EXERCISE 1. (18 marks) Technology: Spark SQL. Your task is to: • Compute the amount of trips starting from and finishing at each station_name. Complete the function my_main of the Python program. o Do not modify the name of the function nor the parameters it receives. o The entire work must be done within Spark SQL: § The function my_main must start with the creation operation ‘read’ above loading the dataset to Spark SQL. § The function my_main must finish with an action operation ‘collect’, gathering and printing by the screen the result of the Spark SQL job. § The function my_main must not contain any other action operation ‘collect’ other than the one appearing at the very end of the function. § The resVAL iterator returned by ‘collect’ must be printed straight away, you cannot edit it to alter its format for printing. Results: Output one Row per station_name. Rows must follow alphabetic order in the name of the station. Each Row must have the following fields: Row(station, num_departure_trips, num_arrival_trips) EXERCISE 2. (18 marks) Technology: Spark SQL. Your task is to: • Compute the top_n_bikes with highest total duration time for their trips. Complete the function my_main of the Python program. o Do not modify the name of the function nor the parameters it receives. o The entire work must be done within Spark SQL: § The function my_main must start with the creation operation ‘read’ above loading the dataset to Spark SQL. § The function my_main must finish with an action operation ‘collect’, gathering and printing by the screen the result of the Spark SQL job. § The function my_main must not contain any other action operation ‘collect’ other than the one appearing at the very end of the function. § The resVAL iterator returned by ‘collect’ must be printed straight away, you cannot edit it to alter its format for printing. Results: Output one Row per bike_id. Rows must follow decreasing order in highest total duration time for their trips. Each Row must have the following fields: Row(bike_id, totalTime, numTrips=total_number_of_trips) EXERCISE 3. (18 marks) Technology: Spark SQL. Your task is to: • Sometimes bikes are re-organised (moved) from station A to station B to balance the amount of bikes available in both stations. A truck operates this bike re-balancing service, and the trips done by-truck are not logged into the dataset. Compute all the times a given bike_id was moved by the truck re-balancing system. Complete the function my_main of the Python program. o Do not modify the name of the function nor the parameters it receives. o The entire work must be done within Spark SQL: § The function my_main must start with the creation operation ‘read’ above loading the dataset to Spark SQL. § The function my_main must finish with an action operation ‘collect’, gathering and printing by the screen the result of the Spark SQL job. § The function my_main must not contain any other action operation ‘collect’ other than the one appearing at the very end of the function. § The resVAL iterator returned by ‘collect’ must be printed straight away, you cannot edit it to alter its format for printing. Results: Output one Row per moving trip. Rows must follow temporal order. Each Row must have the following fields: Row(start_time, start_station_name, stop_time, stop_station_name) For example, if the dataset contains the following 2 trips: o Trip1: A user used bike_id to start a trip from Station1 on 2019/05/10 09:00:00 and finished the trip at Station2 on 2019/05/10 10:00:00 o Trip2: A user used bike_id to start a trip from Station3 on 2019/05/10 11:00:00 and finished the trip at Station4 on 2019/05/10 12:00:00 And the dataset does not contain any extra trip: o Trip3: A user used bike_id to start a trip from Station2 and finish at Station3 anytime between 2019/05/10 10:00:00 and 2019/05/10 11:00:00 Then it is clear that the bike was moved from Station2 to Station3 by truck, and we output: Row(start_time=2019/05/10 10:00:00, start_station_name=Station2, stop_time=2019/05/10 11:00:00, stop_station_name=Station3) EXERCISE 4. (18 marks) Technology: Python (without using the Spark library). Your task is to: • Compute your own sequential implementation of the PageRank algorithm for the nodes of a given graph (e.g., my_dataset_2). Complete the function compute_page_rank of the Python program. o Do not modify the name of the function nor the parameters it receives. o The function must return a dictionary with (key, value) pairs, where: § Each key represents a node id. § Each value represents the pagerank value computed for this node id. Results: Given the requested dictionary, the program automatically outputs one (key, value) pair per line. Lines follow a decreasing order in the page rank value of the node. Each line has the following format: id=key ; pagerank=value EXERCISE 5. (28 marks) Technology: Spark SQL. Your task is to: • Using Spark SQL, compute the shortest path distance from a source node to the remaining nodes of the graph. Complete the function my_main of the Python program. o Do not modify the name of the function nor the parameters it receives. o The entire work must be done within Spark SQL: § The function my_main must start with the creation operation ‘read’ above loading the dataset to Spark SQL. § The function my_main must finish with an action operation ‘collect’, gathering and printing by the screen the result of the Spark SQL job. § The function my_main must not contain any other action operation ‘collect’ other than the one appearing at the very end of the function. § The resVAL iterator returned by ‘collect’ must be printed straight away, you cannot edit it to alter its format for printing. o The difficulty of this exercise is on coming up with your own Spark SQL implementation of the Dijkstra shortest path algorithm. That is, you must implement a Spark SQL program following the steps of the Dijkstra algorithm explained in class. Results: Output one Row per bike_id. Rows must follow a decreasing order in the cost of the path. Each Row must have the following fields: Row(id, cost, path)
You are not allowed to use Spring Boot. You are not allowed to use JPA. Specification A database holds household information • the Eircode of the house • the address of the house • details about the occupants of the house The data about the occupants: • Name • Age • Occupation o “scholar” for school-going children o “pre-school” for pre-school children The application should allow a user to (using a menu based system): • Search for a household by Eircode, listing the details of the people in the household • Add a household, along with its occupant(s) • Add a new person and assign that person to a household • Move a person from one household to another • Delete a household, along with its occupants • Delete a person • Display some statistics* o the average age of householders o the number of students in the households o the number of OAPs (aged 65+) in the household No security is needed. You must write 4 unit tests, Refer to the rubric for expected standards. * Use SQL queries here – don’t jut retrieve lists and do the work in Java as I have seen done in the past – SQL has queries for these. Penalties Penalties for late submission are applied as per section 4.4.2 of the Exam Regulations (see http://www.mycit.ie/examregulations (Links to an external site.)). If you have issues that you think I should be made aware of which may cause you to be late submitting work, please contact me directly using Inbox in Canvas. Do not plagiarize – don’t copy code, don’t provide code, don’t work too closely with someone else on the project. Trying to be clever by taking someone else’s code and renaming classes and variables is not a good strategy and will likely result in a zero grade or worse. Advice o Set up your Spring project first, adding all of your dependencies. o Create a Spring Beans Configuration File, adding beans for a data source and a JdbcTemplate, injecting the data source into the latter. Also add the boilerplate code for h2 (embedded database) and a web server so that you can view the embedded database through a browser. o Design the database. o Create the schema for the database. o Run the program and view the database. o Create data for the database in SQL. o Run the program and view the database. o Create any classes you might need (they usually map to each table in the database). o Create a RowMapper() for each class and/or row from a resultset. o Create one or more repositories/data access objects for queries through the database. o Autowire the JdbcTemplate bean into this class. Run your queries through this bean. o Test the DAO’s methods in the main() method e.g. if you have written a findAll() method for a particular table e.g. findAll() for households, get the main() method to extract the DAO bean, and use that bean to call the method, using a loop with System.out.print() statements to display the details. This is sufficient as proof-of-concept for yourself that your database queries work and you can extract data from the database through the JdbcTemplate bean. o Add a service layer (interface and implementation) which may include business rules e.g. in a save() method you might make sure that the Eircode is unique o Add more methods to the service layer and implement these methods in the DAO layer. o Don’t forget your unit tests – why not create them as you go along.
