I’m an novice to data engineering, i have some data , say json (sample data below) . my understanding of aws glue is, i can create a glue job to read these files from s3 and create glue tables or catalogs (not sure what the difference is here), but will i be able to maintain a foreign key type of relation here . in my example course_id is the foreign key relation . if yes, is there a example for this.
if no, to maintain this relationship , i know i can push this data to a database, and write a script (probably python to do so) . now if i’m writing a python script myself , to take data from a glue catalog to an rds, can that script be a part of glue job as well. how or where do i specify it to run a custom python script.
--courses
course_id, title , building, department
10, 'Business', '123 sample street, 'eco'
11, 'eCommerce', '27 Ersatz blvd, 'Retail'
-- students
student_id, first, last, registration_id, course_id
1, 'Jan', 'Novak', '49151555824', 10),
2, 'Pablo', 'Garcia', '49162555176', 11),
I created a aws glue job to read json files and create a table in aws glue, which works but i’m not sure how to maintain relationship between these data files in aws glue.