{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":686465375,"defaultBranch":"main","name":"recipe_data_processing","ownerLogin":"anguswg-ucsb","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-09-02T22:10:07.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/66215643?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1699549715.0","currentOid":""},"activityList":{"items":[{"before":"fe764be77a9449373740e18fe87ef0210eb1383d","after":"f83963e311b2a1774b247ea01cbbbaffa4399af0","ref":"refs/heads/main","pushedAt":"2024-01-11T02:39:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"updated package lambdas script to remove unneeded files installed from pip, changes to recipe scraper lambda code, the model lambda code, and the batching/messaging process with SQS and lambda, decreased number of executions and increased message batch size","shortMessageHtmlLink":"updated package lambdas script to remove unneeded files installed fro…"}},{"before":"3c16426cf53c7da66af28a85698037e6c268aea7","after":"fe764be77a9449373740e18fe87ef0210eb1383d","ref":"refs/heads/main","pushedAt":"2024-01-06T21:13:30.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"cleaned up terraform code and put each resource type in its own script, created lambda funtions for chunking CSV and using hugging face model to extract ingredients, created shell script to make initial resources that stay static for the rest of the infrastructure","shortMessageHtmlLink":"cleaned up terraform code and put each resource type in its own scrip…"}},{"before":"2e2eabe11056dbe69db1ee67343af9dbb74e21bb","after":"3c16426cf53c7da66af28a85698037e6c268aea7","ref":"refs/heads/main","pushedAt":"2024-01-02T14:51:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"reorganizing terraform files by moving each resource type into its own file, created a dockerfile and folder for extracting ingredients from raw ingredients list using hugging face nlp model, changed etl pipeline to start by CSV upload to an S3 bucket instead of JSONs","shortMessageHtmlLink":"reorganizing terraform files by moving each resource type into its ow…"}},{"before":"31094ef35a27232e1c86f8f586a369a284330a99","after":"2e2eabe11056dbe69db1ee67343af9dbb74e21bb","ref":"refs/heads/main","pushedAt":"2023-12-29T19:56:03.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"trying to push commit that changes lambda code to be triggered by SQS queue and process Sqs messages in batches, also aded terraform IaC to build the required infra, changed the name of the lambda handler function and function code folder","shortMessageHtmlLink":"trying to push commit that changes lambda code to be triggered by SQS…"}},{"before":"3f28dfa484b5cdbefbf7858bb3c7adef203d4192","after":"31094ef35a27232e1c86f8f586a369a284330a99","ref":"refs/heads/main","pushedAt":"2023-12-28T19:05:18.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"updated recipe scraper scripts and put driver code into scrapers/runners/ directory, updated lambda deployment scripts and terraform resource","shortMessageHtmlLink":"updated recipe scraper scripts and put driver code into scrapers/runn…"}},{"before":"3de4971ab63dc8f0a956418ed6705d4e66ab676c","after":"3f28dfa484b5cdbefbf7858bb3c7adef203d4192","ref":"refs/heads/main","pushedAt":"2023-12-27T00:46:39.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"terraform and lambda python code for handling recipe JSON uploads to s3 bucket and trigger scraping process and data dump to stage S3 bucket and error logs in a dynamoDB table","shortMessageHtmlLink":"terraform and lambda python code for handling recipe JSON uploads to …"}},{"before":"35b3286821eb84824a4306a59d278f243eeca22c","after":"3de4971ab63dc8f0a956418ed6705d4e66ab676c","ref":"refs/heads/main","pushedAt":"2023-12-23T21:19:45.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"removed scrape ops api key from lambda function code, secret has been revoked and is not usable anymore","shortMessageHtmlLink":"removed scrape ops api key from lambda function code, secret has been…"}},{"before":"4e4564dc5aeb0fd16378ab8bb705f56271055ac9","after":"35b3286821eb84824a4306a59d278f243eeca22c","ref":"refs/heads/main","pushedAt":"2023-12-23T21:15:37.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"finalizing scraper code to use in lambda function, created a reduce_to_json.py script to reduce the finalized processed dataset into individual json files that will be uplaoded to s3 for further processing, added sh/package_lambdas.sh script for zipping up lambda function code and dependecies","shortMessageHtmlLink":"finalizing scraper code to use in lambda function, created a reduce_t…"}},{"before":"f07277d17f436fa9c065240abdaee668f1dd2038","after":"4e4564dc5aeb0fd16378ab8bb705f56271055ac9","ref":"refs/heads/main","pushedAt":"2023-12-21T21:14:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"added scrapers for allrecipes, fooddotcom, and began adding an aws lambda function and terraform code for processing recipe URLs","shortMessageHtmlLink":"added scrapers for allrecipes, fooddotcom, and began adding an aws la…"}},{"before":"48a20ea4c5a244991a9ba59b8f100373f15dd1da","after":"f07277d17f436fa9c065240abdaee668f1dd2038","ref":"refs/heads/main","pushedAt":"2023-12-14T17:53:09.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"moved scraper functions to a seperate scraper_utils file and made a new scrapers/ directory to use for compiling scraped datasets, added TODO on allrecipes.py","shortMessageHtmlLink":"moved scraper functions to a seperate scraper_utils file and made a n…"}},{"before":"b227caef25bc924a850e998441b5cd720a2cc65c","after":"48a20ea4c5a244991a9ba59b8f100373f15dd1da","ref":"refs/heads/main","pushedAt":"2023-12-09T14:55:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"first attempt at creating recipe data scrapers for all recipes in recipe_scraper.py file","shortMessageHtmlLink":"first attempt at creating recipe data scrapers for all recipes in rec…"}},{"before":"c0d14f9ebcb178e267fbe3810005e66ad9ef1b3b","after":"b227caef25bc924a850e998441b5cd720a2cc65c","ref":"refs/heads/main","pushedAt":"2023-11-20T16:01:14.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"Added a sorting step to the process_dataset_recipeNLG function to sort the ingredients by lowercase alphabetical order, added some code in processing_raw.py that adds each processed dataset to a list and then concatonates all of the procecssed datasets after they have all been added to the list, then this is saved as the final recipe dataset","shortMessageHtmlLink":"Added a sorting step to the process_dataset_recipeNLG function to sor…"}},{"before":"a815dd4255f3fc041af85c8421fb3ea219bb3816","after":"c0d14f9ebcb178e267fbe3810005e66ad9ef1b3b","ref":"refs/heads/main","pushedAt":"2023-11-18T20:46:41.000Z","pushType":"pr_merge","commitsCount":4,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"Merge pull request #6 from mkayeterry/main\n\nAltered structure and function of processing_raw (formerly db_prep)","shortMessageHtmlLink":"Merge pull request <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1993449093\" data-permission-text=\"Title is private\" data-url=\"https://github.com/anguswg-ucsb/recipe_data_processing/issues/6\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/anguswg-ucsb/recipe_data_processing/pull/6/hovercard\" href=\"https://github.com/anguswg-ucsb/recipe_data_processing/pull/6\">#6</a> from mkayeterry/main"}},{"before":"e60051851750e4cca1c753af5d0f3d7f4a04215b","after":"a815dd4255f3fc041af85c8421fb3ea219bb3816","ref":"refs/heads/main","pushedAt":"2023-11-11T20:30:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"cleaned up NER dataset processing functions and put into a single function, created a function for creating unique ingredients dataset","shortMessageHtmlLink":"cleaned up NER dataset processing functions and put into a single fun…"}},{"before":"b8f311e00e8a9c5f35a3d60f1c952f8f04292a18","after":"e60051851750e4cca1c753af5d0f3d7f4a04215b","ref":"refs/heads/main","pushedAt":"2023-11-09T17:41:12.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"deleted a bunch of old scripts and reorganized db_prep and left TODO comments in db_prep and utils.py","shortMessageHtmlLink":"deleted a bunch of old scripts and reorganized db_prep and left TODO …"}},{"before":null,"after":"6722162db4bada0b9c8dd21dd29a7fe76ba03d72","ref":"refs/heads/old_infra","pushedAt":"2023-11-09T17:08:35.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"stashed old infrastructure layout in this branch","shortMessageHtmlLink":"stashed old infrastructure layout in this branch"}},{"before":"a7e03176c9c3ccb2372764dfcc7cb7c7a912701d","after":"b8f311e00e8a9c5f35a3d60f1c952f8f04292a18","ref":"refs/heads/main","pushedAt":"2023-11-01T21:10:53.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"added unique ingredients table to shell script and to postgres database","shortMessageHtmlLink":"added unique ingredients table to shell script and to postgres database"}},{"before":"067f58816ec4a4446d1dd9131c3974c7b06af27e","after":"a7e03176c9c3ccb2372764dfcc7cb7c7a912701d","ref":"refs/heads/main","pushedAt":"2023-10-25T12:50:23.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"updated security group of ec2 and added .DS_Store to gitignore","shortMessageHtmlLink":"updated security group of ec2 and added .DS_Store to gitignore"}},{"before":"b972c9575fe86e6008d37119de7155c36d1be887","after":"067f58816ec4a4446d1dd9131c3974c7b06af27e","ref":"refs/heads/main","pushedAt":"2023-10-19T12:36:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"cleaning up db_prep script","shortMessageHtmlLink":"cleaning up db_prep script"}},{"before":"c51323e17e6dd244cec38d156ad2e43db8f4915b","after":"b972c9575fe86e6008d37119de7155c36d1be887","ref":"refs/heads/main","pushedAt":"2023-10-19T12:19:45.000Z","pushType":"pr_merge","commitsCount":5,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"Merge pull request #5 from anguswg-ucsb/ec2_postgres\n\nChanging from Aurora serverless DB to EC2 postgres DB","shortMessageHtmlLink":"Merge pull request <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1952101419\" data-permission-text=\"Title is private\" data-url=\"https://github.com/anguswg-ucsb/recipe_data_processing/issues/5\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/anguswg-ucsb/recipe_data_processing/pull/5/hovercard\" href=\"https://github.com/anguswg-ucsb/recipe_data_processing/pull/5\">#5</a> from anguswg-ucsb/ec2_postgres"}},{"before":"779560424545270cf67eb7a63c3978ca4f9f1822","after":"f275bc547289c85f00d4f14ab8bddaf2b8a5ef9e","ref":"refs/heads/ec2_postgres","pushedAt":"2023-10-19T12:17:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"moved old aurora TF code","shortMessageHtmlLink":"moved old aurora TF code"}},{"before":"a7ef16e6b399d17a70c2f99fbd08b58f54f38094","after":"779560424545270cf67eb7a63c3978ca4f9f1822","ref":"refs/heads/ec2_postgres","pushedAt":"2023-10-19T12:10:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"created terraform scripts to use ec2 db instead of aurora serverless","shortMessageHtmlLink":"created terraform scripts to use ec2 db instead of aurora serverless"}},{"before":"f0f8d6e4023de97cca9024b0604ef0f6c46acee5","after":"a7ef16e6b399d17a70c2f99fbd08b58f54f38094","ref":"refs/heads/ec2_postgres","pushedAt":"2023-10-17T12:32:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"updating s3 copy csv to db lambda function","shortMessageHtmlLink":"updating s3 copy csv to db lambda function"}},{"before":null,"after":"f0f8d6e4023de97cca9024b0604ef0f6c46acee5","ref":"refs/heads/ec2_postgres","pushedAt":"2023-10-16T22:10:19.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"refactoring teraform infrastrcuture as code to run a posstgres db on an ec2 instance instead of on aurora serverless","shortMessageHtmlLink":"refactoring teraform infrastrcuture as code to run a posstgres db on …"}},{"before":"ebf2de43012184f81b03ade299832554fb4fde9b","after":"c51323e17e6dd244cec38d156ad2e43db8f4915b","ref":"refs/heads/main","pushedAt":"2023-10-13T14:53:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"updated table creation in lambda function created by TF","shortMessageHtmlLink":"updated table creation in lambda function created by TF"}},{"before":"c68f45647c209d56aaaeea52c0abbb2701226ac5","after":"ebf2de43012184f81b03ade299832554fb4fde9b","ref":"refs/heads/main","pushedAt":"2023-10-10T13:34:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"updated gitignore and working on creating tables and inserting data functions","shortMessageHtmlLink":"updated gitignore and working on creating tables and inserting data f…"}},{"before":"5037081456b46454ab4c18e7677665655ec012d1","after":"c68f45647c209d56aaaeea52c0abbb2701226ac5","ref":"refs/heads/main","pushedAt":"2023-10-07T19:50:52.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"removed old code from db_prep","shortMessageHtmlLink":"removed old code from db_prep"}},{"before":"8155188b2b662b39d6f2e801c38e3968a09645b9","after":"5037081456b46454ab4c18e7677665655ec012d1","ref":"refs/heads/main","pushedAt":"2023-10-07T19:39:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"removed rest_api directory and added RDS proxy server for lambda and RDS","shortMessageHtmlLink":"removed rest_api directory and added RDS proxy server for lambda and RDS"}},{"before":"7ff648829b6eef3826dfd5e40ee09ae4c6e9d9a2","after":"8155188b2b662b39d6f2e801c38e3968a09645b9","ref":"refs/heads/main","pushedAt":"2023-10-07T16:02:47.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"working on setting up RDS Proxy for lambda, removed tempcode runner file","shortMessageHtmlLink":"working on setting up RDS Proxy for lambda, removed tempcode runner file"}},{"before":"1c9416be376c6c683c402867e9de64efbc91631a","after":"7ff648829b6eef3826dfd5e40ee09ae4c6e9d9a2","ref":"refs/heads/main","pushedAt":"2023-09-25T23:44:22.000Z","pushType":"pr_merge","commitsCount":2,"pusher":{"login":"anguswg-ucsb","name":"Angus Watters-NOAA","path":"/anguswg-ucsb","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/66215643?s=80&v=4"},"commit":{"message":"Merge pull request #4 from anguswg-ucsb/rest_api_aw\n\nworking on terraform scripts for uploading data to s3, lambda functio…","shortMessageHtmlLink":"Merge pull request <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1912442771\" data-permission-text=\"Title is private\" data-url=\"https://github.com/anguswg-ucsb/recipe_data_processing/issues/4\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/anguswg-ucsb/recipe_data_processing/pull/4/hovercard\" href=\"https://github.com/anguswg-ucsb/recipe_data_processing/pull/4\">#4</a> from anguswg-ucsb/rest_api_aw"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"Y3Vyc29yOnYyOpK7MjAyNC0wMS0xMVQwMjozOTo0Mi4wMDAwMDBazwAAAAPc2UkF","startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wMS0xMVQwMjozOTo0Mi4wMDAwMDBazwAAAAPc2UkF","endCursor":"Y3Vyc29yOnYyOpK7MjAyMy0wOS0yNVQyMzo0NDoyMi4wMDAwMDBazwAAAAOIqjp_"}},"title":"Activity · anguswg-ucsb/recipe_data_processing"}