updating app on S3 and merge conflicts

.github/CONTRIBUTING.md

@@ -0,0 +1,132 @@
+This document is the single source of truth on how to contribute to this codebase. Please feel free to browse the open issues and file new ones. All feedback is welcome!
+
+----
+
+## Topics
+
+* [Prerequisites](#prerequisites)
+    * [Contributor License Agreement](#contributor-license-agreement)
+    * [Code of Conduct](#code-of-conduct)
+    * [Setup Development Environment](#setup-development-environment)
+* [Contribution Workflow](#contribution-workflow)
+    * [Feature Requests and Bug Reports](#feature-requests-and-bug-reports)
+    * [Fixing Issues](#fixing-issues)
+    * [Pull Requests](#pull-requests)
+    * [Code Review](#code-review)
+    * [Documentation](#documentation)
+* [Maintainers](#maintainers)
+
+----
+
+## Prerequisites
+When contributing to this repository, please first discuss the change you wish to make via a GitHub issue, Slack message, email, or via other channels with the owners of this repository.
+
+##### Contributor License Agreement
+At the moment, we can only accept pull requests submitted from either:
+* Splunk employees or
+* Individuals that have signed our contributors' agreement
+
+If you wish to be a contributing member of our community, please see the agreement [for individuals](https://www.splunk.com/goto/individualcontributions) or [for organizations](https://www.splunk.com/goto/contributions).
+
+##### Code of Conduct
+Please make sure to read and observe our [Code of Conduct](https://github.com/splunk/security_content/wiki/Code-of-Conduct). Please follow it in all of your interactions involving the project.
+
+##### Setup Development Environment
+see [Developing Content](https://github.com/splunk/security_content/wiki/Developing-Content)
+
+## Contribution Workflow
+Help is always welcome! For example, documentation can always use improvement. There's always code that can be clarified, functionality that can be extended, and tests to be added to guarantee behavior. If you see something you think should be fixed, don't be afraid to own it.
+
+##### Feature Requests and Bug Reports
+Have ideas on improvements? See something that needs work? While the community encourages everyone to contribute code, it is also appreciated when someone reports an issue. Please report any issues or bugs you find through [GitHub's issue tracker](https://github.com/splunk/security_content/issues).
+
+If you are reporting a bug, please include:
+
+* Your operating system name and version
+* Any details about your local setup that might be helpful in troubleshooting (ex. Python interpreter version, Splunk version, etc.)
+* Detailed steps to reproduce the bug
+
+We'd also like to hear about your propositions and suggestions. Feel free to submit them as issues and:
+
+* Explain in detail how they should work
+* Note that keeping the scope as narrow as possible will make the suggestion easier to implement
+
+##### Fixing Issues
+Look through our [issue tracker](https://github.com/splunk/security_content/issues) to find problems to fix! Feel free to comment and tag corresponding stakeholders or full-time maintainers of this project with any questions or concerns.
+
+##### Pull Requests
+What is a "pull request"? It informs the project's core developers about the changes you want to review and merge. Once you submit a pull request, it enters a stage of code review where you and others can discuss its potential modifications and maybe even add more commits to it later on.
+
+If you want to learn more, please consult this [tutorial on how pull requests work](https://help.github.com/articles/using-pull-requests/) in the [GitHub Help Center](https://help.github.com/).
+
+Here's an overview of how you can make a pull request against this project:
+
+1. Fill out the [Splunk Contribution Agreement](https://www.splunk.com/goto/contributions).
+2. Fork the [security_content GitHub repository](https://github.com/splunk/security_content)
+3. Clone your fork using git and create a branch off of develop
+
+    ```
+    $ git clone git@github.com:YOUR_GITHUB_USERNAME/security_content.git
+    $ cd security_content
+
+    # This project uses 'develop' for all development activity, so create your branch off that
+    $ git checkout -b your-bugfix-branch-name develop
+    ```
+
+4. Make your changes, commit, and push (once your tests have passed)
+
+    ```
+    $ cd security_content
+    $ git commit -m "<insert helpful commit message>"
+    $ git push
+    ```
+
+5. Submit a pull request through the GitHub website, using the changes from your forked codebase.
+6. We utilize [pre-commit hooks](https://pre-commit.com/) to ensure consistent formatting and as an additional method of preventing credentials from ending up in our repo accidentally. We highly suggest you also install `pre-commit` and the hooks for this project.
+
+##### Code Review
+There are two aspects of code review: giving and receiving.
+
+To make it easier for your PR to receive reviews, keep in mind that the reviewers will need you to:
+* Follow the project coding conventions
+* Write good commit messages
+* Break large changes into a logical series of smaller patches which individually make easily understandable changes, and in aggregate solve a broader issue
+
+Reviewers, the people providing the review, are highly encouraged to revisit the [Code of Conduct](https://github.com/splunk/security_content/wiki/Code-of-Conduct) and must go above and beyond to promote a collaborative, respectful community.
+
+When reviewing PRs from others, [The Gentle Art of Patch Review](http://sage.thesharps.us/2014/09/01/the-gentle-art-of-patch-review/) suggests an iterative series of focuses designed to lead new contributors to positive collaboration, such as:
+
+* Is the idea behind the contribution sound?
+* Is the contribution architected correctly?
+* Is the contribution polished?
+
+For this project, we require at least one approval. A build from our continuous integration system must also be successful off of your branch. Please note that any new changes made with your existing pull request during review will automatically unapproved and re-trigger another build/round of tests.
+
+##### Documentation
+We can always use improvements to our documentation! Anyone can contribute to these docs--whether you’re new to the project, you’ve been around a long time, or if you just can’t stand seeing typos.
+
+Here's what's needed?
+
+1. More complementary documentation. Have you something unclear?
+2. More examples or generic templates that others can use.
+3. Blog posts, articles, and such are all very appreciated.
+
+You can also edit documentation files directly in the GitHub web interface, without creating a local copy. This can be convenient for small typos or grammar fixes.
+
+## Maintainers
+
+If you need help, feel free to tag one of the active maintainers of this project in a post or comment. We'll do our best to reach out to you as quickly as we can.
+
+```
+# Active maintainers marked with (*)
+
+(*) Bhavin Patel
+(*) Michael Haag
+(*) Jose Hernandez
+(*) Lou Stella
+(*) Patrick Bareib
+(*) Eric McGinnis
+(*) Rod Soto
+(*) Teoderick Contreras
+(*) Gowtham Rajendran
+```

.github/labeler.yml

@@ -0,0 +1,20 @@
+Detections:
+- changed-files:
+  - any-glob-to-any-file: 
+    - detections/**
+
+Stories:
+- changed-files:
+  - any-glob-to-any-file: stories/*
+
+Playbooks:
+- changed-files:
+  - any-glob-to-any-file: playbooks/*
+
+Macros:
+- changed-files:
+  - any-glob-to-any-file: macros/*
+
+Lookups:
+- changed-files:
+  - any-glob-to-any-file: lookups/*

.github/pull_request_template.md

@@ -9,3 +9,10 @@ _What does this PR have in it? Screenshots are worth 1000 words 😄_
 - [ ] Validated SPL logic.
 - [ ] Validated tags, description, and how to implement.
 - [ ] Verified references match analytic.
+- [ ] Confirm updates to lookups are handled properly.
+
+### Notes For Submitters and Reviewers
+
+- If you're submitting a PR from a fork, ensuring the box to allow updates from maintainers is checked will help speed up the process of getting it merged.
+- Checking the output of the `build` CI job when it fails will likely show an error about what is failing. You may have a very descriptive error of the specific field(s) in the specific file(s) that is causing an issue. In some cases, its also possible there is an issue with the YAML. Many of these can be caught with the pre-commit hooks if you set them up. These errors will be less descriptive as to what exactly is wrong, but will give you a column and row position in a specific file where the YAML processing breaks. If you're having trouble with this, feel free to add a comment to your PR tagging one of the maintainers and we'll be happy to help troubleshoot it.
+- Updates to existing lookup files can be tricky, because of how Splunk handles application updates and the differences between existing lookup files being updated vs new lookups. You can read more [here](https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Admin/PrivateApps#Manage_lookups_in_Splunk_Cloud_Platform) but the short version is that any changes to lookup files need to bump the datestamp in the lookup CSV filename, and the reference to it in the YAML needs to be updated.

.github/workflows/build.yml

@@ -23,7 +23,7 @@ jobs:
 
       - name: Install Python Dependencies and ContentCTL and Atomic Red Team
         run: |
-          pip install contentctl
+          pip install contentctl==4.2.0
           git clone --depth=1 --single-branch --branch=master https://github.com/redcanaryco/atomic-red-team.git
       
       - name: Running build with enrichments

.github/workflows/format_test_results.py

@@ -36,16 +36,14 @@ def main():
     for detection in data['tested_detections']:
         for test in detection['tests']:
             if test['test_type'].strip() == "unit":  # Check if the test type is "unit"
-                name = detection['name'].strip()
-                status = 'PASS' if test['success'] else 'FAIL'
-                test_type = test['test_type'].strip()
-                exception = test.get('exception', 'N/A')  # Get exception if exists, else 'N/A'
+                name = (detection.get('name') or 'N/A').strip()
+                status = 'PASS' if test.get('success') else 'FAIL'
+                test_type = (test.get('test_type') or 'N/A').strip()
+                exception = (test.get('exception') or 'N/A')  # Get exception if exists, else 'N/A'
                 if status == 'FAIL':
                     print(f"{name:<80} | 🔴 {status:<6}  | {test_type:<10} | {exception:<50}")
                 else:
                     print(f"{name:<80} | 🟢  {status:<6} | {test_type:<10} | {'-':<50}")
-    # Print table footer
-    # print(f"{'----':<80} | {'------':<6} | {'---------':<10} | {'---------':<50}")
 
     # Check if total_fail is a valid integer and greater than or equal to one
     print("\n")  # Print a newline for separation

.github/workflows/labeler.yml

@@ -0,0 +1,18 @@
+name: "Pull Request Labeler"
+on:
+- pull_request_target
+
+jobs:
+  labeler:
+    permissions:
+      contents: read
+      pull-requests: write
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with: 
+        repository: "splunk/security_content"
+    - uses: actions/labeler@v5
+      with:
+        sync-labels: true
+        configuration-path: '.github/labeler.yml'

.github/workflows/unit-testing.yml

@@ -24,7 +24,7 @@ jobs:
         - name: Install Python Dependencies and ContentCTL
           run: |
             python -m pip install --upgrade pip
-            pip install contentctl
+            pip install contentctl==4.2.0
            
         # Running contentctl test with a few arguments, before running the command make sure you checkout into the current branch of the pull request. This step only performs unit testing on all the changes against the target-branch. In most cases this target branch will be develop
         # Make sure we check out the PR, even if it actually lives in a fork
@@ -36,11 +36,10 @@ jobs:
             echo "Current Branch (Head Ref): ${{ github.head_ref }}"
             echo "Target Branch (Base Ref): ${{ github.base_ref }}"
             git pull > /dev/null 2>&1
-            git fetch origin pull/${{ github.event.pull_request.number }}/head:${{ github.head_ref }}
-            #We must specifically get the PR's target branch from security_content, not the one that resides in the fork PR's forked repo
-            git switch ${{ github.head_ref }}
-            #git checkout ${{ github.head_ref }}
-            #echo "The target branch for this PR is ${{ github.base_ref }}"
+            #We checkout into a new branch - new_branch_for_testing to avoid name collisions with develop incase the forked PR is from develop
+            git fetch origin pull/${{ github.event.pull_request.number }}/head:new_branch_for_testing
+            #We must specifically get the PR's target branch from security_content, not the one that resides in the fork PR's forked repo           
+            git switch new_branch_for_testing
             contentctl test --disable-tqdm --no-enable-integration-testing --post-test-behavior never_pause mode:changes --mode.target-branch ${{ github.base_ref }}
             echo "contentctl test - COMPLETED"
           continue-on-error: true
@@ -65,4 +64,4 @@ jobs:
           run: |       
             echo "This job will fail if there are failures in unit-testing"  
             python .github/workflows/format_test_results.py >> $GITHUB_STEP_SUMMARY
-            echo "The Unit testing is completed. See details in the unit-testing job summary UI "
+            echo "The Unit testing is completed. See details in the unit-testing job summary UI "

.pre-commit-config.yaml

@@ -1,12 +1,15 @@
 repos:
-- repo: https://github.com/pre-commit/pre-commit-hooks
-  rev: v2.1.0  # Use the ref you want to point at
-  hooks:
-    - id: check-executables-have-shebangs
-      exclude: 'package/bin/da_ess_contentupdate/|package/bin/splunklib/|venv/'
-    - id: check-json
-    - id: check-symlinks
-    - id: check-yaml
-    - id: pretty-format-json
-      args: [--autofix]
-    - id: requirements-txt-fixer
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.6.0 # Use the ref you want to point at
+    hooks:
+      - id: check-executables-have-shebangs
+        exclude: "package/bin/da_ess_contentupdate/|package/bin/splunklib/|venv/"
+      - id: check-json
+      - id: check-symlinks
+      - id: check-yaml
+      - id: pretty-format-json
+        args: [--autofix]
+      - id: requirements-txt-fixer
+      - id: detect-aws-credentials
+      - id: detect-private-key
+      - id: forbid-submodules

CODE_OF_CONDUCT.md

@@ -0,0 +1,3 @@
+# Code Of Conduct
+
+The Code of Conduct for this repository can be found online [in our wiki](https://github.com/splunk/security_content/wiki/Code-of-Conduct). 

README.md

@@ -74,7 +74,7 @@ To view an up-to-date detection coverage map for all the content tagged with MIT
 * [data_sources/](data_sources/): Defines the data sources, the necessary TA or App to collect them and the fields provided that can be used by the detections.
 
 # Contribution 🥰
-We welcome feedback and contributions from the community! Please see our [contributing to the project](https://github.com/splunk/security_content/wiki/Contributing-to-the-Project) for more information on how to get involved.
+We welcome feedback and contributions from the community! Please see our [contributing to the project](./.github/CONTRIBUTING.md) for more information on how to get involved.
 
 ## Support 💪
 If you are a Splunk Enterprise customer with a valid support entitlement contract and have a Splunk-related question, you can open a support case on the https://www.splunk.com/ support portal.