Skip to content

Commit

Permalink
Fix old names in subscriber template and py code
Browse files Browse the repository at this point in the history
  • Loading branch information
SollyzDev committed Jul 16, 2024
1 parent b50ac88 commit c4eb99e
Show file tree
Hide file tree
Showing 6 changed files with 33 additions and 33 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,6 @@ jobs:
- run: |-
mkdir build
yq ".Resources.LogsLambda.Properties.Code.ZipFile = \"$(sed 's/\"/\\\"/g' ./handler.py)\"" cloudformation-stacks/forwarder.template.yaml > build/axiom-cloudwatch-forwarder-cloudformation-stack.yaml
yq ".Resources.BackfillerLambda.Properties.Code.ZipFile = \"$(sed 's/\"/\\\"/g' ./backfill.py)\"" cloudformation-stacks/subscriber.template.yaml > build/axiom-cloudwatch-subscriber-cloudformation-stack.yaml
yq ".Resources.SubscriberLambda.Properties.Code.ZipFile = \"$(sed 's/\"/\\\"/g' ./subscriber.py)\"" cloudformation-stacks/subscriber.template.yaml > build/axiom-cloudwatch-subscriber-cloudformation-stack.yaml
yq ".Resources.AxiomCloudWatchLogsSubscriber.Properties.Code.ZipFile = \"$(sed 's/\"/\\\"/g' ./logs_subscriber.py)\"" cloudformation-stacks/log-groups-listener.template.yaml > build/axiom-cloudwatch-log-groups-listener-cloudformation-stack.yaml
- run: cat build/*
2 changes: 1 addition & 1 deletion .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
- run: |-
mkdir build
yq ".Resources.LogsLambda.Properties.Code.ZipFile = \"$(sed 's/\"/\\\"/g' ./handler.py)\"" cloudformation-stacks/forwarder.template.yaml > build/axiom-cloudwatch-forwarder-cloudformation-stack.yaml
yq ".Resources.BackfillerLambda.Properties.Code.ZipFile = \"$(sed 's/\"/\\\"/g' ./backfill.py)\"" cloudformation-stacks/subscriber.template.yaml > build/axiom-cloudwatch-subscriber-cloudformation-stack.yaml
yq ".Resources.SubscriberLambda.Properties.Code.ZipFile = \"$(sed 's/\"/\\\"/g' ./subscriber.py)\"" cloudformation-stacks/subscriber.template.yaml > build/axiom-cloudwatch-subscriber-cloudformation-stack.yaml
yq ".Resources.AxiomCloudWatchLogsSubscriber.Properties.Code.ZipFile = \"$(sed 's/\"/\\\"/g' ./logs_subscriber.py)\"" cloudformation-stacks/subscriber.template.yaml > build/axiom-cloudwatch-log-groups-listener-cloudformation-stack.yaml
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ Axiom CloudWatch Lambda uses the following CloudFormation stacks:
1. [Create an Axiom account](https://app.axiom.co).
2. Create a dataset in Axiom.
3. Create an API token in Axiom with permissions to ingest data to the dataset you created.
4. [Click this link to launch the Stack](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=cloudwatch-ingester-axiom&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-forwarder-cloudformation-stack.yaml).
5. [Click this link to automatically subscribe to all existing log groups](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=cloudwatch-backfiller-axiom&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-subscriber-cloudformation-stack.yaml).
6. [Click this link to automatically subscribe to new log groups](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=cloudwatch-subscriber-axiom&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-log-groups-listener-cloudformation-stack.yaml).
4. [Click this link to launch the Stack](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-forwarder&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-forwarder-cloudformation-stack.yaml).
5. [Click this link to automatically subscribe to all existing log groups](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-subscriber&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-subscriber-cloudformation-stack.yaml).
6. [Click this link to automatically subscribe to new log groups](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=axiom-cloudwatch-log-groups-listener&templateURL=https://axiom-cloudformation.s3.amazonaws.com/stacks/axiom-cloudwatch-log-groups-listener-cloudformation-stack.yaml).

## Logs Subscriber architecture
## Log Groups Listener architecture

The Logs Subscriber stack does the following:
The Log Groups Listener stack does the following:

- It creates an S3 bucket for Cloudtrail.
- It creates a trail to capture the creation of new log groups.
Expand Down
32 changes: 16 additions & 16 deletions cloudformation-stacks/subscriber.template.yaml
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
Parameters:
LambdaFunctionName:
Type: String
Description: Name of the AWS Lambda Function.
Default: cloudwatch-backfiller-axiom
Description: Name of the AWS Lambda function.
Default: axiom-cloudwatch-subscriber
AllowedPattern: ".+" # required
AxiomCloudWatchLambdaIngesterARN:
AxiomCloudWatchForwarderLambdaARN:
Type: String
Description: The ARN of the AWS Lambda Function that is used to ingest data to axiom.
Description: The ARN of the Axiom CloudWatch Forwarder Lambda function used to ship logs to Axiom.
AllowedPattern: ".+" # required
CloudWatchLogGroupsPrefix:
Type: String
Description: The Prefix of cloudwatch log groups to subscribe to the AWS Lambda ingester.
Description: The Prefix of CloudWatch log groups to trigger the Axiom CloudWatch Forwarder lambda.
Default: "" # all
Resources:
BackfillerPolicy:
SubscriberPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyDocument:
Expand All @@ -26,10 +26,10 @@ Resources:
- lambda:RemovePermission
Effect: Allow
Resource: '*'
PolicyName: axiom-cloudwatch-backfiller-lambda-policy
PolicyName: axiom-cloudwatch-subscriber-lambda-policy
Roles:
- !Ref 'BackfillerRole'
BackfillerRole:
- !Ref 'SubscriberRole'
SubscriberRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Expand All @@ -42,7 +42,7 @@ Resources:
- lambda.amazonaws.com
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
BackfillerLambda:
SubscriberLambda:
Type: AWS::Lambda::Function
Properties:
FunctionName: !Ref LambdaFunctionName
Expand All @@ -52,18 +52,18 @@ Resources:
Code:
ZipFile: |
# DO NOT EDIT
# CI will replace these comments with the code from ./backfill.py
# CI will replace these comments with the code from ./subscriber.py
Role: !GetAtt
- BackfillerRole
- SubscriberRole
- Arn
Environment:
Variables:
AXIOM_CLOUDWATCH_LAMBDA_INGESTER_ARN: !Ref 'AxiomCloudWatchLambdaIngesterARN'
AXIOM_CLOUDWATCH_FORWARDER_LAMBDA_ARN: !Ref 'AxiomCloudWatchForwarderLambdaARN'
LOG_GROUP_PREFIX: !Ref 'CloudWatchLogGroupsPrefix'
BackfillerInvoker:
SubscriberInvoker:
Type: AWS::CloudFormation::CustomResource
DependsOn: BackfillerLambda
DependsOn: SubscriberLambda
Version: "1.0"
Properties:
ServiceToken: !GetAtt BackfillerLambda.Arn
ServiceToken: !GetAtt SubscriberLambda.Arn
StackName: !Ref AWS::StackName
File renamed without changes.
20 changes: 10 additions & 10 deletions backfill.py → subscriber.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
cloudwatch_logs_client = boto3.client("logs")
lambda_client = boto3.client("lambda")

axiom_cloudwatch_lambda_ingester_arn = os.getenv("AXIOM_CLOUDWATCH_LAMBDA_INGESTER_ARN")
axiom_cloudwatch_forwarder_lambda_arn = os.getenv("AXIOM_CLOUDWATCH_FORWARDER_LAMBDA_ARN")
log_group_prefix = os.getenv("LOG_GROUP_PREFIX", "")
log_groups_return_limit = int(os.getenv("LOG_GROUPS_LIMIT", 10))

Expand Down Expand Up @@ -44,7 +44,7 @@ def get_log_groups(token=None):
def remove_permission(lambda_arn):
lambda_client.remove_permission(
FunctionName=lambda_arn,
StatementId="cloudwatch-backfiller-axiom",
StatementId="axiom-cloudwatch-subscriber",
)


Expand Down Expand Up @@ -75,7 +75,7 @@ def create_statement(region, account_id, lambda_arn):
)
lambda_client.add_permission(
FunctionName=lambda_arn,
StatementId="cloudwatch-backfiller-axiom",
StatementId="axiom-cloudwatch-subscriber",
Action="lambda:InvokeFunction",
Principal=f"logs.amazonaws.com",
SourceArn=source_arn,
Expand Down Expand Up @@ -103,22 +103,22 @@ def create_subscription_filter(log_group_arn, lambda_arn):


def lambda_handler(event: dict, context=None):
if axiom_cloudwatch_lambda_ingester_arn is None:
raise Exception("AXIOM_CLOUDWATCH_LAMBDA_INGESTER_ARN is not set")
if axiom_cloudwatch_forwarder_lambda_arn is None:
raise Exception("AXIOM_CLOUDWATCH_LAMBDA_FORWARDER_ARN is not set")

aws_account_id = context.invoked_function_arn.split(":")[4]
region = os.getenv("AWS_REGION")

# create permission for lambda
try:
remove_permission(axiom_cloudwatch_lambda_ingester_arn)
remove_permission(axiom_cloudwatch_forwarder_lambda_arn)
except Exception as e:
logger.error(f"Error removing permission: {e}")

create_statement(region, aws_account_id, axiom_cloudwatch_lambda_ingester_arn)
create_statement(region, aws_account_id, axiom_cloudwatch_forwarder_lambda_arn)

ingester_lambda_group_name = (
"/aws/lambda/" + axiom_cloudwatch_lambda_ingester_arn.split(":")[-1]
"/aws/lambda/" + axiom_cloudwatch_forwarder_lambda_arn.split(":")[-1]
)

def log_groups(token=None):
Expand All @@ -136,14 +136,14 @@ def log_groups(token=None):

try:
delete_subscription_filter(
region, aws_account_id, axiom_cloudwatch_lambda_ingester_arn
region, aws_account_id, axiom_cloudwatch_forwarder_lambda_arn
)
except Exception:
pass

try:
create_subscription_filter(
group["arn"], axiom_cloudwatch_lambda_ingester_arn
group["arn"], axiom_cloudwatch_forwarder_lambda_arn
)
except cloudwatch_logs_client.exceptions.LimitExceededException as error:
logger.error(
Expand Down

0 comments on commit c4eb99e

Please sign in to comment.