Cloud Integration

 Cloud Integration

When we start deploying multiple applications, they will inevitably need to communicate with one another.
Two types of communication mechanism are:

Issues with Synchronous application:

  • Sudden traffic spike
  • suddenly need to encode 1000 videos when usually it was only 10.

Solution:

  • To have a decoupled for of application.
    • SQS: queue model
    • SNS: pub/sub model
    • Kinesis: real-time data streaming model
  • These services can scale independently from our applications.

Amazon SQS:

  • oldest AWS offerings over 10 years old.
  • Fully managed service (~serverless), use to decouple applications.
  • Scales from 1 message per second to 10,000s per second.
  • Default retention of messages: 4 days, maximum of 14 days
  • No limit to how many messages can be in the queue
  • Messages are deleted after they're read by consumers
  • Low latency (<10 ms on publish and receive)
  • Consumers share the work to read messages & scale horizontally


Amazon Kinesis Data Streams

  • Kinesis - real-time big data streaming
  • Managed service to collect, process and analyze real-time streaming data at any scale
  • Amazon Kinesis Data Streaming - low latency streaming to ingest data at scale from hundreds of thousands of sources.
  • Amazon Data Firehose - load Kinesis Data Streams into Amazon S3, Redshift, OpenSearch, etc...


Amazon SNS


  • The "event publishers" only sends message to one SNS topic
  • As many "event subscribers" as we want to listen to the SNS topic notifications
  • Each subscriber to the topic will get all the messages
  • Up to 12,500,000 subscriptions per topic, 100,000 topics limit

Amazon MQ

  • SQS, SNS are "cloud native" services: proprietary protocols from AWS
  • Traditional applications running from on-premises may use open protocols such as: MQTT, AMQP, STOMP, Openwire, WSS
  • When migrating to the cloud, instead of re-engineering the application to use SQS and SNS, we can use Amazon MQ
  • Amazon MQ is a managed message broker service for 
    • Rabbit MQ
    • Active MQ
  • Amazon MQ does'nt scale as much as SQS/SNS
  • Amazon MQ runs on servers, can run in Mutli-AZ with failover
  • Amazon MQ has both queue featur (~SQS) and topic features (~SNS)

Summary:

  • SQS:
    • Queue service in AWS
    • Multiple Producers, messages are kept up to 14 days
    • Multiple Consumers share the read and delete messages when done
    • Used to decouple applications in AWS
  • SNS:
    • Notification service in AWS
    • Subscribers: Email, Lambda, SQS, HTTP, Mobile...
    • Multiple Subscribers, send all messages to all of them
    • No message retention
  • Kinesis: real-time data streaming, persistence and analysis
  • Amazon MQ: managed message broker for ActiveMQ and RabbitMQ in the cloud (MQTT, AMQP.. protocols)

Comments

Post a Comment

Popular posts from this blog

Machine Learning

Image
 Machine Learning Amazon Rekognition Find objects, people, text, scenes in images and videos using ML Facial analysis and facial search to do user verification, people counting Create a database of "familiar faces" or compare against celebrities Use cases: Labeling Content Moderation Text Detection Face Detection and Analysis (gender, age ranges, emotions...) Face Search and Verification Celebrity Recognition Pathing (ex: for sports game analysis) Amazon Transcribe Automatically convert speech to text Uses a deep learning process called automatic speech recognition (ASR) to convert speech to text quickly and accurately Automatically remove Personally Identifiable Information (PII) using Redaction Supports Automatic Language Identification for multi-lingual audio Use cases: transcribe customer service calls automate closed captioning and subtitling generate metadata for medis assets to create a fully searchable archive Amazon Polly Turn text into lifelike speech using deep lea...
Read more

Cloud Computing and IT

Image
Cloud Computing and IT How Website work?   Server composed of: Compute, memory, storage(file or database), network - Routers, switch, DNS server. IT terminology: Network: cables, routers and servers connected with each other. Router: A networking device that forwards data packets between computer networks. They know where to send your packets on the internet. Switch: takes a packet and send it to the correct server/client on your network. Problems: Pay rent for datacenter Pay for power, cooling and maintenance Adding and replacing hardware takes time Scaling is limited Hire 24/7 team to monitor the infrastructure Plan for taking care of disasters. eg: Earth-Quake, Power-Supply issue etc... Demand: How to externalize all above task? Solution: Cloud Cloud Computing: (on-demand/pay-as-you-go/) Cloud computing is the on-demand delivery of compute power, database storage, applications and other IT resources. Cloud services are pay-as-you-go pricing. Provision exactly the right type and ...
Read more

Cloud Monitoring

Image
 Cloud Monitoring Amazon CloudWatch Metrics CloudWatch provides metrics for every services in AWS Metric is a variable to monitor ( CPUUtilization, Networking...) Metrics have timestamps Can create CloudWatch dashboards of metrics Important Metrics EC2 instances: CPU Utilization, Status Checks, Network (not RAM) Default metrics every 5 minutes Option for Detailed Monitoring ($$$): metrics every 1 minute EBS volumes: Disk Read/Writes S3 buckets: BucketSizeBytes, NumberOfObjects, AllRequests Billing: Total Estimated Charge (only in us-east-1) Service Limits: how many you've been using a service API Custom mertics: push your own metrics Amazon CloudWatch Alarms Alarms are used to trigger notifications for any metric Alarms actions Auto Scaling: increase or decrease EC2 instances "desired" count EC2 Actions: stop, terminate, reboot or recover an EC2 instance SNS notifications: send a notification into an SNS topic Various options (sampling, %, max, min, etc...) Can choose the...
Read more