#StackBounty: #amazon-web-services #amazon-ec2 #ssh #amazon-elastic-beanstalk #ssh-tunnel ssh tunnel script hangs forever on beanstalk …

Bounty: 50

I’m attempting to create a ssh tunnel when deploying an application to aws beanstalk. The script is hanging forever on the deployment. I want to put the tunnel as a background process that is always connected on application deploy and I can’t seem to understand why it is hanging forever

The script is

  "/home/ec2-user/eclair-ssh-tunnel.sh":
    mode: "000500" # u+rx
    owner: root
    group: root
    content: |
      cd /root
      eval $(ssh-agent -s)
      DISPLAY=":0.0" SSH_ASKPASS="./askpass_script" ssh-add eclair-test-key </dev/null
      # we want this command to keep running in the backgriund
      # so we add & at then end
      nohup ssh -L 48682:localhost:8080 ubuntu@[host...] -N &

and here is the output i am getting from /var/log/eb-activity.log

[2019-06-14T14:53:23.268Z] INFO  [15615] - [Application update suredbits-api-root-0.37.0-testnet-ssh-tunnel-fix-port-9@30/AppDeployStage1/AppDeployPostHook/01_eclair-ssh-tunnel.sh] : Starting activity...

The ssh tunnel is spawned, and I can find it by doing

[ec2-user@ip-172-31-25-154 ~]$ ps aux | grep 48682
root     16047  0.0  0.0 175560  6704 ?        S    14:53   0:00 ssh -L 48682:localhost:8080 ubuntu@ec2-34-221-186-19.us-west-2.compute.amazonaws.com -N

If i kill that process, the deployment continues as expected, which indicates that the bug is in the tunnel script. I can’t seem to find out where though.


Get this bounty!!!

#StackBounty: #amazon-web-services #aws-sdk #aws-batch #aws-sdk-cpp Why can I not override container variables for multi-node parallel …

Bounty: 250

I am working with AWS Batch. My goal is to create a multi-node parallel job through the AWS SDK for C++. For this, I have created a job definition as per the instructions here.

I am working with the AWS C++ SDK, and I noticed that when I try to override either environmental variables or commands, nothing is actually transferred to the job.

Interestingly, the same code works perfectly fine for a job which uses a normal job definition (as opposed to a multi-node one):

#include <aws/batch/BatchClient.h>
#include <aws/batch/model/ContainerOverrides.h>
#include <aws/batch/model/KeyValuePair.h>
#include <aws/batch/model/SubmitJobRequest.h>
#include <aws/core/Aws.h>
#include <aws/core/utils/Outcome.h>

int main(void)
{
    Aws::SDKOptions options;
    Aws::InitAPI(options);

    Aws::Batch::BatchClient batchClient;
    Aws::Batch::Model::SubmitJobRequest submitJobRequest;
    Aws::Batch::Model::SubmitJobOutcome submitJobOutcome;
    Aws::Batch::Model::ContainerOverrides containerOverrides;
    Aws::Batch::Model::KeyValuePair envVariable;

    envVariable.SetName("foo");
    envVariable.SetValue("bar");

    containerOverrides.AddEnvironment(envVariable); // This does nothing for a multi-node job definition.
    containerOverrides.AddCommand("foobarbaz"); // This does nothing for a multi-node job definition.

    submitJobRequest.SetJobName("myjob");
    submitJobRequest.SetJobDefinition("arn:aws:...."); // This string is an example. I have used the actual job definition ARN.
    submitJobRequest.SetJobQueue("arn:aws:...."); // This string is an exmaple. I have used the actual queue ARN.
    submitJobRequest.SetContainerOverrides(containerOverrides);

    submitJobOutcome = batchClient.SubmitJob(submitJobRequest);

    Aws::ShutdownAPI(options);

    return 0;

}

Should I be using a different API for multi-node parallel jobs?


Get this bounty!!!

#StackBounty: #amazon-web-services #terraform #amazon-elastic-beanstalk #terraform-provider-aws AWS: add second internal load balancer …

Bounty: 50

Is it possible to manipulate beanstalk’s autoscaling group and target group in terraform adding additional (internal) load balancer? If yes, how?

I want to have 2 load balancers one internal and the other one public.
I found this workaround from AWS:

https://aws.amazon.com/it/blogs/networking-and-content-delivery/using-static-ip-addresses-for-application-load-balancers/

Is there any other smarter solution?


Get this bounty!!!

#StackBounty: #amazon-web-services #amazon-ec2 #amazon-vpc #amazon-elb EC2 VPC Intermittent outbound connection timeouts

Bounty: 50

My production web service consists of:

  • Auto-scaling group
  • Network loadbalancer (ELB)
  • 2x EC2 instances as web servers

This configuration was running fine until yesterday when one of the EC2 instances started to experience RDS and ElastiCache timeouts. The other instance continues to run without issues.

During investigation, I noticed that outgoing connections in general sometimes experience large delays:

[ec2-user@ip-10-0-5-9 logs]$ time curl -s www.google.com > /dev/null

real    0m7.147s -- 7 seconds
user    0m0.007s
sys     0m0.000s
[ec2-user@ip-10-0-5-9 logs]$ time curl -s www.google.com > /dev/null

real    0m3.114s
user    0m0.007s
sys     0m0.000s
[ec2-user@ip-10-0-5-9 logs]$ time curl -s www.google.com > /dev/null

real    0m0.051s
user    0m0.006s
sys     0m0.000s
[ec2-user@ip-10-0-5-9 logs]$ time curl -s www.google.com > /dev/null

real    1m6.309s -- over a minute!
user    0m0.009s
sys     0m0.000s

[ec2-user@ip-10-0-5-9 logs]$ traceroute -n -m 1 www.google.com
traceroute to www.google.com (172.217.7.196), 1 hops max, 60 byte packets
 1  * * *
[ec2-user@ip-10-0-5-9 logs]$ traceroute -n -m 1 www.google.com
traceroute to www.google.com (172.217.7.196), 1 hops max, 60 byte packets
 1  216.182.226.174  17.706 ms * *
[ec2-user@ip-10-0-5-9 logs]$ traceroute -n -m 1 www.google.com
traceroute to www.google.com (172.217.8.4), 1 hops max, 60 byte packets
 1  216.182.226.174  20.364 ms * *
[ec2-user@ip-10-0-5-9 logs]$ traceroute -n -m 1 www.google.com
traceroute to www.google.com (172.217.7.132), 1 hops max, 60 byte packets
 1  216.182.226.170  12.680 ms  12.671 ms *

Further analysis shows that if I manually detach the ‘bad’ instance from the auto-scaling group, removing it as a load balancer target, the problem instantly goes away. As soon as I add it back, the problem returns.

These nodes are m5.xlarge and appear to have excess capacity, so I don’t believe it’s a resource issue.

UPDATE: It seems related to load on the node. I put load back on last night and it seemed stable, but this morning as load is growing, outbound traffic (DB, etc.) start to fail. I’m really stuck not understanding how this outbound traffic is being impacted at all. The other identical node has no issues, even with 100% of the traffic versus 50%.

traceroute to 54.14.xx.xx (54.14.xx.xx), 1 hops max, 60 byte packets
 1  216.182.226.174  18.691 ms 216.182.226.166  18.341 ms 216.182.226.174  18.660 ms
traceroute to 54.14.xx.xx (54.14.xx.xx), 1 hops max, 60 byte packets
 1  * * *

What is the 216.182.226.166 IP? Is it related to the VPC IGW?

Node stats:

  • m5.xlarge
  • CPU ~ 7.5%
  • load average: 0.18, 0.29, 0.29
  • Network IN: ~8M bytes/minute

UPDATE: With 1 of the 2 nodes attached to the load balancer, things appear to run stable — with all traffic on one node. After I add the 2nd node to the load balancer, after some period of time (hours – days), one of the nodes starts to exhibit outbound connection issues describe above (connection timing out to database, Google, etc.). In this state, the other node is working fine. Replacing the ‘bad’ or reinstating it in the load balancer allow things to run fine for a while. These images use Amazon Linux 2 (4.14.114-103.97.amzn2.x86_64).


Get this bounty!!!

#StackBounty: #python #amazon-web-services #automation #amazon-product-api Programmatic Checkout from Amazon.com ( & Amazon.in )

Bounty: 200

I am trying to automate the purchase of products listed on Amazon.com in Python. We would like to make 1-click purchase with the saved payment details.

I understand Zinc enables the automated product purchase through their API. Does Amazon have a Sandbox account to test this ? Any idea how this can be done

Any suggestion, pointers in this regard will be really helpful.


Get this bounty!!!

#StackBounty: #active-directory #amazon-web-services #aws-directory-service Is it possible to choose which objects get synchronized in …

Bounty: 50

In Azure AD Connect Sync, it is possible to configure filtering. This is described as:

By using filtering, you can control which objects appear in Azure Active Directory (Azure AD) from your on-premises directory. The default configuration takes all objects in all domains in the configured forests.

I need this similar functionality in AWS Managed AD synchronized to an on-premises directory. Is it possible? If so, how?


Get this bounty!!!

#StackBounty: #.net #amazon-web-services #asp.net-core #amazon-ec2 #.net-core Service segmentation fault in AWS EC2

Bounty: 250

I have my service running in EC2 (under systemd). It is a self-contained app built for .Net Core 2.1.
From time to time (a few times a week) it crashes with SEGV.

Apr 30 21:20:51 ip-10-4-226-55 kernel: traps: App.Name[26176] general protection ip:7f22da3609da sp:7f1fedf11510 error:0 in libc-2.26.so[7f22da2e3000+1ad000]

Apr 30 21:20:51 ip-10-4-226-55 systemd: appname.service: main process exited, code=killed, status=11/SEGV

Apr 30 21:20:51 ip-10-4-226-55 systemd: Unit appname.service entered failed state.

Apr 30 21:20:51 ip-10-4-226-55 systemd: appname.service failed.

For some reason, crash dump is not created (even though I removed the size limit).
How can I investigate the problem further? What can be the source of the issue?


Get this bounty!!!

#StackBounty: #amazon-web-services #apache-spark #pyspark #aws-glue AWS Glue pushdown predicate not working properly

Bounty: 500

I’m trying to optimize my Glue/PySpark job by using push down predicates.

start = date(2019, 2, 13) 
end = date(2019, 2, 27) 
print(">>> Generate data frame for ", start, " to ", end, "... ")
relaventDatesDf = spark.createDataFrame([
    Row(start=start, stop=end)
])
relaventDatesDf.createOrReplaceTempView("relaventDates")

relaventDatesDf = spark.sql("SELECT explode(generate_date_series(start, stop)) AS querydatetime FROM relaventDates")
relaventDatesDf.createOrReplaceTempView("relaventDates")
print("===LOG:Dates===")
relaventDatesDf.show()

flightsGDF = glueContext.create_dynamic_frame.from_catalog(database = "xxx", table_name = "flights", transformation_ctx="flights", push_down_predicate="""
    querydatetime BETWEEN '%s' AND '%s'
    AND querydestinationplace IN (%s)
""" % (start.strftime("%Y-%m-%d"), today.strftime("%Y-%m-%d"), ",".join(map(lambda s: str(s), arr))))

However it appears, that Glue still attempts to read data outside the specified date range?

INFO S3NativeFileSystem: Opening 's3://.../flights/querydestinationplace=12191/querydatetime=2019-03-01/part-00045-6cdebbb1-562c-43fa-915d-93b125aeee61.c000.snappy.parquet' for reading
INFO FileScanRDD: Reading File path: s3://.../flights/querydestinationplace=12191/querydatetime=2019-03-10/part-00021-34a13146-8fb2-43de-9df2-d8925cbe472d.c000.snappy.parquet, range: 0-11797922, partition values: [12191,17965]
WARN S3AbortableInputStream: Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
INFO S3NativeFileSystem: Opening 's3://.../flights/querydestinationplace=12191/querydatetime=2019-03-10/part-00021-34a13146-8fb2-43de-9df2-d8925cbe472d.c000.snappy.parquet' for reading
WARN S3AbortableInputStream: Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.

Notice the querydatetime=2019-03-01 and querydatetime=2019-03-10 its outside the specified range of 2019-02-13 - 2019-02-27. Is that why there’s the next line “aborting HTTP connection” tho? It goes on to say “This is likely an error and may result in sub-optimal behavior” is something wrong?

I wonder if the problem is because it does not support BETWEEN inside the predicate or IN?


Get this bounty!!!

#StackBounty: #mysql #amazon-web-services #cpu-usage #mysql-replication Amazon Aurora read-only slaves slowly increase to 100% CPU and …

Bounty: 100

We have a master database instance hosted on AWS Aurora (mysql) and have many read-only slaves being replicated from it. The master and the 4-12 autoscaling slaves are currently of db.r4.4xlarge size and engine version: 5.7.12.

Each slave comes online and performs for a few days but over that time its CPU usage slowly increases until each one has to be individually killed. Once killed another is automatically spun up and it continues.

Here’s the performance graph of the slaves:
enter image description here

As you can see at 11pm our warehouse closes and CPU utilisation falls until the next day when it spikes and climbs above the previous day’s. This increases day on day until it reaches 100% and has to be killed.

Have any of you guys every seen this pattern before and could give us a hint on where the problem might lie?


Get this bounty!!!

#StackBounty: #api #rust #amazon-web-services User registration in Actix w/ DynamoDB

Bounty: 50

I am trying to puzzle out api based user registration for a service I am building, but I have a strong feeling that my code is not optimal.

It feels like passing “state” in the way I did prevents the server from being multithreaded, as I have no clue how actix could share data that isn’t Send or Sync across multiple threads.

I also interact with DynamoDb by using a .sync() call on the request, which means that i block on waiting for Dynamo to respond to me. This seems like it would be bad for performance of a web server but I don’t know how to avoid it without something like async/await.

Another issue is that I cannot guarantee that a user has a unique email without making a second table in Dynamo. I don’t want to expose a user’s email to the outside world, so for everything else I would need to use some sort of other unique identifier (here I went with a uuid), but for registration I would still need to check that requirement…somehow. But if I make a second table I introduce the possibility of an inconsistent state since to my knowledge I can’t make dynamo affect two tables atomically.

I also just returned out the created user id to the client, but is there something more useful I could have done? Should I return a token in this response? More info about the created user?

Any and all feedback would be greatly appreciated.

use actix_web::error::InternalError;
use actix_web::http::StatusCode;
use actix_web::{http, server, App, Json, Responder, State};
use bcrypt;
use core::borrow::Borrow;
use core::fmt;
use failure::Error;
use hocon::HoconLoader;
use log::{info, warn, Level};
use maplit::hashmap;
use rusoto_core::{Region, RusotoError};
use rusoto_dynamodb::{
    AttributeDefinition, AttributeValue, BatchWriteItemInput, CreateTableInput, DynamoDb,
    DynamoDbClient, GetItemInput, GetItemOutput, KeySchemaElement, ProvisionedThroughput,
    PutItemInput, PutRequest, WriteRequest,
};
use serde::{Deserialize, Serialize};
use serde_dynamodb;
use simple_logger;
use std::collections::HashMap;
use std::fmt::{Debug, Formatter};
use std::str::FromStr;
use uuid::Uuid;

type DynamoRecord = HashMap<String, AttributeValue>;

/// The info we need from a person in order to create a user for them.
#[derive(Serialize, Deserialize)]
pub struct UserRegistrationInfo {
    email: String,
    password: String,
}

impl Debug for UserRegistrationInfo {
    fn fmt(&self, f: &mut Formatter) -> fmt::Result {
        write!(
            f,
            "UserRegistrationInfo {{ email = {:?}, password = <SENSITIVE> }}",
            self.email
        )
    }
}

/// The canonical user record that is actually stored in the database.
#[derive(Serialize, Deserialize, Debug)]
pub struct User {
    id: String,
    email: String,
    hashed_password: String,
}

impl User {
    /// Creates a User from the bare minimum info we received from the user. Note that
    /// this function is not pure as it involves at the very least generation of a unique
    /// id for the user.
    pub fn from_registration_info(registration_info: UserRegistrationInfo) -> User {
        User {
            id: Uuid::new_v4().to_string(),
            email: registration_info.email,
            hashed_password: bcrypt::hash(registration_info.password, bcrypt::DEFAULT_COST)
                .unwrap(),
        }
    }

    fn to_dynamo(&self) -> serde_dynamodb::error::Result<DynamoRecord> {
        Ok(serde_dynamodb::to_hashmap(self)?)
    }

    fn from_dynamo(m: DynamoRecord) -> serde_dynamodb::error::Result<User> {
        serde_dynamodb::from_hashmap(m)
    }
}

pub struct System {
    pub config: Config,
    pub dynamo_db: DynamoDbClient,
}

impl System {
    pub fn from_config(config: Config) -> System {
        let dynamo_db = DynamoDbClient::new(config.aws_region());
        System { config, dynamo_db }
    }
}

/// The data returned to the user in the event that they successfully register a user
#[derive(Serialize)]
struct SuccessfulRegistration {
    user_id: String,
}

fn register(
    registration_info: Json<UserRegistrationInfo>,
    state: State<System>,
) -> Result<impl Responder, failure::Error> {
    info!("Creating a new user");
    let new_user: User = User::from_registration_info(registration_info.into_inner());
    let put_result = state
        .dynamo_db
        .put_item(PutItemInput {
            table_name: String::from("chat_users"),
            item: new_user.to_dynamo().unwrap(),
            ..PutItemInput::default()
        })
        .sync()?;

    state
        .dynamo_db
        .batch_write_item(BatchWriteItemInput {
            request_items: hashmap! {
                state.config.users_table() => vec![
                    WriteRequest {
                        put_request: Some(PutRequest {
                            item: new_user.to_dynamo().unwrap()
                        }),
                        ..WriteRequest::default()
                    }
                ],
                state.config.login_table() => vec![
                    WriteRequest {
                        put_request: Some(PutRequest {
                            item: new_user.to_dynamo().unwrap()
                        }),
                        ..WriteRequest::default()
                    }
                ]
            },
            ..BatchWriteItemInput::default()
        })
        .sync()?;

    info!("Successfully created a new user");
    Ok(actix_web::HttpResponse::Ok().json(SuccessfulRegistration {
        user_id: new_user.id,
    }))
}

/// Creates the user table. Note: Dynamo doesn't have a "create if not exists" so this needs
/// to be externalized to a good ol' cloudformation script or similar.
fn create_tables(
    client: &DynamoDbClient,
) -> rusoto_core::RusotoResult<(), rusoto_dynamodb::CreateTableError> {
    client
        .create_table(CreateTableInput {
            attribute_definitions: vec![AttributeDefinition {
                attribute_name: "id".to_string(),
                attribute_type: "S".to_string(),
            }],
            provisioned_throughput: Some(ProvisionedThroughput {
                read_capacity_units: 5,
                write_capacity_units: 5,
            }),
            key_schema: vec![KeySchemaElement {
                attribute_name: "id".to_string(),
                key_type: "HASH".to_string(),
            }],
            table_name: "chat_users".to_string(),
            ..CreateTableInput::default()
        })
        .sync()?;
    Ok(())
}

#[derive(Debug, Clone)]
pub struct Config {
    aws_region: Region,
    users_table: String,
    login_table: String,
}

impl Config {
    pub fn aws_region(&self) -> Region {
        self.aws_region.clone()
    }

    pub fn users_table(&self) -> String {
        self.users_table.clone()
    }

    pub fn login_table(&self) -> String {
        self.login_table.clone()
    }

    fn load() -> Result<Config, failure::Error> {
        let hocon = HoconLoader::new()
            .load_file("./application.conf")?
            .hocon()?;

        let aws_region = match hocon["aws"]["region"]
            .as_string()
            .as_ref()
            .map(String::as_str)
        {
            Some("local") => {
                info!("Using "local" as our AWS region");
                Region::Custom {
                    name: "local".to_string(),
                    endpoint: "http://localhost:8000".to_string(),
                }
            }
            Some(region) => Region::from_str(region).unwrap_or_else(|_| {
                warn!("Unknown region: {:?}, defaulting to us-west-2", region);
                Region::UsWest2
            }),

            None => {
                warn!("No region provided, defaulting to us-west-2");
                Region::UsWest2
            }
        };

        let users_table = hocon["dynamo"]["users_table"].as_string().unwrap();
        let login_table = hocon["dynamo"]["login_table"].as_string().unwrap();

        Ok(Config {
            aws_region,
            users_table,
            login_table,
        })
    }
}

fn main() -> Result<(), Box<std::error::Error>> {
    simple_logger::init_with_level(Level::Info).unwrap();

    let config = Config::load()?;
    server::new(move || {
        App::with_state(System::from_config(config.clone())).route(
            "/register",
            http::Method::POST,
            register,
        )
    })
    .bind("127.0.0.1:8080")
    .unwrap()
    .run();

    Ok(())
}
```


Get this bounty!!!